 Welcome everybody. Good afternoon. Thank you for everybody here in person and for the millions watching at home. It's nice to see everybody. Hopefully the first day was great. My name is Jeff Strauss. I'm a technical program manager at Snapchat. Today I'm going to be talking about the interview environment and how to create a more inclusive, diverse, equitable and aware environment for the interview itself. So today's talk will be brief, but we'll go through a history of the interview as it stands today and then some strategies for creating a more inclusive environment for the interview process itself. Welcome. That's me. I saw this on LinkedIn the other day. I thought this was apt for this talk. Craig Fenton, director of strategy and operations at Google, posted this and said some of our most successful candidates had average grades, didn't go to the best schools, and had great street smarts, weren't necessarily academically skilled in the traditional sense, but had a lot of other skills. I thought that was really apt for this talk because part of this talk as we get towards the end talks about how do we assess those skills, rather than just reading bullet points on a resume and having those talking points, a very sort of traditional interview format. How do we break out of that traditional format, make it a little bit more modern, a little bit more exciting, and a better assessment of those types of skills that may not be reflected on a resume for a number of different reasons that we won't necessarily get into today, but still are within somebody, somebody has the aptitude to be able to represent those skills. So we'll talk a little bit about what's going well traditionally at companies now. There are a growing number of pre-higher inclusion, diversity, equity, and awareness initiatives at companies. We see this manifested through a couple of different ways. One starts with the question like how do I create my dream team? Where and how do I source top talent? So we have inclusive inbound and outbound recruiting efforts nowadays. We can source from more underrepresented group like sources nowadays rather than traditional networks that people who are employed currently have. Typically we see people recruiting from their own networks, which tend to be more homogeneous than diverse in a lot of cases. So we have a lot of those recruiting efforts. We see things like applicant tracking systems, making strides in terms of removing biased language in job descriptions, making the job descriptions a little bit more inclusive. I have a couple of examples. You see this in things like removing barrier words, words like dominant, words like understanding that may be more traditionally associated with males or females to a little bit more sort of gender neutral. And then you have a general narrative shift from screening people out. So we want to screen candidates out to screening candidates in and sort of that narrative shift. Although utilitarian, utilitarian, utilitarianly, the end result is maybe similar. The mindset shift is very important to screen people in and make a culture inclusive rather than exclusive. And finally we hear it with how do I build networks? How do we expand referral networks to incorporate more diverse groups? Because like we talked about before, the majority of employees refer people from their own sort of homogeneous groups. Key question being how do we assess talent? And there are a lot of those pre-hire initiatives that are doing quite well at companies right now. Companies are including those more and more. And you see a growing number of post-hire idea initiatives also. This is after a candidate has been employed at a company. They've got me offered and they've accepted. You see this with a lot of equity and inclusion training at companies, right? These equity and these diversity trainings are becoming more and more normal. They generally focus on bias awareness and elimination, battling biased systems to ensure colleagues are reflected, expected and respected. Diversifying working networks like we talked about before, allyship. You see this manifested not only in traditional training, like there's inclusive leadership training that managers might go to or bias training for employees. You see these in ERGs and groups like your chat groups, meetups that may occur inside or outside of the company. So there's a lot of those that are going on now. And it manifests in a couple of particular domains post-hire as well. One is development. How do we make sure that there's equitable access to development for people on the team or in the company? Performance management. So if somebody is performing well or not performing well, how do you make sure bias doesn't influence who you rate as a success? How do you define success equitably? How do you ensure people are managed fairly? And how do you manage exits fairly? And it manifests in promotions. So you see it like how do you make sure there's equity and equality in the promotion process? What do you look for when you want to promote somebody? And how do you ensure that there are opportunities for underrepresented groups to advance into leadership positions? So post-hire, there are a lot of great initiatives that focus on those questions. But right now when it comes to the interview itself, we have typically a standard template. It's my words. We have a hiring manager that may be part of an interview panel. We have HR that may be part of an interview panel. We have a cross-functional partner, a technical assessor, and a cultural assessor. And some of those might overlap. The HR person might be the culture person, and the hiring manager might be the technical person. But that's generally the standard template that we have for an interview panel. And they're chosen on the basis generally of two things. Subject matter expertise and availability, right? Who has the domain knowledge to assess an individual? And is that person available to actually do the assessment? And there are some problems with that standard template. Three big ones are inconsistency, fatigue, and reluctance. Inconsistency, because if a role, for example, has eight candidates that they need to interview and that subject matter expert is only available for three of those interviews, you have to find somebody else to do it. As soon as you find somebody else to do it, a lot of things about the interview change, right? The standard bar is no longer the same because you have a new person coming in that may have individual biases that are different than the other person. They may interpret responses differently. They may ask different questions if they have to be like, oh, this person didn't show up today or can't do it today. We need to find somebody to fill in. All of a sudden, the standardization changes. And that's problematic in an interview. Fatigue, I think if anybody here has been on multiple interview panels, I don't even have to go on, right? It can be tiring. It takes a lot of time out of the day. Sometimes it's a struggle to find things to talk about. Your risk running the same... I think I speak for everybody potentially when I say that you've gone into a job interview where you've been asked the same question from multiple people over and over again. What's your interest in the company? Let me repeat it. So there's that level of fatigue and there's reluctance. Fatigue leads to reluctance, right? The more you have to do it over and over again without sort of like a respite, the more you're unwilling to do it. And that manifests itself in the interview, right? We're talking about the interview were, the interview panel. Now that manifests itself in the interview. There's a lot of research. The slides and including the references to this research are on the website. But there's a lot of research that shows candidate discomfort for a lot of reasons in an interview, especially underrepresented candidate discomfort. It's uncomfortable enough being in an interview for a new job, but underrepresented groups feel it even more. This is manifested typically in increased pressure to conform to assumed or perceived interviewer expectations. We see it a lot manifested in how an underrepresented group dresses for an interview, for example, in a hairstyle that a candidate may choose to wear. You hear it in speech. A candidate may change their pattern of speaking to conform to a perceived expectation within that job interview. In-group and out-group bias can be particularly strong here. In-group and out-group bias is a subconscious reaction to seeing somebody that we identify with as part of our group versus an other. And that could be manifested pretty strongly in an interview. Favoring an in-group member tends to grant them a measure of trust and regard them more positively than an out-group member. And I'm going to talk about how we resolve this issue. But finally, there are microaggressions as well that you see manifested in interviews. A couple of examples being questions like, where are you really from? You're so articulate, like little phrases like that that come across as microaggressions. And the more frequently, or the less you attempt to interrupt these biases, the more they'll manifest. And the interview process not only stays the same, but can get worse for the candidate. And we'll talk about that in a little bit as well. So all of these interview problems come from that standard template that just sort of ad hoc picking of, this person knows what they're doing, this person's that subject matter expert, and this person is available. So let's get them in and, okay, cool, we've created a hiring panel. So we want to do it more thoughtfully. How do you interrupt bias in the interview process? First, create an inclusive panelist pool. There are a lot of strategies to do this. I'm going to enumerate a couple right now. A couple of things that are really important to have in an interview panelist pool are buy-in from the interviewers. Interviewers have to be bought into the process. So I think if anybody has had this experience, I know I have, it's a little bit different when I'm enthusiastic about giving an interview, rather than HR reaching out to me and saying this other person is available, are you available to do an interview? Right, buy-in is really important. And there are a lot of different ways that you can solicit opt-ins from people. But that's really what you want. You want people to opt-in. You also want to set context about those opt-ins or about those interviews with those people who opt-in. What I mean by that is people typically anticipate, or I don't know how people anticipate their interviews, but typically it's not, I'm on a panel for a particular role, especially if you're a cross-functional partner, especially if you're a cross-functional partner. If you're a hiring manager, you're there all the time, potentially HR as well. But if you're a cross-functional partner or another domain expert and you're pulled in for an interview, there may not be an expectation that you're in that interview for the entire length of the role of that interview rather than the number of candidates for that interview. So it's detrimental to the process if, for example, there are 10 candidates for an interview and the strategy is you'll do five of them because that's five hours of your time or however many hours of your time, including prep, and then somebody else will do the other five. The expectations should be set that you're there for the role. The panel should be consistent. All right. You can reduce that fatigue that we talked about earlier by rotating per role. That's a really easy one. Although because the commitment has to be set for the role, there may be those roles that are like seven candidates, 10 candidates, and that could be a little fatiguing. But if you have buy-in from the individual, that helps a lot. That resolves a lot of that fatigue. It's also important to have a group of diverse interviewers. The interview pool should be representative of the diversity of the company, representative of the diversity of the team, and not a homogenous group. And it's really, really, really important as well that every interviewer be trained to deliver interviews. I think that's an oft-overlooked component at companies. People are just like, here's this person's resume. Go in, talk to them about their expertise. Will they be a fit for the team? Will you be able to work with them? Will you be able to communicate with them? And typically it's like, let's go through the resume. Let's look at the bullet points. Let's pick a couple of things to talk about. Tell me about your time here. Tell me about a time you had a challenge that you had to overcome. Very sort of like routine questions. But there's a lot that can be gleaned from behavioral interview training, which is really important. This training can include inclusivity and unconscious and implicit bias training. You may not be able to eliminate it, but you can at least recognize it. It includes feedback training because you'll be debriefing. There'll be a post-interview component, hopefully, where you speak with the rest of the team and discuss the candidate itself. That should be held in an inclusive method, and feedback training is really critical to be able to do that, to be able to eliminate the sort of microaggressions and biases we spoke about earlier. When referring to a candidate later on. I'll give you a little bit of example about what we do at SNAP and what I've done at other companies prior. Before an interviewer is selected, before an individual is selected as an interviewer, they go through an interview certification process. And that certification process includes several hours of training, one-on-one training, video training, group training, that sort of thing, to be able to do a lot of these things. We have inclusive leadership training, feedback training, organizational conversation training. That's also known as conversational leadership training. And I'll talk a little bit about that in a few moments when we get to the holding the actual interview and methods for actually conducting the interview. But those are all really important ways to strengthen the interview process to be more equitable and inclusive, especially for those candidates that are typically from underrepresented groups and aren't able to receive that. They don't see that elsewhere. The other really important thing to interrupt bias in the interview is to standardize your questions, your discussions, and your evaluations. In my experience, and you may have experienced this as well, different candidates might receive different questions from different people. You might pull in person X, and their conversations or tone or the direction that they take the interview may be completely different than anybody else. And it's per candidate. Let me specify per candidate. So candidate A might receive a completely different set of questions and conversation than candidate B. And it's okay to have open conversations and be extemporaneous, but questions that you ask for assessment purposes should be standardized. Otherwise, you have no standard by which to evaluate the individual. If you have different questions for everybody, some people might excel, and some people might not. So it's really important to standardize those questions, standardize the discussions as well, and standardize the evaluations afterwards. Some strategies for this include creating hiring templates or interview templates, rather, where somebody's not just going into an interview blind, here's a resume, here's a background of this person, but here's a template for this role that you're interviewing for. Here are the questions on the template. Here are the discussion points on the template. And here's a behavioral rating. So a behavioral... I won't go so far as to say analysis, but like a behavioral rating for the candidate. And what I mean by that is, you know, some people may have seen, I know I have the one to five rating, no to yes, right? Strong yes, right? I'm giving this candidate a strong yes, and that's a five for when I want to hire them. Strong no is like a one, right? And you have that sort of rating scale in between. Rather than having a strong yes or no, perhaps it might be more effective to have a behavioral scale, where, for example, a five might be yes, did exhibit this behavior, or no did not exhibit this behavior. And it gives you a little bit more insight into rather just a yes or a no, I want to go for this candidate, but did this candidate, how strongly does this candidate exhibit the behavior that I'm trying to seek out? And in the slide notes that I have, I'm not going to go too deep into it, but in the slide notes that I have, there are some examples of how to create that interview template. There's a couple acronyms in there that you can use to set something up, but it's important to standardize all of those questions without every interviewer, even if an interviewer is sort of substituted for whatever reason, you can still maintain that consistency across interviewers for the role. And finally, iterating on the conventional framework. So, so far, so far in this quick talk, we've talked about a little bit of the context, a little bit of the history. We've talked about our sort of standard interview template and a couple of ways to, you know, make it more standardized. And now we'll talk about some creative ways to maybe change up that sort of routine to evaluate, like we talked about earlier, try to evaluate those behaviors that may not come across as bullet points on a resume. So these are some of the ones that I really like. How many people in their interview processes share questions with candidates ahead of time? I'm curious to know. I haven't seen that very much. We like to share questions with candidates ahead of time. We do it for a couple of reasons. One, it allows for really thoughtful and deep answers, right? As opposed to asking somebody, what's your interest in the company or have you heard about our latest lens product or do you know what Bitmoji is or asking those sort of questions and getting an answer on the fly? It allows an individual to answer more thoughtfully. We do this with our engineers as well, like if anybody has given a take-home coding test in the past, right? And we come back and have discussions about that. It's a little bit different than the whiteboard experience in the room. So we really like that. And we get to evaluate when we have those responses. We get to evaluate a couple of things. We get to evaluate written communication style. We get to evaluate competency of answer. And we get to evaluate preparedness. Did you return these questions to us when we asked for them? The other thing that I really like to do in particular in order to help me evaluate an individual's aptitude to perhaps mentor somebody on the team or help somebody else develop is to ask a candidate to teach me something that they're an expert in. Like assume I know nothing and just teach me something that you're interested in. I want to evaluate how you communicate, how you instruct. Let me ask stupid questions that cause other experts to bite their nails and grind their teeth and see how you respond to that sort of thing. And finally, one which is a little bit harder to implement but is really fun if you can, are games. So this is a really huge departure from your routine. I have a resume in front of me and I'm going to ask you questions about it. One of the things that I like to do and we at SnapR Values are kind, smart and creative. So one of the ways that we like to think that we're being smart and creative and kind in this interview process is to ask candidates to play games with us. We typically choose games where an individual has to work with others towards a common goal, right? And so we'll bring in a couple of other interviewers from the panel and we'll play, you know, in a virtual world you can do something like a game called Codenames is a fun one if you've ever played that game. But there are a couple of other ones as well and you can play them in person also. But this allows you to evaluate how a candidate works on a team, how they communicate with each other, if there's a little bit of pressure added, how they deal with that. It allows you to really understand much more thoroughly from behavioral standpoint how a candidate will actually function, you know, rather than just acing the interview because they interview really well, right? Because there are a lot of those candidates, I personally try to hire double A candidates is what I call them. I say they have a positive attitude and an aptitude. If you have the aptitude to learn a positive attitude, you'll go pretty far. And a lot of stuff is learned on the job. Not everything. You have to have a fundamental basis, but if you have the positive attitude and the aptitude to learn, you know, the results are very surprising. So trying to understand how to structure an interview in order to tease those things out rather than just what's on the resume and the reason being this, you know, I might be preaching to the choir, but reason being, especially regarding underrepresented groups, not everybody's at the same starting line. And there are people who have had a tremendous amount of opportunity, but there may be others who haven't had that opportunity, who are very strong or will make incredible cultural additions to the company, right? That you may not see reflected on paper. And those are the, like, that's the true, those are the true gems, right? Those are the real things that we try to tease out to create an inclusive and diverse and equitable culture. And that helps everybody. And finally, it's important to evaluate, iterate, and experiment. So after we have an interview loop like this, we ask a couple questions. We, first, we ask ourselves that the process worked. If we hire a candidate, were we able to hire the strongest candidate, right? Did we do a good job at that? Were we able to make better decisions based on this process than we may have made with a standard, like, you know, routine process? What areas of this process may have been most impactful, right? Do we glean the most information about our candidates by asking them to teach us something? By asking them to write responses for us? Or was it the whiteboard test inside the room, right? Which things actually worked? And we'll focus on those areas. Where should more time be invested in the process? If things didn't work quite right, but it shows promise, can we invest more time there and iterate on that process? And finally, two of the biggest things are, after a candidate is hired, ask them for feedback. How did you feel your interview went? And talk to the panelists for feedback as well, right? Was it more difficult for the panelists? Sometimes this can be a really difficult process for the panelists to acclimate to because it's so different than your standard, I have a resume in front of me, and I'm just going to ask sort of routine questions. Sometimes candidates can feel locked in, especially if there's that, if you know there, do you have, okay, here are my eight questions that I have to ask of everybody. It's hard to deviate from that. You can have those, this is where sort of conversational leadership training comes in very well because it trains you to have open, but not aimless conversations, right? So it makes it a little bit more conversational. And the improved framework, if you're able to sort of like, you know, create something like this in your place of employment, does a few things. It reduces bias. It dramatically improves the candidate experience. If for not only trying to reduce bias in an interview by making it different, it's a different type of interview. It's much less stressful. It can be much less stressful, but you don't lose any of the capability to extract the information that you want to extract from an interview. In fact, it may even be more enlightening, and it's a much more positive experience for the candidate. It can improve the quality of new hires, and it creates brand ambassadors, to use that buzzword, even if a candidate doesn't get hired, they leave saying like, wow, my interview experience at this company was incredible. Unlike anything I've ever had before, I would love to work at a place like that. And for those people that are there already that made it through the interview process, it creates a deeper emotional connection to the company when you go through an interview process that's so novel and different and is really caring towards the individual, right? You're not just sort of like sitting behind a pane of glass, assessing a person like rapid-fire, but you're involved in the process with that individual. So it creates a really deep emotional connection which translates in a number of ways like into the workplace, that individual's quality of work, their partnership on the team, or in the organization, et cetera. Many thanks to you for showing up this afternoon. Hopefully this was beneficial in some way. But if you have any feedback or questions, I'm here for the next few minutes. Yes, absolutely. I'm not going to hawk myself here, but I will upload a new version of this slide deck by the end of the day today, and I'll put a link to a couple of them that will link externally to some of those templates just to give you a verbal overview of what they are every team in our company. I'll speak for SNAP specifically. Technically, I'm not here as a representative of SNAP. I'm here on my own, but I will let you know what we do at SNAP. Every single team has a standardized interview template. So every team, and that team, together, along with the hiring manager, leadership, and other people who are involved, cross-functional partners, et cetera, come up with interview questions that are pertinent to that team, and that's on the standard template for the interviews. So a new interviewer on a panel, for example, just has to go to that folder, say, okay, here's the role I'm interviewing for, and here's a whole lot of documentation about it. Here's the standard interview template. Here's, you know, behavioral conversational notes. You know, idiosyncrasies of the team, for example, things to look for. On every one of those templates, there's that behavioral assessment I mentioned. So as opposed to like a one-to-five yes or no, they did like a four. They did a pretty good job, but who knows really what that means. With every question that's asked, there's a little, you know, a one-to-five that's did this person strongly represent the attributes that we're looking for? Some of those might be, for example, did this person step up to the plate, right, or did they shy away from challenge, right, or did, you know, were they going at it gung-ho? Those are the sorts of things that we want to like evaluate specifically. And when we look at, when it's a very methodical process as opposed to more ad hoc, which it typically can be in an interview, there's a lot that can be gleaned from that. And so we use that data a lot. We're also careful not to lean too heavily to like represent an individual as a number, right. These are individuals that are coming in and everybody has their own unique skills and talents and ideas. But we do leverage that sort of behavioral model to give us a good middle ground about somebody's skills and to be able to evaluate, to assess them, but also to allow them to enable them to express themselves authentically through the interview. Does that answer your question? Yeah. Okay, you're welcome. You're welcome. My pleasure. Yeah, and so I leave back to LA tonight, but before I land, if you all want to check on the website here, I'll have like a version two of my slideshow up and it'll have a couple links in there. All right. If nothing else, thank you so much. Appreciate the time.