 Hello again. How's everyone doing? Feeling good? Pumped? Fired up? Yeah, I feel the energy. It's awesome So I am pleased to introduce Shyam who will be giving a great talk on a really important topic Which is hiring security. So please give a warm 1700 hour welcome to Shyam Cool. So a brief introduction about myself. I'm the head of product and application security at Coinbase In terms of ownership, we've got everything from user security like user-facing products to crypto nodes and then hardware and up It's kind of our purview So also for this talk This is a brief case study in terms of how we built our hiring pipeline Why we hire as we hire and kind of who who we look for Again, there are a lot of caveats here First is these are there are a lot of opinions on how to hire well and what you should look for when you're building a program These are my opinions on what I look for and what I think are important. These aren't the only opinions You're welcome. I encourage you to tell me that I'm wrong Second I'm using Coinbase as a case study here So I'm trying to avoid this sounding like a sales pitch as much as I can but I'm gonna reference kind of what we do What we are what's going on a lot in order to kind of get that information across Last there's a lot of simplification. That's going to be happening here There's many months and years of work that went into getting to where we are right now a lot of that I have to sum up in a slide, so I'm going to be glossing over some stuff. It'll be Maybe not that obvious So first off like any good case study. Let's start with the problem statement Coinbase at its core has four assets that we really really care about The first is cryptocurrency and we have a lot of it that can be irrevocably lost Second is user data in order to operate in the US, you know, we have to collect a lot of user data And we have to keep it for a long time. It's probably one of our largest liabilities The third is actual user trust that we'll keep the first two items safe and secure and Finally, it's an agile in engineering team that heartily embraces near continuous delivery With those four problems kind of established and kind of the the assets that we have to protect We've got two basic goals Keep those assets keep the engineers moving fast I really have nothing else to add here like this is about as simple as it can get in terms of summing it all up There's a lot when it comes to actual operations to enable these but at the end of the day That's that's all it is So what do we decide to do? There's three basic things that we did first. We want to integrate with sprint planning sprint retros kind of how our dev teams work So that's be present when the work is getting done. So we understand how it gets done. No kind of what how the sausage is made The second is work in the pipelines themselves. The devs have set up CI CD. They have automated testing They have code reviews for every pull request Add into those pipelines and processes all the stuff that we want to get done If they're doing automated testing build security tooling that sits right next to it If they're doing code reviews anyway, make sure they know when and how to tag security people into those code reviews And finally if they're you know, they're using feature flags and whatnot to ship fast get security teams opted into those feature flags Make sure that we're part of the test cohorts The last thing that we really wanted to do was not be a security team that only finds bugs, but also like starts to fix them We we think that there's no point if all you can do is kind of raise the flag We need people who are at least capable of patching small issues Maybe not the big gnarly deep ones, but at least superficial ones Essentially, let's not be dependent on another team in order to get our work done So with that, how do you to establish kind of who you're looking for? There's a neat little trick that just happened, which is as I described all the different pieces of what we're looking for We've kind of written up a job description It'll take a little bit of massaging to make HR and talent like actually happy before you can start to publish that But it's kind of four core things three of which are most important right now Person needs to be able to code review They need to be able to write security tooling. That's right in our DevOps pipeline They need to write do enough security engineering to be able to fix and like prevent core security bugs and code We also would love some some architecture reviews and threat modeling But if you can do the first three you can solve most of the problems So with that, let's start talking about the actual kind of hiring process. This is the meat of What we do and why we do it So again before we get too deep into that let's figure out what we want to optimize for We chose a couple core aspects first We want to minimize the number of hops we send a person through It's super painful being in a hiring pipeline, right? There's lots of people to talk to you're constantly being judged It's no fun. We want that information in that process to be kind of as clear and as efficient as possible at every step Because of that we also want to get as much like kind of signal as we can out of each hop While not adding a ton of cost to doing each hop whether that's the cost on the person who's giving kind of being interviewed or the interviewer on the other side End of the day. We ended up with three hops in about nine total hours of work Of course, it's not nine hours from start to finish It'll still take you know a couple a couple weeks to get through the fastest We've seen this kind of go through is about a week and a half So starting off there's two phone calls the first phone call is with our recruiting team It covers the basics Are you employable in the US or wherever else we're trying to hire you? Do you actually understand what the job is and do you want this job? Do you know are there any other kind of vagaries or things we need to understand about how you do work that makes you like Not necessarily want to work with us. Also, do you care enough about Coinbase to actually work here? again, the stress of having to protect billions of crypto is enough that you know You need to self-select in and it's easy to select out. We want you to make that decision as soon as possible There's a second phone call and that's with the the lead That call is not any kind of screening. That's mostly to answer questions for the candidate What do you want to know about the program? What do you want to know about how the team works? What do you want it like what questions can we answer? Is there work from home and so on so forth? We found that there wasn't a ton of signal and actually asking kind of probing questions getting information or trying to judge a Candidate at this point over the phone. Nobody really knows them. You have a resume to work off of and nothing really else So with that This is generally a very quick pass-through. There's very few candidates that actually stop at this point Here's where we do most of the work We provide a code sample that we asked are these developers and these security engineers to take a look at We look for three core things Please find the issues. Please prioritize the issues. Please fix the highest priority issue If you go back to what we said we wanted to enable our team with in the earlier slides There were three things that we wanted people to do right. We wanted them to be in code reviews finding bugs We wanted them to be able to fix their own bugs And we wanted them to kind of work with these dev teams understand how those pipelines and processes work Know how to prioritize get stuff in the right place This gives us kind of the most signal as far as the technical implementation of the job as we can possibly get Right. We're we're using this as the make or break for 90% of candidates One thing that we did to kind of balance out how important this is is we made a very robust internal rubric for how to grade this Which means every single vulnerability that we found or has been found or we know about has it's it's kind of its score What the vulnerability is and what an expected fix looks like We did this for two main reasons one We found that we weren't being consistent in scoring across candidates and given how important we made this that that was not That was a non-starter. We had to be better at that Second is we needed to reduce bias, right? Like part of inconsistency is some candidates are being selected out because they didn't write the report the right way Or they fixed a bug that somebody else didn't necessarily think was high priority We decided for our team these are high priority bugs This is what it looks like everybody like we agree with this rubric so everybody can kind of grade the same way For a candidate this is about four hours for for them to get through it's about 250 lines of code it's Sinatra It's not very hard to look at but still we there's about I don't know 50-ish bugs that you could find if you start looking for them When it comes to grading it takes us about 30 minutes to get through a report Worst case is about an hour like an hour where somebody fixed about 15 of the bugs that they found instead of just the first one What do we look for in this rubric right? They're scoring. Did you find enough critical bugs that you can we trust you to find critical bugs? Did you actually fix the bug right? Right? There's no point if you didn't fix the bug properly You have to fix it correctly And also did you write it up well enough that a dev team a dev team could kind of consume it and understand it Right is your prioritization there is that report good? This is however the only gate until they actually get to the the in-person review in Person is much more lightweight. All we're looking for at this point is Given that you've just you know shown competency in the technical part Can you actually get along with people and do understand it? For those of us who've been doing security for a while This is probably the hardest part of the job that isn't actually finding the bugs It's convincing everybody else to fix them It's being able to communicate about them. It's being able to work with people kind of in person and get through it And again, this is where we lean this interview towards we've we're comfortable with your tech competency up in this point Right the take home has established it We just want to know how well do you do with people? So we break that out into three main interviews The first one is like a pair programming we've asked you to fix a bug That's usually like the one critical bug takes about a line to fix and the code sample In this case, it's a couple more lines of code Can you do simple enough integration work that you could build enough tools that work in our dev pipeline, right? Like we need people who can automate against curls We need people who can write unit tests and so forth this only tests up to that point We're not looking for esoteric kind of Competencies in weird algorithms. We just want to know can you get stuff done? Again, this is a pair programming that's led by a developer on our team Not necessarily somebody on the security or the app sec team We want to know can you work with like a dev and can you guys sit together and get some stuff done? Again, the answer to looking for is enough code to write a simple task and work with a developer The second kind of major part is an hour-long session that we have it's a whiteboarding session We'll put we'll have a developer like an actual developer on our team put together kind of a rough sketch of an idea that they want to do Initially, that's all it was we have however kind of gotten rubrics there as well We realized that developers a didn't really know what they What were to go with one of these questions once they started asking and they got stuck and B? We started getting bias again in terms of poor questions being added. So again rubric everything But here it's kind of how we actually do work Developer will shoulder tap you you'll they'll sit for a half an hour to an hour and whiteboard something out We realize what we what we wanted was somebody who could go back and forth with the developer get an understanding of kind of What are they building? How are they building it where security controls need to be? But we also wanted a security person in that room to actually judge the quality of the interview The developers were judging against kind of the wrong thing when it came to came to output here They were saying hey is this person friendly? You're good to talk with I also want to know Can they find critical controls that need to be implemented do they understand where problems are going to come up? Again, we're we're looking for two things one is can you find those? core security concerns early and second is can you work with the dev while they're still just like at the whiteboard stage What what we want to do is by moving kind of to the left It's you're not going to have documents you're not going to have like large architecture diagrams or whatnot You're just going to have people with ideas who want to do stuff. Let's figure out how to help them do that securely The finals what I'm going to say is our softest interview question and this is probably again It's more fraught with bias and we don't have a good solution for how to reduce bias here What we're looking to see is how well can this person work with an edge manager on a different team? Right again like as an app site person a product like person you're going to be embedded with or working with all these different developer developer teams that are doing different things How well do you can you handle the conflict that arises when a dev team is pushing to ship features? And you're pushing to make features more secure This is you know, but can you handle the socio-political aspects of the job right? Can you have those tough conversations, but you know relevant, but to your work? Again, we see very mixed kind of results here and a lot of this Unfortunately is due to interviewer training and kind of interviewer experience if you're not good at giving behavioral questions And doing behavioral interviews this this section is really hard to run well So a quick summary as to where we are so far It's about an hour on the phone Just to are you interested? Do you get the job? Are there questions we can answer about the job? It's three to four hours on a take-home again async. It's this is how you'll do most of your work, right? There's code that's sitting there stuff that needs to be built Let's make sure that it's secure that you've done it and you can present that information back Again 30 minutes on our site to actually review it make sure you've done it, right then three to four hours in person Covering all the stuff that actually has to do with you being a person doing a job that has other people on the other side With that said there's a set of failure cases that I'm worried about and I've brought over and over again and the biggest one here It's bias Right. What is bias cause? What does it do? It's you exclude good candidates You also fail good candidates in your pipeline when you shouldn't have done so How do we introduce it even accidentally into the cycle? It's the activities that you choose and how you choose to grade against them So even in our case like we've acknowledged that by asking candidates to do take-homes We're cutting out a whole swath of candidates that otherwise would apply That hurts us, but we don't have a good way of replicating some of this we have ideas around reducing this requirement for people who come in as as Referrals or we've worked with before but again, that's like going to be the exceptional case. It's not going to be the generic case We also lose candidates to interviewers who don't know how to interview their parts, right? If you don't know how to run this behavioral if you're doing this pair programming and you're not looking for just base level code Competency you're actually asking for much more esoteric or harder problems Like we're going to lose candidates because our interviewers are doing poor jobs But there are paths to success here, right? These are fixable or mitigatable for the most part Again, we can have more paths to success for where we want more data. You know, can you do the core job? We want more activities people can choose from like there's going to be people who can't do the take-home What's the next best thing we can offer? Again those grading rubits that we've set up and having very kind of formalized Here's the definition of the interview is all about reducing some of that bias as well And finally, it's you sometimes have to rotate your interviewers, right? If you have the same people talking over and over again and they are asking and falling into the same traps, right? You're gonna have to a be there to know that they're making these mistakes and be actually rotate people in and out So you get fresh sets of eyes so There's a core principle going all the way back to the beginning is your program who you hire Why you hire it's all down to the people at the end of the day as much as you know Companies like to abstract out people using like metrics and goals and mission and so forth like levels Nothing's going to make as much of an impact as who you hire And we're going to break down who you hire into two key pieces. What is your problem? And how are you going to get the right people to solve that problem? We made the summation here very quickly in terms of what coin basis problem is We have lots of crypto Crypto we lose irrevocably We need to make sure like code doesn't get out that lets us lose crypto very quickly same with user data same with user trust With that we needed people who could work with dev teams early work with code before it was deployed and be able to you know add to our build pipelines and fix pipelines to make sure that Again bad code wasn't getting shipped But again, it's these people that we like aim to hire and like that we have hired their knowledge their skills and their hard work That actually makes things happen so Quick quick outro here It's if you agree with me if you disagree with me if you want to hit me up If you have any questions about kind of why we chose our pipeline if you're working on your own pipeline You want feedback in terms of how to make it better or how to change it problems that we may have seen Hit me up on my email. I don't really use social media I don't like it. It's very biased Again, thank you Yeah, so the way we modulate some of this and I to gloss over it for this interview is you take the different aspects of the job And you determine kind of how important they are and how difficult they are So when we first did some of the problem discovery here at Coinbase We were like hey, this this is a rails code base. It's been around since 2011 We really need like competent rails devs that needs that requires more seniority As that kind of starts to go down in terms of we've fixed it up or we have fewer bugs that we see coming in We can modulate kind of how good a candidate needs to be in that particular category of work One thing that we haven't found a solution for was that cross-team collaboration bit At that point like even our definition of like level one entry level has way higher cross-team collaboration and kind of socio and Empathetic requirements than any other team on the company Application product security is seven plus a couple contractors How many how many candidates do you reject at the third stage? Which is the which is the bucket of the three that typically rejects them? Yeah, I'll answer the second question first Most drop-off we see is people going from phone call to not doing the take home. The next drop-off is after the take home Rejections after the in-person interview is probably 10 percent to 20 percent We see enough drop-off in the earlier pieces that most people who make it that far And they're they're generally have been reasonable people because they've had jobs in the past We'll pass at that point Yeah, so what we found was it wasn't necessarily bad as the like written job description and people self-selecting wasn't nearly as good and so By making the take home we actually ended up making our job description better Which then reduced the need for our take home it also kind of trained our recruiters in terms of what we were looking for Which also reduced the need of take home So it's kind of like we made ourselves do this work and then we had it in place then we started using it But we didn't need it as much That's where we keep looking for a uniform Yeah, it's gonna happen as well like your interviewers will make mistakes at some point They will just ask technical questions when really they want to know how well are you interacting here? We you can't get away from that So Do you find more success with people in security who are a generalist or especially it's like for example someone who knows like five different languages Decently or like one or two I'm gonna say it really depends on the problem scope Part of what we defined here is we wanted people who could work at the level our developers are working and with them That implied we needed somebody who is really good at rails in particular, right? We De-prioritize their need for kind of threat modeling an architecture review that only comes way later in the life cycle and the whole reason We did that is because this is like a roughly a relatively baked application. It's big It's there's lots of pieces to it, but it's not changing architecturally a lot So we didn't need somebody who had seen a lot of problems or seen a lot of things. They just really knew rails pretty well I'm being asked to do this Let's shut after oh Want to take we tell them it they should aim for three to four hours, but we don't actually time it Alright, thank you