 Thanks for that. Yeah, so the first, sorry, I'm Sid. I'm CEO of GitLab. Alex, you want to introduce yourself? My name is Alex Becker and I am VP of Engineering at Hacker One. We met at a Hacker One event. It was an awesome event in San Francisco on a Saturday. You guys distributed a whole lot of book bounties, hundreds of thousands of dollars. And in the margins, we were talking about an interesting thing that we're doing at GitLab. And an interesting thing Alex is doing at Hacker One. The thing you were doing, Alex, is hiring with a pitch up front. And I thought that was super interesting. We're joined in this call by Barbie. She's the chief culture officer at GitLab. We'll publish this recording and Alex had something really interesting he shared. So I'll let you, maybe you can describe what you do. Are you talking about the hiring with the pitch up front topic? Yes. Okay. So today at Hacker One, well, first and foremost, our direct managers own all aspects of hiring into their team. Obviously, there's a number of ways that they can collaborate with either other managers or other functions, such as recruiting for various support functions, but they own the recruiting process into their team. And as you know, about half of our engineering team is in Groningen, Netherlands, where the market is, the talent market is fairly small. And so it was very difficult for us to hire local candidates. And what we've done instead is we have obtained a permit to sponsor visas for people coming from all over the world. And one of the challenges we ran into as a result was overcoming all sorts of cultural biases and cultural barriers for people to agree to essentially uproot their entire life and move to a foreign country. And what we figured would work well and have shown to have worked well for the nine months or so we've been doing this is we have revamped our hiring process in addition to effectively interviewing people half a world away, starting off with pitching the opportunity by the direct manager themselves to the candidate to essentially up their level of excitement to join Hacker One so that the conversion rate from the candidates we reach out to and who agree to interview with us, that they persist through the entirety of the interview process and have a kind of strong belief that the Hacker One opportunity is the right one for them, not just in terms of joining us as a company, but also essentially turning their entire life upside down and taking what is an enormous risk. And so when I talk about cultural barriers, what I mean by that is a lot of the folks we hire tend to come from places that are much more conservative and so they don't necessarily see relocating for work or leaving their family behind as either practical or even a possibility. And so we try to overcome things like that with getting really excited about the opportunity to join Hacker One, which then ups the conversion rate for people to work through the interview process and hopefully at the end of it, if both sides see a fit that they agree to join us. That's really interesting and what is your pitch like? Do you stress the fact that it's a product company instead of consulting or that it's a fast-growing startup or do you pitch where they emigrate to, the Netherlands and what new opportunities they get or do you pitch people that have done it before and found it a good experience? How do you pitch the opportunity? That's a great question. It is a mix of all of the above. It really depends on the candidate we're talking to. Certain people we talk to have a lot of ambition across the board. People who are even willing to consider the idea of uprooting their life and moving halfway across the world tend to come with a certain level of ambition. And so we try to tap into it by talking about all the great opportunities we offer at Hacker One. The great thing about being a growing company is we do have a lot of opportunities for people to take advantage of if they were to join the company and that tends to feed into their own personal and career ambitions. So we really talk a lot about the growth of the company, what are some of the opportunities in front of us from a business standpoint, from an industry standpoint, talk about what we've been able to accomplish so far in our track record and then helping them understand how can they fit into that as well as talking about our culture which kind of touches on the other topic that I know you want to discuss, which is the tying or having a strong association between the work that engineers do and the business goals of the company. So the positive business impact and really painting a picture for them that this is an opportunity for them to come in and really have that kind of an impact. This also works well with, as an example, we hired a few engineers from places in the world, female engineers, who come from places where it's not common for engineers to speak up in meetings. For example, there's a lot of the cultural norm is directive comes from the top and you pretty much are just expected to follow orders and execute. And people get excited about the opportunity of joining a company where they can have a voice, they can assume more ownership. And so all of that is also rolled into our pitch. There are certain candidates who have, you know, certain barriers to moving over that are unrelated to the work itself. For example, folks with families and children, we do a lot of selling on, you know, how great a place of Groningen is. We actually have done quite a bit of research, especially more recently as we set out to hire more senior folks about things like schooling and healthcare and all the other benefits to moving a family over. And so a lot of those selling conversations then tend to pivot to some of those sort of lifestyle conversations for folks who are considering moving to Groningen. So we really tend to tailor the pitch to what the situation is, if that makes sense. That makes a lot of sense. Obviously the hiring manager is a great person to give that pitch. What do you think about, how do you feel about like an in-house recruiter giving that pitch? Someone that works at the company so they can speak with conviction about these things? So the way that I think about recruiting is, the next thing about where we are as a company today is we're small enough of a scale where small enough of a company size where scaling or hiring isn't yet a bottleneck. And so all of the engineering managers can really take the time to learn and understand hiring, which to me is essentially a conversion funnel. At the bottom you have people who show up on the first day. At the top are people that we identify to be potentially good candidates who we run outreach for and all the different layers in between. And I think I've counted probably 10 to 12 of them. And it gives an opportunity for managers to really understand what those layers are. In part because in the future we expect them to having understood the inner workings of them and how to optimize for them that they can then subsequently operationalize them. Meaning that they can distill certain important elements of each of the layers of hiring down to a set of instructions for example that they can then pass on to somebody else who can then execute. So as an example a lot of our managers do their own sourcing and in part it's because it gives them an opportunity to both perfect their pitch as well as learn how to effectively filter out to find people that they... In the future my hope is that... I'm sorry? Yeah, there was a bit of lack and I lost you at how to filter out. Okay, so they get a sense of how to filter out to be able to find the people that they think could be good fits for their teams. And then in the future the hope is that having all of that knowledge will allow them to then operationalize that layer of the hiring process and essentially create something that they can hand off to a recruiter for example to be able to execute on that and be as effective as them. So if they have a great sense of what kind of messaging works with what kind of people they're looking to hire whether it's skill set or seniority of part of the world and what messaging resonates with people and this comes into the... now we're talking about the focus that... or the emphasis that we put on or either it's the company or the industry that we're in or the technical opportunities or whatever it is that initial sort of outreach topic quote unquote that they can then pass that on to recruiters to be able to execute on and be as effective at. And so now we're at this phase where we can do a lot of these things that don't scale longer term we want to apply that knowledge to be able to operationalize each of the steps of the hiring process. And have you tried that yet transferring something to a recruiter because what I have found I think the model you're using is great in terms of I think that in addition to the benefits you've mentioned my experience is also that many candidates would be much more responsive in having that conversation with the hiring manager than the recruiter that they feel they actually can have a good technical discussion that they feel they actually can learn something from it even if they don't take the job and that they actually feel like they can get to know the actual team environment management style. So I find that that helps as well. So have you found... have you tried to transfer that on to a recruiter and found that you've been able to succeed with it being just as effective or have you found that there is some loss from that connection not being directly from the hiring manager? I mean there obviously is expected to be a loss because as you said people are more likely more willing and excited about talking to a direct hiring manager. What I was referring to is specific to sourcing as an example which means that if we don't feel too confident about the conversion rate of passing on the sales call to a recruiter we can still maintain that or have that be owned by the direct hiring manager. The nice thing about it is because if each direct hiring manager is responsible for hiring into their own team then it can scale a lot more versus sourcing tends to be a bottleneck a lot quicker. So the sourcing part is to try to alter recruiters more recently and have seen some initial early success with it and I would say that the outcome of that was as good as a direct manager would be able to pull off and yet we still do all of the sales calling and this is more specific to Grunningen than San Francisco we do still most of the sales calling by the direct hiring manager. Obviously you know longer term across all the different steps of hiring we'll have to be selective about whether we feel comfortable kind of scaling out horizontally depending on the kind of maintaining the conversion rate but we haven't run into that those problems yet. Cool really interesting model I don't have any further questions about this before we move on Barbie you anything you want to ask or add I see you shaking your head yeah it's my understanding that you were interested in how we make sure our engineers care about our business goals is that the case or I mean though that was a back and forth discussion you and I had and I think you had some good ideas on it I shared our sort of methodology with you as well so oh I forgot I forgot what that was can you refresh my mind we talked about the at least from hacker one's perspective the importance of engineers caring about business goals and essentially seeing the kind of the fruit of their labor if you will being the or being tied to a positive impact on the business and not necessarily you know how technically complicated problem that they're solving or the code that they're shipping to production or the features that they're shipping really about the positive impact their work has on the overall success of the business and we sort of built out a whole ecosystem around it including how we how we hire how we communicate to candidates how we internally set compensation and leveling thresholds so how do you weave that into for example compensation do you when when kind of promoting someone you look at how they infected business goals or how do you do that yeah so our we have a leveling framework that is entirely focused on level of impact so we don't we explicitly exclude things like skill whatever that means to people you know years of experience age what school they come from what past companies they've been at what degree they have and focus exclusively on what positive impact there has their work had on the success of the business and when the promotion process works such that the direct manager would draft a essentially a proposal for a promotion that describes that business impact and then we as a as an engineering management group get together to to debate that in the end it is the decision that's still up to the hiring manager or the direct manager however the entire discussion is focused around the impact on the business we talked about you know these are the goals that we set out for ourselves this is what we've been able to accomplish this is what this person role role or impact was to be able to accomplish those things and then for each of the levels we have certain definitions of what level of impact their work should have and then over time as more and more people get promoted each one of those documents serves as essentially a you know kind of like I'd like to think of it as a court of law system where you have the language of the law then you have every time it goes through a challenge through court there's a good data point a proof point to support that and so over time these these levels and level of impacts that are associated with it are backed up by historical promotions in the documents that were written to support them as a kind of an example or representation of that level if that makes sense yeah that makes sense I'm looking at one of our promotion documents and we do have like technical skill and what we for example say is like you have to be able to write more modular well tested maintainable code and we link to kind of code they've written that proves that they they possess that skill and you can yes it's it's a skill on the other hand it's like the you look at the actual results they got the actual code they wrote that that proves that they have that now we luckily also look at other things like leadership and code quality communication performance but I like your way of tying it even more to business goals and it's something we'll consider for sure yeah well one of the ways that we try to mitigate the risk of you know the people talk a lot about 10x engineers and or ninjas or rock stars and whatnot which we like to joke that are people who write a bunch of code and get a bunch of credit for accomplishing certain things and then leave a ton of maintenance debt for the rest of their group to maintain as a result of that quote-unquote velocity but we try to manage rather than incorporating it into into the the promotional cycle and therefore it gives us an opportunity to kind of simplify the promotional process where the entirety of a promotion is based on the overall impact on the business and because it is squarely tied to the to the goals of the company that it is sort of independently verifiable so anyone looking at the justification for the promotion can understand the reasoning for it because it is tied to company goals that we have all aligned behind as an organization so it even goes beyond engineering. Yeah we know at higher rock stars I just read something about guns and roses and they tend to show up late under the influence and trash the dressing room so we try to prevent those hires. Although I love guns and roses, it's for the record. I just say don't hire them in your development team. Absolutely, that's their point. I think that's really interesting and I think we tend to, we try to tie it to results. For example when we give a bonus and I think we can do it a bit better when we when we promote people so that's something we'll be looking at. Something else we discussed maybe for the benefit of the viewer because your memory is probably better than mine but is kind of making sure that there's alignment throughout the company what the business goals are and what we do for example there is our number one metric is incremental revenue that we get on our subscriptions and we have a monthly goal for that and if we achieve that goal everyone gets a free dinner to kind of celebrate that and another thing we do is we make sure that there's clear objectives and key results that start from high three high level objectives for the whole company and that trickle down to all the different teams and those are public and and those are made kind of bottoms up by the teams that have to have to make them while making sure they're aligned. Barbie do you have anything to to ask or add? Yeah I would love to know how or who is responsible for the team level OKRs that you mentioned that you derive from the company business goals that's one of the things that we're setting out to do right now as kind of a middle layer between the company goals and what the teams are working on day to day and I'd be curious to know what the process is currently for you. Yeah I'd be happy to show you if you Google GitLab OKRs you come to a page and you find all our OKRs and they start with an objective this is the objective one of the three then some key results that are associated with that and if you look at how that came together it's all version controlled in Git it won't surprise you at a company called GitLab and then you can see that there's a whole diverse set of people that contributed to this like this is one of the last contributions from someone that recently joined the company and he he updated the hiring ones he updated like how we're doing so far which is great and how do you how do you kind of formalize them at what point do you rubber stamp it to say this is what it is? Yeah it's they're never done so they're always they're always open to change if the companies if the priority shift we change the OKRs we won't be afraid to do that now what we try to do is before the quarter starts make sure that they're kind of up to snuff and then sometimes the first few days of the quarter they they still move and after that there's there's a significant drop-off but we will never like chase an objective that has become redundant because the world changed and how did how did the individual contributors the middle management and the functional heads how did they all fit into the process? Yeah so they can kind of add their make their own merge request with proposals so kind of I I create the top three ones and then all my reports start filling it out below that and then all their reports start filling it out below that and then some of them do it in meetings and they submit it as bigger merge requests but as we get more familiar with the process I hope to kind of see more and more like team leads contributing we used to do them to the individual level but it wasn't very useful so now we we stop at the at the team level because kind of the team is account accountable for reaching the objective. Yeah that makes sense I'm I'm of the same belief. One of the one of the big benefits I believe in having engineers and engineering teams work directly tied to the business goals and the business outcome of the company and the biggest benefits is it shifts perspective on the value of quote-unquote shipping because getting something out into production in and of itself doesn't provide much value to the company it's really about the value that the customer derives from what was built which in the end will sort of strongly correlate with the long-term success of the business and so one of the benefits that we've seen and in aligning engineering teams behind operational goals is getting something out the door isn't I mean it's an important milestone in and of itself is not the quote-unquote desired outcome for us and this is where some of our OKRs and important metrics come into play is it is being referenced as what we call the success criteria and that success criteria does not actually include shipping the thing the thing shipping the thing is just a milestone and people kind of care beyond code being in production to extract as much value out of what we've built as we possibly can to help the business you know move the right metrics essentially yeah I'm curious to know how do you how do you decouple sort of success or accomplishment if you will in engineers mind with getting code out to production because that tends to be generally celebrated especially as a company grows and scales and your overall scope of work is smaller and smaller percentage of the overall engineering function and what the company is trying to do how do you how have you been able to decouple that if you will that makes sense yeah makes a lot of sense I'm not sure we we totally solved this but one interesting thing we do is one of our values is to iterate so to ship very small updates and another thing in our strategy we have a strategy called breath over depth and that means that when we do something we tend to kind of show where where it's heading and then do the minimal iteration the minimum viable change so right now we're in a phase where get lab used to be version control now it's kind of the most important tools for the developers planning coding and testing that code but what we're doing now is we're going to go to a single application for the whole DevOps lifecycle so that all these operations tasks come with it packaging releasing configuring and monitoring what you may and that's a whole lot of scope and we're adding that in a single year to the product now that's that's a really high pace so on one hand we're telling people like our developers are creating that entire scope but they're doing it with a very minimal first implementation and after that kind of we get those small pieces out the door and then we kind of wait for for feedback we we have open issue trackers and then customers will tell us hey I would like this and sometimes it's users sometimes it's customers many times it's like for example sales people commenting in the issue tracker I have a customer with 300 people here's the Salesforce link and they want they want to see this because XYZ and then it's up to the product managers to kind of prioritize things that people are are waiting for so if we're beyond like the the initial showing the direction and we're kind of filling in filling it in almost always that's based on on kind of customer and user requests okay that makes sense I guess a related question on the OKR front how do you determine what metrics matter from the perspective of being able to influence them so kind of distinguish distinguishing key results from KPIs how do you determine which of the key results actually matter for the engineering team to be able to move I'm not sure there's a I have a cookie because I answer for that it's always it's always kind of hard to to set those key results we try to we try really hard to to pick something that makes sense and that's quantifiable and that's hard to gain and that will really move the business forward but it's always hard to pick the exact metrics is there I'm not sure like a great question though I think it's a great question in consideration Alex and I think it's something we can actually get better at yeah that's one of the things that I always personally struggle with is picking the right target and in our world one of the challenges that we have is you know hacker powered security is not an established industry so we're you know to be extreme are kind of in the polar opposite of building a better mousetrap and so as a result there aren't a lot of kind of standard practices for what it means to build a hacker powered security platform and as a result they're not a lot of acceptable abyss of our consumer e-commerce space you know best practices that we put that would be a little bit of us to to get a speed on but in our case it's not unusual for us to care about certain metrics that wind up being not as important as we originally believed them to be we're constantly trying to iterate on the process and get better at it but as one of our one of our main struggles is when we try to set these success criteria is not always straightforward to figure out what exactly we're trying to shoot for yeah it's I think that's that's a hard thing and by the way your audio is sometimes a bit lagging I think it's your internet connection but what they if I look at your market it's kind of it's a two-sided marketplace what they told us at White Combinator what to look for in a two-sided marketplace is number of messages kind of exchanged kind of number of messages going between the hackers and the companies paying that should probably be one of your key performance indicators hmm do you have any good documentation say from White Combinator that talks about this no I don't I can try to look it's not my it's not my metric I can try to imagine why for example you can also look at revenue and I'm sure you're doing that and that's a really good metric and I'm sure you're already looking at it but for example if there's not for profits which hackers hack for free like that's still a contribution you could also look at like number of cases opened and closed but if there's little interaction between the two parties it's kind of a warning sign that something is kind of going wrong on your platform it's becoming less of like a community it's becoming very transactional it's becoming easier to disrupt so the number of messages is kind of an engagement thing and of course the more programs the more hackers etc then automatically messages will go up as well so you're able to measure both programs, matches hackers and engagement all with one core metric hmm yeah well one of our challenges around engagement which is kind of a more non-traditional to cited marketplaces the fact that hackers could be engaged with customer programs without the customer program ever knowing about it hackers could find a program that they like and spend a much time hacking on it and not necessarily finding anything and therefore there's not much to report and the company as a result never actually knows about it even though there was engagement from the hacker standpoint and therefore when we talk about engagement there's definitely the more obvious metrics around report submissions and things like that that we actively measure and care about but those tend to be more as I think about outcomes of engagement and not directly correlated to engage or it is correlated but not a direct representation of engagement we have certain customers who run very competitive bounty programs bug bounty programs on our platform we draw a lot of engagement not necessarily producing a lot of submissions and it's sort of a high risk high reward for hacker to engage in trying to hack a program with high payouts because the company expects the submissions to be few and far between but a lot of hackers trying and so in those scenarios it's a little harder for us to pin down engagement for any given program based on the its attractiveness, its payout and a lot of other criteria that we don't have visibility on the effect of for a hacker community if that makes sense I see the problem and when you kind of have legacy vendors to give you a nice report even though they didn't find anything they give you that nice reports with zero vulnerabilities that you can send to your customers in this case it's harder and I'm sure you've thought about different ways to kind of capture the attention that hackers do via the log files via their connections and all the other things but I see the problem I don't see a big solution maybe one way I could see is maybe connect those hackers together to kind of if there's like a form environment like what have you tried like I've tried this but it's not working like you can see kind of activity that way but I also know that in that community it's much less common to kind of collaborate and share your methods than in some other communities I've encountered so that might be a challenge there and one of my one of my the outcome that I wanted to produce as part of effectively setting success criteria is more around setting on a an effective methodology or a system to be able to determine valid key results and I expect that given the kind of complexity and lack of an existing space to give us guidance that a lot of our a lot of our metrics that we come up with will be somewhat good approximations not necessarily as ideal as we would like them to be as long as we have some system to both set them consistently enough so that we are at least trending in the right direction but also that we can iterate and improve on a process over time and then being able to scale it out across the entire function so that as we grow and scale that each of the teams can effectively set their own key results following this methodology and hopefully over time we can get to a place where the outcomes of those processes are effective key results that we can all get after which is what the ideal outcome that we're looking for. One thing that we looked at a lot of metrics to kind of look at the effectiveness of development teams like we make software for developments and operations people so like how do you measure their effectiveness and there's a long of a lot of wrong ways to measure it like lines of code written and metrics like that and even things like velocity points and things like that don't really make sense one thing that's very telling and very much harder to game is cycle time so our premise now is that GitLab can make your DevOps cycle 200% faster and that is what is like the time between deciding to do something and seeing the results of that back in revenue or engagement or user happiness and we think that's a very important metric and I'm seeing more and more on the internet when people talk about the security profile of certain companies that they talk about their response time faster able to respond to something and I know that HackerOne is very much trying to help companies to do better there and to drive them towards that in the future maybe if you look at like how secure is a company maybe you look at like how high are the book bounties there awarding how fast do they respond and then maybe an indicator of how fair they are in communicating and awarding those bounties but the response time is becoming the defining metric it's easy to measure it's fair to measure and in our business all the companies are becoming software companies and how fast they can respond is the definition and it's kind of the difference between if you look at a very effective fast moving startup and an old company that hasn't undergone a digital transformation yet most of the time you'll find that it costs months to get some change out the door and every week they can save there they get closer to having motivated people and to being able to follow the rapid changes in the marketplace I would actually love to hear when you mention that the value you provide to customers is you mentioned 200% increase in velocity 200% faster develop cycle so where it used to take you say six weeks now it takes you two weeks between deciding to do something and seeing the results so how do you help customers track that that's one of the other things that we always wrestle with is how do you showcase objective measure of value from this so there's a functionality in gel lab called cycle analytics in it you get the results like on how fast your project is going and what we intend to do is graph that and set it out against another thing the DevOps score how much of gel lab you've embraced and our thesis is that the more of gel lab they embrace the faster they can ship and that's the data points we're starting to collecting and the cycle analytics and the DevOps score is that proprietary score that you've assembled or is it based on some more fundamental first principle like data such as time between X and Y so the cycle analytics output just a time so it's just the average or mean time between saying yes we're going to do this and when did you actually deploy the code and start collecting metrics so that's kind of unambiguous and the DevOps score is more score like gel lab consists of like 10 major parts which parts of those is your project using to what effect and then put a percentage on that okay that's very interesting I think that's something that we could we could definitely do that's awesome I took some notes the idea between that is that within a bigger organization there's multiple projects going on and if you can show that the projects that are more embracing your product have better results then there's a case for the leadership to say okay well everyone's got to start embracing it that like the divisions using Hacker 1 can respond faster to bugs then they might mandate it across the company and for your cycle analytics what scope do you consider at what scope do you consider something to be a project that's worth tracking the cycle of if that makes sense yeah so everything that is a project in GitLab and everything that has a separate code repository is a project in GitLab then we measure the time between creating an issue for it and scheduling it and reviewing it and rolling it out to production got it just for me to get a sense of that scope what is a a good cycle average cycle duration if you will is there some kind of standard that kind of independently verifiable across various organizations or is it sort of like velocity really depending on how much time it takes to start up is shipping in less than a week we're shipping now kind of every month and there's a bunch of companies that's shipping that takes a quarter to get something out the door that makes sense that's really helpful I would love to check out some of the marketing material for it because I think there's a lot we can learn from that cool I'll send it to you and you can also Google GitLab cycle analytics cool those are all the questions I had on my end for me too Barbie no I'm good this was interesting to see how you were doing things there Alex I appreciate your time likewise thanks so much for getting on a call thanks Alex take care