 All right, we'll go ahead and get started. Hello, this is the Practical Guide to Building an Apprenticeship. My name is Megan Tew. I am an engineer at CallRail. And you can find me at Megan Tew on Twitter, GitHub, most internet places. And I have the pleasure of giving this talk with y'all today, because last year I got to do something really awesome. So I never thought I'd be able to do at my company, CallRail. I got to create a program for entry-level developers to bring new people into this industry. It's something I'm really proud of. Thank you. So I love apprenticeships. And I think they're a severely underrated part of our professional systems. And I think they deserve more attention for a few different reasons. So for one, they fulfill a very real need. So with coding bootcamps and it generally being easier to learn to code, there are a ton of junior developers out there. But so rarely do we see actual junior engineering positions. But there are a ton of fantastic eager folks that are just waiting for the opportunity to be seen. But further, when we do see these junior positions, they're asking for things like three years of experience. But there's got to be some kind of role in the middle there. Apprenticeships fill that hole. They also help unify and engage your team, so all of you can rally around the common goal of teaching a new developer who's just excited about learning and growing and being part of your team. They also help us create more diverse and inclusive organizations. So historically, the tech population tends to look all the same, especially in the United States. We have a bit of a white male problem. But more and more, we're seeing juniors that are coming out of code schools or that are self-taught, that are members of these groups underrepresented in tech. And apprenticeships provide these folks the jumpstart that they need into long-lasting engineering careers. So what's an apprenticeship? So it's an entry-level program designed for brand new software engineers to sustainably grow their knowledge, establish a solid foundation for their craft, and learn from their more experienced peers. And for us, the way this manifested was a four-month-long program with the ultimate end goal that our apprentices would be hired on afterward as full-time engineers. And a question I tend to get when I talk about apprenticeships is how they differ from internships. Well, for one, internships tend to focus on a very specific project. You may think, oh, we'll hire an intern to write our API docs. They also tend to have a negative connotation. So internships tend to apply like they'll be doing dirty work, like, oh, don't worry, we'll just make the intern do that stuff. It's not great. Internships also tend to be shorter term, so you might have a college intern that stays with you for a semester and is gone after that short period. Apprenticeships, on the other hand, focus on, instead of doing one thing, they focus on sustainable growth so that the apprentices get exposure to lots of different learning opportunities. This kind of program is investment and them as developers. It also implies you'll be honing their craft. So it's long-term programs, typically. So our program was four months long or three, but some can be greater even than a year. These are longer programs focused on growth, not specific things. So now you want an apprenticeship over an internship, it's time to lay the groundwork. So this is all the pitching, planning, and preparation that goes into actually doing the thing. So what do you need to get it going? Well, you need three things. You need a plan. You need cash, because if you expect someone to dedicate all their time and energy and learning and growing, they shouldn't have to worry about putting food on the table. And you need buy-in. So ideally, you have buy-in from the majority of your engineering organization, but that's not always possible. So at the very least, have enough to where everyone that your apprentices will interact with are on board and ready to go. Most importantly, though, the most you really need buy-in from your boss. And this can range in difficulty depending on the size of your organization. So in for-profit companies, they're likely to need to justify the time and money costs of the program. So you've got to lay it out. In for-profit organizations, it tends to be material with focusing on money. So you'll eliminate your onboarding costs. Instead of spending time teaching them about the organization and getting your new engineers up to speed, that's built into the apprenticeship program. As you're teaching them engineering best practices, they're learning your best practices. They're learning your business domain. It also helps eliminate recruiting costs. So especially in the United States right now, we have pretty hard time hiring engineers, typically. But we have to pay tens of thousands of dollars to have someone do it for us. But with apprenticeships, there is such a huge demand for these entry-level roles that people will be bombarding you with applications for this kind of program, and you won't have to worry about that. It also helps with easier hiring of seniors. So I know for myself as a senior engineer, I want an opportunity to help grow other developers. I want to be able to transfer my knowledge and help grow an organization. And that tends to be common among a lot of senior engineers. So when you have an apprenticeship like this, a program like this, that says, we focus on learning. We focus on growth. Come do those things with us. They're also the material benefits of an apprenticeship. So increasing diversity of background and skill, improving your team's connectivity, and even increasing developer productivity. So if you have a senior engineer pairing with one of your apprentices, those two heads are always better than one. And yes, the apprentice will be learning a lot along the way, but your senior engineer will probably be learning just as much if not more during the process. So once you've convinced your boss and you're ready to start, there are a few things you need to decide. Program length, so how long it's going to run. Number of apprentices you'll have, and then who's going to do what. So these defined roles are actually really important in a program like this. For program length, we decided that on four months, which actually ended up being three, but I'll touch on that later. Number of apprentices, we said we wanted two because that's perfect for pair programming, and it's enough that we can also split them up without taking all the time away from every team. And then who's going to do what. So we decided there would be one program coordinator slash mentor combo being me, have a manager for the apprentices to help them along their way, and then multiple reviewers to help us out with the application review, code challenge review throughout the hiring process. You also have to decide what they should know when they're coming into the program, what they should learn when they're with you, and how they'll learn it. And these things will vary per organization as your values differ. We decided that we wanted our apprentices coming in to know how to build web apps. They should have built a few Rails apps before and that they should know things like basic control foe, just basic programming concepts. We knew we wanted them to learn refactoring and testing and programming best practices, but most especially confidence in themselves as developers and in their skills because that's huge as you're a growing developer. We knew we wanted them to do pair programming because that's a great way to transfer knowledge as well as work with multiple different teams. What's most important during planning is that you really, really need a game plan. So I did a ton of research in order to create a thorough plan leading up to this program and never once did I feel like it wasn't necessary. Every bit of knowledge I was able to absorb from someone else made our program better. And I kept referring back to our master plan throughout the entirety of the program. So this is the actual outline from my plan and includes things like our goals, what they'll cover, what they'll learn, how we'll do it, all those things. And all this planning was really crucial going into implementation. So actually doing it. And we split this into three phases. You've got hiring, onboarding, and growing. So let's talk about hiring. Our goal here was to establish an unbiased, even playing field for our candidates. And this goes back to our overall goal of increasing diversity. And it's so crucial to make sure that everyone has an equal chance in getting a shot in this program. You've got to remove as much bias as possible if you're going to diversify anything ever. So our hiring process had three parts. An application that they submitted, a code challenge, and then a final interview. And it's important to note we had end-to-end an organization throughout this entire process. So when a candidate submitted an application, they were assigned a hash candidate ID. And that was the only way we referred to them until the final interview. And that was key, like we didn't know their name, we didn't know their email, we didn't know anything about them other than what they submitted. That really helped us reduce the bias there. So our application looks a bit like this, you don't need to be able to read it. We told them about us as a company, who we were looking for, gave them a nice schedule of what the process would be. And then we asked them things like, like give us a link to a place where we could view their work, like on GitHub or a personal website. And it's really important to note that we actually never used that until it was tie-breakers going into the final round. So we still didn't know who they were, but we could use that in the final selection right before we were able to talk to them face-to-face. It was a hard process. And then the bulk of the application was questions about them. We asked things like, what are you looking to get out of this program? Things like, what does your background bring to the table? What's really interesting about this question is that specifically in the application, we said we don't want folks that have had professional programming experience before. We want, like you should be brand new to this industry, that's who we want, that's who we're excited about teaching. But despite this and their answers to this question, candidates consistently tried to make up for the fact that they didn't have that technical experience. They were tripping over themselves to say, I'm sorry that I'm new, I'm sorry I'm at the beginning of my career. The problem with lack of junior jobs is so bad that they're willing to say anything to just get in the door and they're so sorry about something they shouldn't be sorry about. We also asked, what parts of programming excite you the most? This question was really cool because we really got to get the sense of who was eager, who was excited to learn, who really wanted to be doing programming. We asked them, what makes you unique? And this question didn't end up being great. Ultimately we wanted to get a sense of like just who folks were maybe outside of tech or whatever, but it didn't end up so great. People maybe ended up still talking about tech to try to really push through. We even got like personally identifiable information in this question. So like someone actually told us their height in that question, not quite what we were looking for so we need to rework that next time. We also asked them to tell us about something that challenged them while they were learning to code and that they ultimately conquered and then how they did it. This was awesome question and it really helped us with the decision-making process because the worst questions we were able to sort of weed out the candidates that didn't necessarily put effort in their answers so like, oh yeah I pushed through and I did it. But the best answer is we were able to see their thought processes and how they like stepped through each problem that they were working on, what websites they read for documentation, how they debugged. This was a fantastic question. So to review these applications, we created a rubric to standardize the review as much as possible and a rubric as much bias as we possibly could. We asked our viewers about where in their learning process do you think they are? Just to get a sense because we didn't necessarily want folks that had never written code before at all but we also didn't want folks that already had full-time engineering jobs. And we had our reviewers rate each application on eagerness and excitement, curiosity, and drive. We rated these on a scale of one to six so there was no easy out by just picking that middle number like we always do, right? But this numerical data was key for us because it allowed us to make some really interesting data-driven hiring decisions. So if we had applications that were consistently scored high among our reviewers, we could just push them through that next round. This person definitely feels like a fit for the program. And on the converse, if they were consistently scored negatively, we're like, okay, well maybe this person isn't a fit for the program. Let's not move into the next round. Our next step is the code challenge. So, making a code challenge. First, we needed to decide what we wanted to know, like what we wanted to test in this challenge. And we knew control flow, we wanted to test for things like, can the candidates write idiomatic ruby? Just things like that. And then you gotta figure out what you want. Or the format that you want. So I decided that a TDD style task would work best here. So we basically gave them a file full of failing RSpec tests and then they had to write code to make those RSpec tests pass. Then you had to actually write the thing, which is way easier said than done if you've never done it. It's very hard to reverse engineer for what you think the candidates would be able to do. And then leave additional room to allow them to go above and beyond and really show their skills. The way we implemented this, so we gave each candidate their own GitHub repository and they could push to the master branch when they were finished. Looks a bit like this. It was for a program called Greater. And Greater helps teachers grade multiple choice tests and then gain valuable insights on their students' performance based on aggregate data. And the review was pretty extensive. We knew based on the coding schools in Atlanta that a lot of our applicants would never have touched automated testing before. And so we wanted to make sure that they had everything that they needed to be successful, that they knew how to run these tests. They knew what it looked like when they were on them so they know they're on the right path. And we gave them an empty class called Greater where they would write their code and we gave them a file full of R-spec tests. So, and we specifically used some different techniques that we thought they wouldn't necessarily be too familiar with so they could learn a bit as they go, like hoisting with lets. And as we gave them these different scenarios, these different tests when setting up the context, we found that it was very key to give them these comments, right? So, if we're, if we've got a method called score that is grading the test and we pass in nil, well, why might I be passing nil? That seems like a very strange answer to have. But if you provide them the context like, oh, that student just forgot to answer that question so they didn't have an answer and Ruby is in nil. Nil is how in Ruby we represent nothing. And that gives them that context, they need to really grasp onto the problem and understand it. Those are the, those comments are the only context that your candidates have because all they have is what you give them so you really need to give them everything they can to be successful. To anonymize, we had, which is, this is really cool, a script to scrub the commit's author info and then submit a pull request to a separate submission's only repository. And these commits were now only associated with that hash candidate ID. And this was key for us because it preserved that commit's history without risking identifying these candidates. So if our candidates were the kind of people that had like all their work in one commit and they pressed that up, we could see that. If they had multiple commits step by step as they worked through the problem, we could see that too. So we could see their thought process. That was great. And this pull request format worked really well because it allowed our reviewers to assess these submissions in a very familiar environment. So at our company, we use the GitHub built-in tools for code review. So reading pull requests really felt natural to the reviewers and it was in, like, they didn't have to worry about learning a different tool to read code to then assess that different code, right? And we standardized the code challenge review in a similar way to the applications. We used a re-book again, asking our reviewers to gauge each application on maybe can they use control flow? Are they comfortable using Ruby? And these were aligned with the goals that we set out at the beginning. Like, we knew we wanted to test for these things. Let's make sure those things are being met. We also asked our reviewers to provide feedback directly to the candidates on their code. We knew that we wanted this program to be an investment in the community's junior developers and many of those developers wouldn't have received code review ever before and that's gonna be really intimidating, especially when you're submitting code for someone else to look at and then maybe hire you based on. So we felt important to give them actionable feedback so they can learn and grow even if they didn't necessarily get selected for the program. So after the code challenge is the final interview and this is the first time that we've been able to put names and faces to their answers, which is really cool. Our final interviews consisted of three parts, a product and company intro so they could get a sense of who we are and what we do a little bit better. And then a technical conversation and a behavioral chat. And the technical chat is quite different than it would normally be for experienced engineers because you can't really use the same terms or ask the same questions as you normally would because you can't guarantee that the person you're talking with is gonna know exactly what you're talking about. And then there's the disconnect that makes it just awkward for everybody involved, right? So instead, we used the code challenge as a focal point of that technical discussion because that was code that we had both seen and that we could talk about easily. And that enabled us to ask things like, what challenge do you, while you were able to complete this task? So, and that let us see how they were able to work through problems and then how it was then reflected in their submission. We could see what their code looked like versus the challenges that they said they faced. And then we used the feedback that the reviewers gave in the rubric to give them in the interview. And so this allowed us to see how they process feedback in real time, how they asked questions to better understand the problems that they're faced. It was pretty great for us. So our behavioral chat was a bit more like standard job interview questions. But one of the things that we got, unique things we got to ask them was, what would you change about the school you attended? Be it a code school or a college or your self-taught like curriculum. This was fantastic because we were able to get a real idea of like what their learning needs were so we can then assess if they were a fit for the program and then if they were like, how better to match what exactly that they need, how we can best serve them going into the program. So ways to improve hiring, I would really recommend to use an HR platform. We, at this time in the company, we didn't have one yet. And so it ended up being a mismatch of like Google Forms and Sheets and Docs. And it was kind of a mess. It worked for us, but that time used to coordinate all those different Google Suite products was probably better spent like actually in the program. I would also recommend not limiting your code challenge to one programming language. So at CallRail, all our server-side application code is in Ruby and as a Rubyist, that's all I knew. And so I didn't feel comfortable testing someone on code that maybe I wouldn't have been able to write myself. And so, but we had a bunch of great applicants that either came from JavaScript backgrounds or PHP backgrounds, and once they got to the code challenge, we're maybe a little too challenged in having to learn Ruby and then write Ruby for this challenge. And so I think we really missed out on some great candidates that we could have had if only we were able to write in JavaScript maybe. So definitely try to branch out into more of the one language. Also, make sure to be very direct with the application questions that you're asking. So your candidates are just trying to answer the question that you have asked them and they're not trying to deduce some hidden meeting behind the scenes. So ask directly what you want to know from them. Now that you've hired some fantastic apprentices, it's time to onboard them. So our goal is here to set the apprentices up for success and teach them the essential concepts that they need going through the program and later throughout their careers. We really wanted to give our apprentices the same onboarding experience as our regular developers so that they would be on the same page with like that common knowledge set and they still knew what everything was in our system going forward. This includes for us things like info sessions on our different applications and the important parts of our code base. One apprentice like specific thing we did during onboarding was a scavenger hunt. And this was really cool. We basically gave them most of questions like tell me the name of the controller action that does this thing. And then they had to work backwards based on the context clues and the questions to figure out the answer. And this was fantastic because it really helped them become more familiar with this large production code base, the largest they had seen yet and but in like a very low key chill way. It was nice. And in addition to those regular onboarding info sessions, we also gave them specific lessons on more foundational concepts like object-oriented programming and debugging. These are things we typically expect that our more experienced engineers will have going in but we didn't expect of our apprentices. So we wanted to make sure that they had the right stuff upfront. Waste to improve, I'd recommend giving them early exposure to any practices that your team has with things like sprint planning or point estimation. About halfway through the program, I had an apprentice ask me what a sprint was. And that to us seems almost obvious if you've been doing this for a while but to someone that's just coming in that of course they don't know what a sprint is, right? And because of the model that we chose, they didn't necessarily get exposure to being in sprint planning early on but those things are important to our teams. So definitely do that earlier rather than later. Once you've onboarded your fantastic apprentices, it's time to grow them. So there's a lot of info out there about engineering leadership that you can get elsewhere including some of the previous wonderful talks in this track but I'm just gonna talk about like apprentice specifics. So for us we really want to be able to anticipate their needs, provide an inclusive and friendly environment for them to grow and learn in and increase apprentice independence because I believe that independence is one of the key markers of experience in an engineer and we wanted to help them grow on that path, be more independent. So our growth period consisted of and first an initial ramping up period where they were working on like small projects. Maybe they were fixing a bug or resolving an error or making a small CSS tweak but they worked up from there and this was so that they could build up not only their skills but confidence in their skills. They could see themselves getting better and doing more complex work. The second phase were team rotations with our different development teams that were working on customer facing product and this was fantastic because it gave them exposure to all different levels of the stack, different parts of the code base and even different ways of running teams. And this ramping up really showed them how the larger projects that they were doing later on in the program were made up of the smaller ones that they were doing early on. We also made sure to set early expectations for them. So when they came in, they had a roll sheet like this. Software engineer one is our equivalent of junior engineer and we said to be hired on afterwards, which is our goal, please try to hit most if not all of these boxes. These include things like personally verify that your changes work is intended and lean on the expertise of your coworkers to improve your work. We also read this book, Book Club Style, Apprenticeship Patterns by Dave Hoover and Adelaide Oceanier, it's one of my favorites. And this is a fantastic book that gives newer developers a better way to view and intentionally craft their careers as engineers in the longer term. And it gave them a common vocabulary to talk about their progress and experiences with me as their mentor and as well as each other. We also made sure to establish regular communication through one-on-ones. So I, as their mentor, met with them each week and their manager met with them on a regular basis so that they could stay on track and keep updated with their progress. We also felt it important to include them in their team rotation decisions because your apprentices will learn and grow best if they're doing the things that they actually enjoy. One of the goals of the program was to actually help them figure that out. What do they enjoy? Do they prefer front end? Do they prefer back end? Do they wanna do a little bit of all of it? And so by letting them sort of follow their own path, they were able to actually find those passions. We also wanted to give them unstructured time to learn freely because this is such a learning-focused role more so than any of our other engineering roles. We felt it important that they could be free to explore what interests them so they could truly make the curriculum and the program theirs. And most importantly, we were not afraid to let them fail. So there will absolutely come a time when you're sitting down with an apprentice and you're paring and they say, oh, what if we try this? And you think, hmm, well, that will definitely fail. But you have to shut your mouth and just say, yeah, let's give that a shot because the most important way to learn, like the most crucial thing for them is to find out what that error message is, like why that happened and then how they resolve it. And you don't get that by just telling them, oh, let's not try it. You've got to see the actual result of their thoughts. This is really key. So ways to improve the growth process. Make sure to look for early warning signs and be proactive about intervening. So junior developers tend to have a harder time admitting that they're struggling, often because they don't feel like they truly own their role yet. They don't feel like developers yet. They're still trying to prove themselves even though they've already proven themselves. They're here in the program. So make sure you're able to recognize signs of I need help and be able to support them in afterwards. Also wanna make sure that everyone involved in the program is on the same page regarding their expectations. So we had a situation where the team leads didn't necessarily know what the apprentices knew as they were rotating on their team or what they should be teaching them. And the apprentices didn't necessarily know what they were supposed to be learning. So this is actually not too hard of a fix. We create a rubric that everyone keeps track of as you go through the program, keeps up to date so that you know they're hitting all the boxes. It's a pretty easy fix, but it's really important in tracking growth and showing the apprentices that they're on the right track. So we will be taking all these ways to improve and the lessons we've learned into our next round of the program this year where we start this process all over again. And to wrap up, we created a master plan for our program. We used a fair, anonymous hiring process to hire fantastic apprentices. We set them up for success throughout onboarding and we grew them into fantastic software engineers. Our results with our program, I'm really happy to say we hired on both of our apprentices afterwards. They've been with us since September now, so our program ended up being three months, four months after, about seven months total. And they are both productive members of their teams. Their teams wouldn't wanna be without them at all. They've been doing fantastic and really proud of them. So some bigger lessons we've learned throughout the apprenticeship creation process. When you're hiring, make sure to listen to early hunches you have about candidates. We had one candidate that submitted an application and they seemed potentially overconfident in a way that could maybe hinder collaboration, but the rest of their application was really strong. So we pushed them through to the code challenge and their code challenge was really strong too. So we pushed them through the final interview, but talking to them face to face, we realized that that initial hunch turned out to be true. And I believe that egos don't really have a place in teamwork or collaboration because they kind of get in the way of having an equal shot. So make sure to go with your gut in these decisions. Also wanna make sure to give yourself a lot of time for this process. So our program was greenlit in June last year and we sat down to plan and realized that in order to end before Christmas, we couldn't have a four month long program. We had to have a three month long program because this does take a long time. Not only to have them there, but the lead up time making sure people could submit two week notices beforehand and hiring and all the planning. It does take a while, but it's very, very worth it. And finally, always, always work with your goals in mind. It will be very tempting to do things in an easier way or the way someone like asks you to do them, but you've gotta remember that you're in this to make a very real impact. With this program, I'm happy to report we provided W2 jobs that paid a living wage with full benefits, which is not typical at all for entry level positions. But these kinds of jobs are the kind of small change that can become something bigger. This is our opportunity to make a big change on a small scale. In tech, we often say that we want to bring different kinds of people into this industry. We say we wanna value education and growth in our organizations. And we say we wanna help junior developers. We wanna give back, right? This is a concrete way to do all of those things. So go start entrepreneurship. Again, I'm Megan Tiu. I'm at Megan Tiu on Twitter and GitHub. I wanna thank all of these fantastic people for helping me either during the research phase or actual implementation of the program. I will be taking questions off stage afterwards and I'll be around the whole conference and I can talk about apprenticeships all day. So please come chat. Thank you.