 Welcome. Thank you for joining us today. I really appreciate the folks that are here in attendance. Thanks a lot. We'd like to talk to you about Drupal as the basis for tech education and to give you a playbook around running an internship at your company based you know around what we get in the in the Drupal world. We did this over the last 14 weeks and we just hired six amazing people. Meet the graduating class of Bill Design 2017 our multidisciplinary internship program out of a class of 12. These six rose to the top and were hired at phase two exactly one week ago so we're cutting it pretty close on the timing of all of this and we really want to tell you about it. All six started as junior talent and all six are available today. Hi my name is Christopher Bloom and I'm front-end lead at phase two technology. I'm Caitlin Loos. I manage our creative services department. Our third partner in crime here Chris Wright is out celebrating his birthday as part of the team. So Bloom and I are an unlikely pair. He's on the technical side. I'm on the creative side and what brings us together here is that we're teachers and mentors first. We know that it's the culture of teaching and mentorship can transform organizations and that's why we're here. So why do we care about junior talent in the first place? We want to constantly infuse our workforce with energy and diversity that comes from the next generation. It's a really good financial decision. We get great value from junior talent and as a workforce model it just makes sense to have your senior people doing things that make sense for them and your junior people doing things that make sense for them and everybody's learning at a level that's appropriate for them. So companies will age out if you're not able to replenish with younger talent and new talent. Your workforce will get older, more expensive and you'll miss out on fresh new ideas. So we're all here at DrupalCon and we all got here from very different paths. I'm going to ask you a couple of questions. How many people in this room came from, say, a computer science or engineering background? All right. How many people arrived to Drupal and DrupalCon from, say, liberal arts background like English, philosophy, history? Interesting. All right. How many people with no school or not a lot of school? Interesting. All right. So, you know, we all took a pretty different approach to get here and this always gets me thinking, like, what if there was a consistent program that allowed, that provided more guidance to bring people into the Drupal world? So first a clarification about internships. An internship is a very specific kind of employment defined by the Department of Labor. Specifically, we want to point out the internship experience is for the benefit of the intern. This is really core to our philosophy on this. An internship is not hiring somebody for a short-term project. An internship is not giving education through exposure alone without a real defined training. It's definitely not cheap labor when you count all the costs of running an internship in your program and it's not coffee runners and gophers. Real quick. I want to jump in because that quote right there about the internship experience is for the benefit of the intern. That's from the federal government, right? That's not our kind of fluffy approach to this. Like, that's actual guidelines from the Department of Labor. So we're going to get into the details of our program, but we want to point out just a few key things here. I'm not going to read all this stuff, but we call our program Build Design and that's because it encompasses the wide breadth of what we do in agency life. A couple things to call out. Paid. We highly encourage that your internships are paid for a couple of reasons. The number one reason is we don't want to just hire people who can afford to work for free. This is a major component in recruiting when you're talking about diversity, when you're talking about people at different points in their career. Second is that working for free is really hard for anyone. So if you focus, if you do an unpaid internship, the focus is on the hire at the end, not being present during the program and having an enriched experience during the program. And finally, from the slide before, it's actually really hard to have something qualify as a true internship and there's a lot of paperwork and whatnot and you avoid all that if you pay your interns. So we highly recommend you pay your interns. Second one I want to call out is real client experience. At the end of the program, we did a survey of our interns of what was the most important, what were the most important benefits to them, including being paid and above all else was being on real client work. This is really, really hard to make it happen. We're going to get into some details about ways to make that work. The third one is a real defined curriculum. Much more to come on this, but this component is really key for getting junior talent to the place that we need them to be, to become billiable, to go on to delivery projects and to really minimize the risk of what junior talent might or might not be able to bring in. So I want to talk about our grassroots approach to kind of making this happen in an organization. And the first image there in 2015 was the first iteration of this. That's two design interns under Caitlin. We saw some success there. So we brought it forward in 2016 and we added developers on that on that crew and we had six total interns and we learned a lot. We tweaked some things and we tried it this year in 2017 up to 12 total interns, right? That's a pretty big growth and we made a lot of changes incrementally between each. Now, your grassroots, the way you approach this might be different. This is our experience and maybe that can help. This is us pitching the idea a couple of years ago in the company, right? Let them know. Let them know what you're doing, right? Tell them what you're planning and tell them what your metrics are so that you can be measured on the success of the success of the program. We wanted to start small and really serve a specific discreet need. By that I mean like hiring a developer and a designer specifically. So you've got something to compare against. You need to be able to show results each time so that you can justify growth from one iteration of the program to the next. This is a little controversial, but we kind of had to work for free in this first version. What I mean by that is we just did this on top of the day job, right? It might be similar. There might be different mechanisms within your organization, but you need to maybe be prepared to kind of take that on. You need to start small also specifically to manage risk. Whenever there's people and organizations involved, there's risk, right? Contain that. It makes making the business case a lot easier if you can talk about the risk. And then just be relentless about constantly improving. Get that feedback. Feedback is hard and it's really easy to get feelings hurt. It might have definitely been a few times on this, but you've got to hear it right to be better. So just to walk you through a little bit of what this grassroots model might look like, this is what it looked like for us. So the first round is very grassroots gorilla innovation. We work for free to get it done. Year one, we had two interns and we got 16 employees in the company engaged. So before you start thinking that this is new people, you have to hire. These are people already in your company, already serving important functions in your company, just bringing them in to get engaged in this program. That's not 20 interns. No, definitely not 20 interns. So at any scale, these are the main people that you need to have involved to make this work. So first is executive support. You need an executive champion who can help you navigate the inner workings of your organization and make this happen. Second is HR. So from recruiting to hiring, performance management, you really need to get this stuff right. You need a great advocate on your HR team. Next is program directors. These are the people who are kind of overseeing the entire program. If you're here and you're passionate and want to start it, this will probably be you. Mentors, that critical one-to-one relationship between mentors and interns. You got to have some great mentors in your team. And then finally, teachers. This is the way to really get a lot of people involved in your company with a low investment. So they come in, they talk about what they know and make themselves available to the interns to answer questions. And then you have more people involved and aware and engaged in the program. So year two, this is what it looked like for us. Everything scales up a little bit. We get a little bit more support. We're able to make a case for more hours and resources. The case becomes a little bit easier. We add to the mentor pool. So having one mentor at least for each intern is really, really critical here. So we want to really build up that mentor pool. And finally, this is, well hopefully not finally, next. This is year three. So this is widespread buy-in after proven results. At this point, we've already hired five people as full-time employees from the internship program. We all get hours booked and whatnot. The case is easier to make. The team gets much bigger. 60 employees engaged, which is really exciting. These are all people who are participating in a culture of mentorship and teaching. I want to call out a couple of people here on our side who have been really, really important. Our CEO, Jeff Walpole, has been supportive of the program since the first iteration and really helped us navigate the inner workings of phase two to make it happen. Second is Jamie Steiders here with us today. She's our VP of people. And she came in this year with amazing advice and leadership to really help us mature and optimize the program, which we really needed to do at this at this stage. So find your advocates at the executive level because you're going to need them. So let's talk about recruiting to get into this process a little bit. And I wish I had some really great silver bullet to make recruiting a lot easier for you. But you're going to interview. It's hard and it's tedious and it takes a lot of time. To put in perspective, here's some some numbers of our total pipeline of recruits. And I want you to look at that big 60 right there because that's the number of hours we did in the month of December to finally land on the cohort that we have. This is for this recent year, this recent 12 interns, 60 hours. And that's that really ensures like you get to know who's coming in, you get to know them well, you can know the capabilities and where they want to go and if they're a good fit. That early front load the work here I think is is a really important thing. Other things to really consider around recruiting is you got to show up, right? You got to get out to the community events. You got to go to the code schools that are nearby. You've got to participate, not just recruit. When I find being able to go out and you know, present and give back to the community, speaks a lot to the organization that interns might get an opportunity at. You're looking for culture ads here, not just culture fits. Because if you're always just looking for a culture fit, you're going to get the same thing over and over and over again. So really, really look for the enthusiasm and really look for not just the technical skill. Pay attention to the wording of your of your listings. Look for really heavily gendered language. Look for constructing descriptions of your program that are as inclusive as possible. Bring a lot of people to the table. This is something that's kind of overlooked. And this is why HR is awesome. I suck at HR and thank you HR for what you do. All right. So here's just some hiring tips for those that are considering starting their own internship program. These are things I've learned after 60 hours of interviews in December and many, many, many interviews. If you're pulling from a pool of code schools, and I know we have this really huge surge of code schools out there in the last five years, I think it's a really good thing. If you can assume a technical baseline, that means you can be looking for other factors in humans beyond just do you know what a Drupal hook is, right? This is really important to us. What was important to us might be different for for your organization, but we were looking for passion, drive and collaboration, right? Because anybody with enough enough enthusiasm can learn this stuff, right? I like to say energy plus time equals awesome, right? We really wanted to feed into into that formula. A trick I use this one easy trick is when you're interviewing folks that are coming out of code schools, get familiar with the curriculum of the code school, go to the applicants, get her profile and remove the assignments from the profile. What's left is kind of an insight into what they're interested in, what they're playing with and what they're doing on their own time. So what it does is that you see what they do as part of the curriculum, but you also get to see maybe what they are doing on their own and it tells you a lot about where they're going as a developer specifically. I ask a question. The would you do this for fun on a Friday night? I don't mean would you work for free on a Friday night? What I mean is would you do you find this enjoyable? Are you playing with triple or JavaScript or anything? Are you getting that kind of spark where you want to do this on your own? That's what we're looking for. We're looking for that passion. So especially in a highly skilled environment, finding junior talent is really, really hard. We've heard a lot of companies say we only hire senior developers and we think there's a way to crack that net, but you really have to understand how junior talent is different. So first, not all superstars are born that way. A lot of times inexperience and lack of confidence mask really amazing people. So you really need to give them a chance to show their brilliance. Second is when you're hiring for non junior staff, you have a checklist of skills you need and experience you need and with junior people, they oftentimes don't have that. So we're looking for potential here and we also really need to give them that experience, expose them to the environment and see how they react and respond. What comes along with that is you need time, right? You need that. That's why we do our program for 14 weeks. You need time to expose them to all these different things, see how they respond and react, be able to help them where they're weak and especially guide them where they're strong, which leads to the point, be flexible and where they land. So a lot of what we do is very related. The skill sets are very related. When you expose them to everything that we do, they may find a different course that makes more sense for them. And that's a really good thing to identify that early. One of our interns from last year actually Janine came in from design design school and she went through the program, got exposed to the more technical side and she's now one of our product managers, which is a great path for her. So the main message here is really you get back what you get. Let's talk about junior talent as it's applying to your company for your program. There's a phenomenon I like to call the code school filter. I really like the schools. I think they're a really good thing. I think it's a really great alternative path for a lot of folks. What I mean by this is the folks that are coming from code schools to your application process, they've already demonstrated a level of commitment and a level of ability to stick with it. Often what we're dealing with with new junior people is all that uncertainty. Like, can you actually do this? Are you going to stick around or are you just going to flame out? When you see people make it through and graduate from these things, you get a base level of technical competency, which we talked about earlier, but you also have an expression of I'm in it, right? I'm here. This isn't my first step. This is my third or fourth step. It really lets inexperienced people also early identify where they might want to go, what kind of development they're more interested in. So you get clarification a lot earlier on if it's going to be a good fit. So I really like code schools. I'm going to give a big shout out to one in Portland called Epicotus. They have a Drupal track, which is rare, right? Most of them are Rails programs or purely Node, but Epicotus actually has a full Lampstack Drupal track, which does a really great job of ensuring that we've got a really good pool to pull from. Okay, so your recruiting is done and you're ready to start. There are a few things you really should keep in mind, some kind of reality checks when you're when you're considering a program. First, this isn't your main job. We're talking side hustle, right? You got to keep, you got to find a way to be able to do this along with your main job. Second is your interns are self motivated and smart. So trust them and really give them a chance. Third is define a program timeline and number of hours that works for you. And I'll be the first to admit we haven't practiced that yet. We've tried different things every single time and it really depends on your needs as an organization. So in terms of the program duration, you want enough time that they can get that experience. You can evaluate them, but you don't want it to go so long that they're like, okay, already, I'm ready to start this thing. And then the number of hours per week as well, you need enough time for them to be able to get something done and accomplish what they need to, but not so many hours that you're spending way too much time managing them. We only did on 16 this time. We think it's a little light. So good luck with that one. If you figured out, make sure interns have the materials that they need that they have laptops and give them the ability to install things on their machines. We find out some internships actually don't let interns install stuff. And we feel that kind of hinders a lot of actual learning and progress, making part of your team. If you're doing this internship in person, highly recommend a physical space for them to be in, especially in the beginning. It does a couple of things. One really helps them form a great team bond, helps them learn to be good team members with each other and then translates to other project teams. The second team is they learn to help themselves and be a collaborative group which really cuts down on your time and instructing them on some of the easy stuff. Okay, so this is kind of an outline of the program structure that we had this time around. So we're going to go into each of these in a little bit more detail, but I just want to kind of call out the high level here. So we did our program for 14 weeks. That's down from the last time when we did 18 weeks. It was a little more time than we needed. So the first thing is self study Trello board. Bloom's going to go deep into detail on this to show you. But essentially, get them going on Trello, get them really self sufficient on working through a lot of the material that they need. Second is guest lectures from people in your company. So have people come up and talk about what they do. This is such incredibly valuable information to your interns. Third is very quickly on we introduced the advisor network. These are people who are available to the interns. They're not their mentors but they're available to the intern, their subject matter experts so they can schedule time with them, talk to them about, you know, all the things that they're interested in. When you're an intern, it's really hard to reach out to senior people and say, I'm curious about this. Can you tell me more? This is a group of people who have given their blanket permission to do that. The other thing about advisors is that they're there for the mentors as well. So if you're a mentor and you suddenly have a project on fire and you just need you need to focus on that. The advisors can step in and help mentor your interns for a little while. It's also great for young mentors and new managers. They can go to the advisor network to get help on managing their interns. So they're kind of doing both of those things. A little bit deeper into the program, we introduce the mentors to the interns and start pairing them and very quickly after that, start them on actual client work. As much as possible, we try to pair the interns with the same project that their mentors on. Doesn't always work out. This year, we added two new things, the innovation pitches in the Acquia Builder exam. Innovation pitches was really trying to capitalize on the new ideas and energy of the interns. And we asked a simple question what should we do to improve phase two? And we gave them all 10 minutes to present to some senior leadership and got some great ideas. And then the Acquia Builder exam really helps us measure their technical capabilities. So that's kind of the high level. All right. Let's talk some of the meat and potatoes of the curriculum. This is my favorite part because I get to play with Trello and I love organizing and setting this stuff up. So I have an instructional background and I found that when I teach my classes around web design, this collection of bite-sized concepts is really, really helpful. The goal here is to allow for self-study. We really want this group of cohort to sit together, work together, and work through a lot of concepts. And yeah, go grab Trello. This is there's a there was a theme throughout our program of good, not great. We made it perfect. It would be many full-time jobs to actually get it honed exactly right. So get good enough, right? Trello is one of these things. A lot of these little bite-sized concepts work well in the Trello card space, right? So what we do is we take all of this stuff. I wish someone would have told me when I started, right? I had to screw it up a ton before I figured it out because there isn't really a way for someone to tell you everything that's involved in the development kit, right? So what we do is we take this board and our first column is info. It's just information, right? It's a really quick place where all the interns can go for canonical information. The second column it's kind of blurry now. That is what we call the icebox and it's a collection of ideas from around the company. Everyone is invited to just start throwing things in there. That might be good. Things that you got to know. Bash tips and tricks that various get things. All of these kind of things that, you know, there might not be an official thing for it, but there might be a great article that someone should go read and we just dump them into that icebox. Including the interns. The interns added a lot of cards to this section, which is great because they're improving it. Exactly. A lot of times we just assume they might know something like, there's the board. And then every column after that is just a single week of the program. We dump all the cards into the first week. The cards that are not finished, that are not completed. They just get moved to the next week. The cards from that week that are not completed just get moved to the next week. And what you end up with is a self documenting record and an instant at a glance visual of where you're at in the program. You'll see this triangle form as they're completing these cards. This is an example of what one of the cards might look like. And this is something as simple and really we go this granular of set up Google Drive locally, right? There is an assignment and there is a deliverable. And it's very easy to know what you should be reading, what you should be looking into, and then how are you going to report back? We generally have our interns just comment. They comment with the link on the card. It's very easy. We assign every intern to every card. They take themselves off the card when they're finished. We know at a glance who hasn't done this yet. So you're going to notice a number of patterns that are going to arise out of your curriculum. Here, it's pretty obvious, use that first set of columns for your onboarding. Self-study, all your onboarding is very tedious, can be very tedious. Some companies have really great handbooks. Some companies have really gigantic wikis, which may or may not be updated. This is one of those things where you can just kind of get it all in one space. It's always in front of people's eyes so it can always be improved. Allow for the weeks in the middle for self-study. Yes, I'm actually saying have people in your company who are sitting down and just learning, just learning, right, together as a group, as a team. And then ultimately, we want to taper up into delivery as we get interns put on real client work. OK, so they move out of the self-study curriculum phase and they move into actual delivery for clients. These are just more general concepts that might help setting up curriculum. Teaching in the past, I found that this is a helpful approach, starting on front-end concepts and moving to back-end concepts. What I mean by that is there's a lot of things in the browser that inform the contraptions and structures that we have set up on the back-end. Usually, this is for the work we do is for display, for consumption on the front-end. Also, hey, front-end, that's way easier to do than back-end, right, looking at Chief Frank. So there's a lot of really, really good tools here. We'll start with really specific tooling like pattern lab in the front-end and then move to very deep Drupal API stuff for back-end training. To speak to that a little bit more, you're going to notice patterns of general academic concepts and then specific implementations of those concepts in your company. I'll give you an example. Early on, you're going to want to maybe talk about Git branching and all of the different types of Git workflows, very, very purely not specific of what your company does. You'd be shocked at how many people don't actually know a lot about Git, right? I certainly still don't. It's been many years. But if you can approach this with a kind of an academic look at Git, Git branching, various kind of technology around that, and then later on, you say, this is how we use Git. This is our polar cross model. This is very specifically how we name things and why. That way, you don't end up with a cohort of folks that kind of really only know what the way you do. And I think that kind of helps us be more holistic developers. I'm not going to read every one of these points about Drupal, but that is the title of this talk, right? Drupal as tech education. And for all of its works and its size and its complexity, I actually love Drupal as the stack to learn around, because you have to learn so much. Like, there's an entire university that you could put around Drupal because it touches all of these things. You've got DevTools, right? All your command line stuff we just mentioned Git. We've got all of the JavaScript concepts that are in core and evolving constantly. We've got a lot of markup decisions, right? Theming. I only have three little points here, but my entire job is one of those bullet points, right? It's crazy. And there's all of this other ecosystem. There's all of, of course, what we generally think of when we think of Drupal development, which is the back end. That's PHP, and there's a lot of stuff there. And then kind of the rise of DevOps and a developer being expected to be very, very flexible in a lot of different areas. There's a lot of work around what we do in Vagrant and Docker, for instance. So I think, you know, Drupal acts as a really good base. It's a good space to work in, but there are edges to it, right? It's the lamp stack. We know we can kind of like touch both of those walls. Oh, last thing real quick on Drupal. In your curriculum, you're generally going to see not a lot of Drupal specifics, and then you're going to ramp up into a lot more Drupal-focused stuff before you eventually move to delivery with your interns. I say you, like, you're going to do this. This is what we did, and this is just kind of what naturally organically arose from our approach. As you see the cards drop off, you see the delivery onboarding, and hopefully we're at a Drupal point where they're actually able to deliver that. One more thing on Drupal. There is incredible learning resources available. Of course, we're all here for Drupal. We're here for the community. A lot of folks entering the Drupal world are kind of shocked to find out how people focused it is for better or worse. But that's what's amazing about it. There's so many resources, groups.drupal.org and IRC. I'm going to kind of shout out to the Acquia cert, the builder test specifically. This is a non-programming Acquia test, and it's kind of wacky, and it's actually a little scary even for seasoned developers, and I really, really like having everyone take it, the developers, and the product managers that are in the program. It shows a lot of nooks and crannies. There's really good study material. It's an excellent opportunity to get all the interns together and study really intensely about Drupal versus, yeah, go watch a bunch of videos. It's a goal and it's something they can focus on. And of course, my favorite thing, I love online tutorials. I love sharing online tutorials. I've broken the bank at Udemy recently. But Drupalize.me, Build a Module, DrupalEasy.com, all really great resources, and I believe we have a Drupalize.me account for every one of our interns. Really helps them fill in the gaps if there's a lot of questions they might have. You can generally find an answer at Drupalize. Also, just send them to all the meetups they can. In Portland we have three different Drupal meetups a month, right? Great chance for them to meet folks. We're really going to involve them in the community and free beer and pizza too, it doesn't hurt. Okay, so this is our last slide on Trello, but just wanted to point out one more thing, which is the instructional involvement really required for the different points in the internship. So when you start out, they're getting onboarded, of course they need a lot of support. So there's a high level there. Then we dip down when they're into the self-study mode. So this is when they're really going on their own on the Trello cards. We ramp back up when it's time to move on to client projects, and then it tapers off as they again learn that world and get through that learning curve. Two things to point out here is that at the different peaks, there are different people who are taking on the heaviest load here. So at the beginning, it's the program directors who are kind of overseeing the entire thing. When we get up to the next point, it's the mentors. So they're doing that really heavy one-on-one work with their interns to get them onto the client projects. In that dip right there, that's when we train our mentors. So this is really, really important to get your mentors trained to be great managers to their interns. We used to try to do this before the program started and it was so hard with so many different details going on at the same time. So now we take advantage of that time when they're doing a lot of self-study to really train our mentors. Okay, so moving on to the next line from the Gantt chart, we wanna talk about our company specialist presentation. So you have a lot of really smart people at your company, have them come up and talk about what they do to the interns. A wide breadth of really understanding everything that goes on in your environment is super important. It's a really great way to get your staff engaged in this and it's a low level of effort for them to present on what they do. So this picture is actually from our second round of internship and at the table we have designers and developers, literally hand drawing logos on a class of Design 101 and the teachers of this class are the two designers that we hired from the first round of the internship. So scary collaboration, it's a good thing. Also we recommend opening these up to the whole company. So there's a lot of introductory material but do you know exactly what everybody at your company does on a detailed level? Probably not, so go to their talks and find out what they do. If you have any kind of structure in your company for knowledge sharing, try to take it over for a little while with intern classes. We have a collaboration lunch every Wednesday and we took it over for two months and had intern courses, time that's already on everybody's calendars and we got a lot more participation in that way. A lot of these become onboarding materials too. Anybody new to your company needs to know these things so record them in their great onboarding materials later. Just to give you kind of a sense of the breadth that we covered, product design and management to sales, the anatomy of a pitch in a proposal and negotiating skills and things like that. Developing a pattern lab obviously, documentation and it's definitely not a boring class, which is great. Fundamentals with aesthetics, Design 101, communications, all of these things. So we have a very wide, well-rounded education and it really exposes them to everything they're gonna need and everything they might be interested in in an agency environment. The last thing to point out here is that we have the two columns, right? We have 2015, 2016. So we're trying to transfer as much as possible from one year to the next so we're growing our curriculum and not having to reinvent the wheel every time. So use the old deck, have the same teacher come back and get better at teaching that same course. All right, let's talk about onboarding interns to client work, right? This is kind of I think the ultimate goal. The feedback we got from the interns was this was the single most important thing of the program was doing it for real and being allowed the trust and the responsibility to do it for real. This is really hard. This is where you enter that minefield of contracts and business and relationships and timelines and deadlines and like a lot of things, there's no magic way to do this, but really ensure that when you're onboarding interns to client work, you're being very transparent, you're being very communicative about it. You're not gonna not tell a client to cut interns working on the project. Be clear about that. But also if you've got good structures that allow safe failure, you're fine. And what I mean by that is putting, say, a gatekeeper between the interns and the stakeholder at the client have a very clear code reviewer that is vetting all work before it even goes up for full request. Have that gatekeeper even maybe access, things like JIRA and communicate that down to the clients because onboarding interns into client systems can be really kind of tough and awkward and slow and overhead and often it's best to just be like, you know what, we're just gonna have a representative of the company, perhaps your mentor, if it's available, be that portal between those two worlds. The controversial statement we're gonna make on this slide is not forcing billability. I know there are internship programs out there that at the end they say, ah, you know, we're gonna have everyone billable as QA. We're gonna have everyone billable as, you know, content entry or something like that. I, we don't think that works. I think by the time you're negotiating contracts and you're talking about extra hours that have to be paid for by a client, they say, well, why do I, you know, why am I gonna pay for this if they're not necessarily senior developers? It becomes so much work around discussing it that by the time you actually have an intern on a project, it might be too late. So this is one of the takeaways that we'd like to give to you and that is just don't try it, or try it, but don't come down hard on it, don't enforce it. So, and then what we find is we're able to get interns into actual development roles on projects, right? They're not here to necessarily do content entry, right? They went to a code school, they're looking to get their hands dirty. This allows them to code for clients without, yeah, without a lot of friction. Speaking of clients, I'd actually like to ask Ashley Feginelli from Weight Watchers to come on up and talk to us. I'm on the Weight Watchers team and Ashley is our product owner there. We replaced a couple of interns with Ashley and honestly, I'd like to ask you, what made you comfortable working with the interns on the project and what was your overall experience with the interns? Hi, everyone. So the first time Chris approached me about having interns on my project, we were up against a seemingly impossible deadline and the number of things I had to accomplish in that deadline kept increasing. So the idea of having additional hands on the project seemed like a no-brainer and this year when Chris approached me again, I got having interns from phase two, it had already been successful last time and when I mentioned to my manager that we were gonna get interns again from phase two, they liked to be in there like, we had interns last time. May have forgotten to mention that to my upline, but it worked out really well and the interns were incredibly passionate. They were really curious about the company and what we do and how it works and specifically what my role was and I got to speak to them about that as well. Thank you so much. So and Ashley's willing to answer your questions at the end of this too from the client perspective, so keep track of that. Weightwatchers.com is still online, so obviously it went pretty well, so. Awesome, so we're gonna talk about a critical component of any employment, which is performance evaluations. So this is one of the places where HR is really, really critical to help you identify the competencies that really, really matter to your organization. For us, we have five of them. Note one of them is technical ability out of five, so we're really looking for well-rounded people here. So starting with teamwork and collaboration, it was really important to us that we build great team members for moving on to their next projects and really minimize a super competitive environment that it can be in an internship program. Learning orientation, this is about how well they learn. Do they know when to seek out help and when to go heads down and figure it out themselves? Do they not get lost in a rabbit hole and come up three days later looking disheveled? Do they know how to kind of navigate this difficult world of constantly having to learn? Technical abilities, can they do the job, of course? Exhibiting our authenticity value, this is one that's specific to phase two, but you might find a similar one in your own organization. This is speaking to that, culture adds, not just culture fits. So are they making great relationships with people outside of their cohort in the company? What else are they adding to the fabric of your culture and do you see them as a cultural add to the company? And then the fifth one is agency skills and the shorthand for this is can you put them in front of a client? So we're talking about written communication skills, verbal communication skills, their overall professionalism and maturity. This is when we saw it really increase a lot throughout the program. So really key here is that every two weeks, mentors are logging both their score and some anecdotes and evidence of why they gave them that score. So we can really track their progress and see how they're doing along the program. These criteria are shared with the interns as well. So everybody knows what the goal is, everybody knows what the metric is. So this then serves as a framework for both keeping track and evaluating the intern and giving them feedback. So giving them feedback on these very specific things so that everybody knows what the goal is and is working towards it. All right, so at the end of the program we asked our interns what the most important thing they learned was and we got several answers like this which is being a good team player is greater than being the best coder. So if you really instill those values they're gonna come through in your intern. Finally, we get to the end of the program and it's time to make the selections. So the way that we handled this is we handed over a packet of information to our people on the delivery team who are doing the final interviews. That packet included mentor evaluations that you already saw. So really detailed feedback about them. Program director evaluations, we were kind of there from the beginning and had the overall picture. Innovation pitch performance, how they did on their innovation pitches, their scores from the Acquia Builder exam and whether or not they achieved the certification. And then we had to add one which is above and beyond acts because we had a lot of interns just kind of go above and beyond. Our little t-shirts here, one of our interns decided to make t-shirts for the group. It's a rubber-decky. There was a big rubber-decky thing throughout everything. We had interns forming self-study groups. We had them reaching out to our subject matter experts and organizing events to learn from them. So we really wanted to capture those things that they did on their own initiative. And we handed these things all over to our delivery team who interviewed them and made the final, final selections. So let's talk a little bit about mentors. We really believe that one of the most important things our senior people can do is train junior people. This model gives you a framework to encourage a culture of mentorship and bring new mentors and managers into the fold. So Jamie has a great blog post coming out about this in a little while, so keep an eye out for much more detail on this. But this is what we did. We did a series of four sessions of mentor boot camp, teaching the fundamentals of being a great manager. And our sessions were teach, reflect and respond, mentoring different personality types, active listening, and then documenting and giving great feedback. We also had weekly check-in meetings with the mentors. These were great meetings where we all got together and we talked about the problems we were having with interns, the successes that we were having with interns. We practiced giving feedback. This was really great for our new managers especially. And it was also really important for us to get a sense of the relative performance of each of our interns because we're only working with one as mentors. Really exciting from this is we got two new managers out of this. So through the program, two people who wanted to try out being managers were awesome at it and are now gonna be managing two of the new hires that we have from the program. So this is a really good way to fast track that and for people to try it out for 14 weeks. If they love it, awesome. If they hate it, it's done in 14 weeks, which is great. Okay, so we're getting to the point of sort of wrapping this all up. You have a great idea. You really believe in this program. So how do you sell it? The first thing is the math, okay? So it has to be a good financial decision long-term. It absolutely has to be. So figure out what it costs and all the costs, not just the cost of the interns, hours, and laptops. Show that you've really thought about what those costs are and what's the return on that investment. Internships are not cheap labor. So be really clear on the payoff, math-wise. Culture and morale. So in the open-source community, especially a culture of teaching and mentorship is really, really key. Figure out how to leverage the program to really add to your culture. Commitment. So this is really hard, especially when you have your normal job going at the same time. You need a great plan, and then you need to be able to communicate that plan and show that you have the commitment to actually execute it. Be persistent there. Marketing and thought leadership, this is a really important one. So take this opportunity to build your brand. You're gonna get some great stories out of this program. Do things like thought leadership, presentations, write blog posts, get great pictures to capture these things and use this as a way to tell your story and build your brand. Huge one, managing the risk of junior talent. So one of the major reasons companies don't take on junior talent is it's really risky. Are they gonna be good? Are they gonna make us look dumb in front of our clients? Are they gonna be more work than they're worth in the end? So basically make the case that we're taking this time to craft them and give them our stamp of approval. That means they've gone through all of these different things. That means that we are confident that they're gonna be great on an actual client team. And then finally, business strategies. This is very specific to your organization, but know what your business strategy is overall and find ways that this program can contribute to it beyond just recruitment. All right, so we've had a lot of lessons that we learned as part of this. A lot of them kind of hard lessons early on and smaller things. We'd like to share a few of them with you. These aren't necessarily recommendations, but we kind of bumped up against these walls. Maybe they can help you if you're intending to do your own internship program. First off, find your allies in the company. We did a little bit of guerrilla innovation here. And it was all really hinged upon having really good folks in the company who could speak for us and advocate for us. Aside from that, and this is a big one, I think we learned over a couple iterations, is I think we need more hours. We did 16 this last round. I think we could probably increase that whatever works well in your organization. I know we need more hours when we got a lot of project managers coming to me that are just begging me for interns, because there's all of these projects that have gaps that maybe an intern can be really valuable on. And I have to say, sorry, they're at 16, and it's Thursday. And I know the interns, they want to be there. We're kind of preventing them from that opportunity. And I know we could benefit from it, definitely. One of the things we learned is ensuring there's a good way for an early kind of off-ramp for people that might come in, it might be overwhelmed, and might need to bounce, right? And that's totally okay. Agency life and we're an agency can be kind of crazy and can be a little bit unexpected for folks. I'd also say in relation to that, being clear about the hiring intent from the beginning is really key. Don't leave a lot of ambiguity about who's in or who's out, right? Try to avoid a little bit of a fight club, sort of mentality there, just be very clear about what your intentions are and what you're looking for and numbers so that everyone's aware. The hardest part of this, and this is really, really tough, you're gonna say goodbye to some really amazing people. You're gonna have a number that you can bring in, you might have to drop a wall down. And it's gonna be really hard to look at someone who is awesome and say, hey, I'm sorry. But here's a good chance for you to advocate very strongly for them after the program. Be that reference, be that letter of referral. It can just be so, so helpful and introduce them to maybe other folks that you might know within the community. Look at phase two and kind of where we're at. We think 12 interns is right about the limit. I don't think there's a lot more we could take on. So look at your organization, do a quick comparison and do the math and say, all right, maybe we don't go for 12 right or maybe we go for 16, right? Completely up to you, but just so you know, I think we hit the wall 12, which is good, we know. One thing for me as some of the technical leadership is I think we do need to hit Drupal earlier, harder. I kind of brought it in a little bit too late, I think. So my advice to you, get into the Drupal a little bit earlier if you can. And finally, we suffered from documentation fatigue the first go round. I think we had something, what was it? Over 900 documents, our second round of the program. We've looked at, I don't know, a dozen of them. Yeah, so there is a tendency to document everything, every little tiny thing for maybe a little bit of CYA, maybe a little bit like, hey, we don't wanna forget this, but ultimately all of that adds overhead and makes the program a lot more of your main job, right? So you don't need to document everything and once again, good, not great. Let's find that middle ground there. So just a couple of resources we wanna point out. On our website, we have a blog series already up about this program, covering a lot of topics that we haven't talked about already, for example, doing some research in our sort of anecdotal evidence about what the next generation of tech is looking for in a company in terms of values and things like that. We've got a couple more to come, so the one that I mentioned that Jamie's doing about training great managers and mentors, and then we're gonna do kind of an interns and mentors tell all reflections after the program, which I think should be pretty fun. We're also gonna be producing a white paper later this summer that talks about all of this information and goes into even more detail about how to do a rule and give you guys a playbook. We're also gonna be releasing a starter Trello board, so we're gonna take our board, we're gonna genericize it, we're gonna open it up and give it to you guys to take and mold and use it as a starting point for your own, so lots more to come on this topic. Cool. And thank you everyone for coming out on the last day on Thursday. I know there might have been a few late nights out there, I'm really appreciative for those that are listening on the video. I want you to know it's standing room only in here. We'd like to take any of your questions, any ideas by the way, we're open to ideas. And a quick shout out real quick before we get going. Please share your feedback at that link and they reminded me, please attend Friday's Co-Sprint if you can. So thank you very much, folks. Presumably, this is not, you're spending a lot of money, so is it always a positive reaction to that sort of experience? So you're right, it does take the right relationship, but one of the ways to communicate it that we've been pretty successful with is talking about essentially, hey, here's an extra set of story points that we're gonna give you per Sprint. We're in Agile Rights, so it's kind of something we can quantify. I think one of the things I try really hard to do is to really emphasize this won't be extra work for you. For all intents and purposes, I'm just gonna look awesome with this Sprint as my story points go up, right? If that's the kind of conversation, we generally see a really low barrier to resistance. There's like, yeah, Bloom's gonna get nine more points of stories done this week. When we talk about it that way, they're generally okay. Now, there are certain contracts out there that have privacy rules or certain contractual obligations. The government's pretty famous for this, right? You're like, nope, you gotta vet everybody, right? They gotta go through a background check. And once again, it's just kind of up to your client mix. Yeah, and one thing that we found is people were actually more willing than we thought they would be. We kind of did an assessment of the projects that we had mentors on, and we picked the ones that we thought people would be okay with and the ones they wouldn't. And we ended up with more in the pile of clients being okay with it than we expected. So it's always good to ask. I just wanted to go quick. How would you look at 12, 14 weeks per month and get a plan for these people who I don't know how good they are yet to be clients with 12 weeks of it? Yeah, okay, this is a big one. And this is actually one that took, we learned a lot in the second iteration of that. We looked at what the curriculum was and we got kind of a velocity after the first couple of weeks, like, all right, when are they gonna be kind of done with our curriculum? And it led us from the very, very beginning. We started talking client work from day zero, right? Get those wheels greased because it's not a quick process. But I would say if you're pulling from a good pool of back to the code school thing, right, you remove a lot of uncertainty from the humans after they've been through that initial vetting process that code school filter site is all about that. Because the people that are showing up after the code school, they've already kind of like made a really clear statement of pretty good, right? We are shocked at the quality and actually the diversity of the people that were applying this year. I'm rambling a little bit here, but we were at 50-50 representation of female male applicants in December. I hadn't seen that in prior years. And the quality across the board was higher. Something's changing, some outreach, maybe code.org is just kicking a lot of ass. We're seeing a really good rising of quality, but we've got that expectation. It makes answering those questions of where we'll be at in eight weeks a lot easier and a lot clearer. The other key player here is your mentors. So they know they've got interns coming on in a little while and so they can start paving the way. They can start figuring out, okay, what can my intern work on? How can I help facilitate this? And they know the project well enough to be able to really start paving the way early on. So give them a couple of weeks to kind of plan that out. So let's see what the benefits of internship program would be, but like just the mental hurdles that they get there, this is going to be so helpful with that. So thank you. A few questions. So, you can ask me a broad range of services. How much of this is, okay, everyone on the internship team, you're all going to be doing design and backend and product management and marketing. Where do you find people in different directions? I'll speak to the curriculum specifically because that's where I spent most of my time there. And one of the things, we actually do make our product manager specifically go through a lot of the technical cards. They've got a really good support structure with the other interns that might be there for the development purpose. It's been really helpful and we're seeing a lot of really technically sharp product managers, project managers coming out of this, being able to kind of like have that cross-disciplinary thing. Interns do a really good job though of sort of self-selecting into roles. And I'll mention one specifically, actually on the Weight Watchers project, we have a very sharp front end, front end developer, lots of JavaScript, lots of CSS architecture kind of stuff. Maybe not as experienced in Drupal, right? Probably couldn't go right a hook from scratch or anything, but really one of the top in the SAS and the JS. So that made it real clear after about four or five weeks, aha, okay, if we've got work coming in, if we've got requests and they're looking for things like, hey, we have a pattern lab prototype, that we've got to bang out in five days. I know exactly where I'm gonna direct her. Yeah, the other side of this as well is that you can recruit from design schools, you can recruit from coding schools. There's no recruitment program for junior sales or junior product managers, and so you get those two funnels essentially, and then our approach is we expose them to as many things as we possibly can. And we say, here's a peek into this world and here's somebody that you can talk to about it. And then they go through that self-selection process of if they're interested, they pursue that. We did an interview with each of the interns, I think it was about two weeks into the program when they'd had a little bit of exposure and we said, okay, you came in as a developer intern, what are you feeling? You're feeling still you wanna go the developer route or developer, but you're kind of interested in product management. They've done 40 cards by that, 45 cards by that point. How you feeling, is this where you wanna be? And then we had one person who came in kind of more on the product management side and she had a real aptitude for sales. And so at that point we made the mentor pairings and we paired people sometimes with two different people to kind of see where their interest lies. This is all about giving them all of the options and letting them kind of choose their own adventure. And then of course guiding them, we can't have somebody who's really excited about being a software architect, doesn't it turn so? Watching them surprise us of where they eventually ended up was one of the real great benefits of this program and seeing people like reach some aptitude that they weren't necessarily just sort of pigeonholed into a role, right, they got to grow. I think the QA team, or the GNS team, with some folks who came in as developers and they're moving into more of a project management role on a testing role, which is awesome. Yeah. So a question I had is that I'm a building remote engineering team in a city that is a college with 160,000 people, 40,000 of them are students. The thing I get is that I've been giving a lot of international student applicants that have no intention of staying in any of our programs. Right? And so what is like a reasonable expectation I can get? I should have reviewing applicants with an intent to hire at the end of the program. Sorry, can you clarify, so with the international student, can you clarify what you mean exactly with the international applicants? So like the majority of them are just doing the sort of things, right? Got it. But if I want to hire these people at the end and they leave, then I'm, you know, you feel like you're just like wasting your resources and trying to follow those sort of actions. Yeah. I defer to our HR representative in the room. No, I'm sorry, I think there are a lot of factors here and this is one of the things we were talking about with communicating the hiring plan. I think you have to kind of define yourself as are you an internship program and at the end, you get a certificate and that's awesome and it's a contained experience or slash and, is this that and then you're looking to hire. I think in general finding people who are interested in a long-term job is gonna get better results for you in terms of performance and whatnot. But the people who move on from your program are number one amazing brand ambassadors and that's really, really important that they go on to speak well about your company, their potential new hires in the future. So that's that. So it's really as much as you can support but if you're looking to hire, maybe focus on the people who you know are looking for a job. But we haven't, I mean all of our interns so far have really been looking for a job. We haven't really encountered that but I think it would be interesting to create a program that's more of like a, here's our stamp, you're also creating a lot of thought leadership there of like this is what we consider, you know, an appropriate training program for a junior developer. Yeah, how much time, is there, is the next session starting? It's the lunch. Oh, it's awesome. Yeah, we, we, we, it's, it's, we've only been going for a couple of minutes. We still have five minutes. Okay, cool, because if you have a whole slide about this, I promise I wasn't thinking about it. So you're here for here, which means you still have some internship hires around your company. Did their subsequent performance as employees shape the internship program at all and did they participate in the internship program or in feedback on the next program? Yeah, absolutely. So we had a requirement that everyone who, who was hired last year had to teach a class this year. Just one class. Just one class, yeah. And they volunteered to do that very happily. We also had one hire from last year who was a mentor this year and that was really exciting to see him kind of move into that role. Our first two interns are now full-time hires and they taught courses as well. So yeah, they're absolutely, thought leaders around the internship, they're advocates and a lot of what goes on is like the interns are like, oh, you graduate, tell me all the things. So it's like a really great community and a lot of people who can kind of answer those questions and make them feel more comfortable to say, oh no, you need to talk to this person. I've been there. So we've got a really great momentum going. We now have 11 people who've been through the program and hired on to phase two. So, that's great. And just to speak to kind of the performance after the internship program forum, obviously having gone to the code schools, having been through it, having been on client work and entering phase two, like they're just, they're off like a rocket. But it's been, one of I think the coolest things for me is they know exactly what to expect, right? And in agency life, it's crazy, it can be crazy and they know, right, and they're ready and they've been kind of inoculated to that. So they're coming in, they're just like, yep, no problem, right? They slide right into it. That's been something really, really great. So it's not a new hire, this is kind of like, ooh, wow, okay, this is a lot, right? They're ready to rock. Yeah. In terms of like knowing what crazy agency life is like having done the internship, did you have any interns know about it? Are they like, yeah, you guys are crazy, this has been great, see you later. We haven't had any, we haven't had any self-select out, right? In the sense that they were like, yeah, this is this is too much. I think there was just so much energy there and so much like willingness to go like prove themselves that they were all, I mean, after having vetted through a lot of the interviews, it was pretty obvious they were in it. I think, so our point about having an off-ramp though is that's one of the things, there are a few people who we think might have self-selected out. Had there been an opportunity for it that wasn't just feeling like quitting. So I think that's one of the things we wanna build in is like, at the four week mark, we're gonna do a check in with everybody and absolutely no hard feelings if anybody's out. And if they wanna keep going, absolutely we're all about it. But I think if we had that opportunity, we might have had over the past little while, one or two who might have dropped out. Apparently there actually is a talk in here at 12 o'clock. Who's the next speaker up? Are you here yet? All right, we still have two minutes, hit me. 16 hours a week. So we, when we hired the interns on, right, they're hired for 16 paid hours in the program. And that was kind of us to protect them, right? And also to just kind of give us the ability to kind of come in at a budget that I think would work. But we did see a tremendous amount of requests for them. A lot, actually to the point where I was saying no because we're at hours very quickly than if we had had more agreement on longer hours to begin with. The billability though specifically on client side is trying to make that work is really, really hard because I think clients have an expectation of say like velocity and quality. And it's not that there's gonna be any bad code that's ever pushed to the client, but they do have an expectation. They think of like the developed quantity they're getting from you as an agency. When we throw interns into the mix, the conversation gets really hard about like, oh, are you selling me less good work? It's like, well, no, you know. So I think being able to provide them as extra available resources without the overhead of like, okay, well, no, let's wheel and deal about what you owe us for this. Is an intern worth half a developer? Is an intern worth a quarter? We couldn't do that math, right? Yeah, we couldn't do that. It got a little painful there. So we just kind of said, you know what? It's costing us more to figure this out and to put very expensive available people working on it. Then it was to just be like, it's extra resources on the project. And just to quantify when we tried doing billable, I think we got like three interns on projects. And without doing it, we got all 12. So it's just a lot easier and we slept more. They are only in the program 64 hours a day for four days a week. Roughly. Roughly, right? There were special circumstances where interns did make it onto projects as basically primaries. And we were able to get them like basically full time and close to full time as is legally allowed. And they were able to be really, really productive there on certain circumstances. But yeah, we limited it to 16 and said we got to cut you off at 16. I think the feedback we got was they would like more time in the organization. So I think we need to probably get out of here for the next talk. Thank you very much. And please find us where we will talk about internships all day. Thank you so much. Thank you. Thank you.