 My name is Rick Elrod. I am on the community platform engineering team at Red Hat, and I work mainly on Fedora's infrastructure. I do a lot of sysadmin tasks. I also occasionally spend my time to other teams like the websites team, but I focus primarily on infrastructure sysadmin work. I want to talk about the onboarding process. I want to talk about including newcomers to be able to easily volunteer and contribute to the infrastructure and how to retain them once they show some interest in doing that. I am going to give a little bit of background on why we care about this problem. Talk a little bit about the history of the infrastructure. I am going to talk about how we currently handle newcomers and onboarding new people and how to retain them. I have some ideas that I am just going to toss out. First things first, these are all my opinions. These are not necessarily the opinions of anybody important in the project. These are just my opinions. The ideas might be horrible, but hopefully not. These are just things that I think might help that I will talk about. Personally, I started out as a volunteer in 2010, eight years ago. In many ways, this has been a very life-changing event for me. I have made many friends in the project. I have gotten a lot of experience. It has been a great thing. I would not change anything about it. I care about Fedora. I care about the contributors, the infrastructure and the continued success of the project. Why should you care? Well, the infrastructure of course plays a very critical role in Fedora. Without the infrastructure, the project is not going to get very far. The infrastructure is inherently community-driven. I will talk about why that is. This is an email back on the mailing list from 2005. This is from Elliott Lee. This is the first thing that I and other people I have spoken to can find that talks about the Fedora infrastructure being kind of an open, community-driven thing. This is saying we can get people shell access to the Fedora boxes and we can start encouraging the community to have access to these boxes and actually start playing a role in maintaining the infrastructure. This is the first call for help that I can find from the community. It goes back all the way to 2005. Since at least this point in time, Fedora infrastructure has been what I will call an open infrastructure. We have encouraged contributions from the community. That email that I referenced was sent out right before Fedora Core 5 came out. Right between Fedora Core 4 and Fedora Core 5. This process of the infrastructure being open kind of goes back quite a while. I just realized I did not start my timer. In 2007, we started using the Puppet configuration management tool. This kind of gave a central place, a central repository for our configuration files and other system scripts and inventory. We gave volunteers access to this repository fairly liberally. It wasn't officially quote-unquote public because back in the early days it had passwords and other information stored in the Git history that we didn't exactly want public. We did give access to it fairly liberally. In 2011, we started the FI apprentice fast group, which I will talk about quite a bit. That leads us to where we are now. The FI apprentice fast group. The idea was that anyone who had an interest in helping out and seeing how the infrastructure works, getting some experience, becoming a part of the community, could sign up. They could join the FI apprentice fast group and they would get access to most of our boxes. It would be read-only access so they wouldn't have pseudo privileges. But they could go in and explore. They could see how things are set up. They could see our monitoring systems. They could see our configuration tooling and how we use it. Again, we add people to the group fairly liberally. We have semi-automatic removal for people who are not using their access. By semi-automatic, Kevin goes through and sends an email out every month. The idea, at least, was for it to be more automated. From the perspective of a new contributor, the processes, they send out an email to the infrastructure mailing list. They express some interest. They say, Hi, I'm so-and-so. I want to help out. People respond to the email. They say, Can you join us at our next weekly meeting? Introduce yourself. Talk to people. Tell us what specifically you want to work on and we'll help you get started. Then they come to-we tell them at the meeting, come to the Fedora Admin IRC channel. We will add you to this FI apprentice fast group. You can get access and start seeing how things work. They get sponsored into FI apprentice. We also tell them we have infrastructure office hours every week. If they have questions, they can come in. This is an hour dedicated just to working with newcomers and helping them get familiar with the infrastructure, with any problems they're having, and they can ask questions. This time is dedicated for them to get help. I want to talk about retaining newcomers. Once they express interest, once they get sponsored into this FI apprentice group, a couple of very unofficial and cursory statistics. Every week in our meeting, we seem to average between one and three newcomers expressing interest. This varies every week. Sometimes we don't get any, sometimes we get more, but I would say on average between one and three people express interest at our meetings each week. If you look at the Fedora Admin logs that happen after the meeting, we get less people than that coming in and actually asking to be added once they express the initial interest. There are currently 27 people in FI apprentice that are not sponsors and not administrators. These are like actual people who express interest and want to start volunteering. We had three people added in June, two people in July. Hurdles. Every step that we ask a newcomer to do, so sending out the email, coming to the meeting, going to Fedora Admin, being sponsored, every step here has two effects. Probably has more than two effects, but I'm going to talk about two effects. The potential to reduce their likelihood to make it through the rest of the hoops. It's kind of self-explanatory. The more work somebody has to do to start volunteering, the less likely that they're going to make it all the way to actually being able to help. If they do make it through the hoops, though, it shows some level of commitment. It shows they're really wanting to help. They're willing to learn our processes and do what they can to help us out. The process is kind of a four-step thing. They create a fast count. They send out a letter to the list. They attend a meeting. They get sponsored into the group. Then once they do all of that work, we kind of run into a problem where this group is called FI apprentice, but the users are not actually apprentice. We kind of leave them on their own. We tell them where the documentation is, which is another thing I'll talk about. We have the office hours, but we don't really do much handholding. We kind of just say, here's what we have. Ask questions as you go along, but this is it. I think that there are some things that we can do to make that a little bit nicer for newcomers, and I'll talk about those ideas. In an infrastructure with roughly 600 hosts, this includes like build VMs and our staging instances and cloud instances and everything, it can be kind of hard for people to figure out what to do. Documentation falls out of date. Some people learn differently than other people. Some people are more hands-on. Some people are visual. Some people read things and understand it, no problem. What can we do to cater to people who learn differently? How do we encourage people to stay involved once they actually make it into FI apprentice, once they express interest and get access to things? How do we encourage them to continue contributing? And also, what is in it for the volunteer? Why should they want to continue contributing and how can we express that to them? I'll talk about some of my ideas. Again, some people learn differently. Some people are visual learners. Some people are hands-on learners. If we want to start catering to people who learn differently, what if we started doing, and this is something that I've talked to several people about and everyone seems to like this idea, what if we did kind of video tutorials, video screencasts that we upload on YouTube or anything else, showing the processes that we do as sys admins every day, altering DNS records, deploying new hosts, taking websites out of the proxy configurations, all these things that we do. What if we made little video tutorials and we could actually link them to people who learn visually? This is something I've started on in my spare time. I haven't actually uploaded anything yet, but I've talked to people, I've talked to Smooch and some other people, and people seem to generally like this idea. People think this is a good idea. Again, some examples, provisioning hosts, how to run playbooks, how to make playbooks, how to update our DNS. It also saves some time answering common questions. We can point to the videos that are already made. It does have the potential problem, same as with our text documentation. It can fall out of date very quickly. This is another potential pain point. As our processes change, the videos will also have to change. I said some people are also hands-on learners. Some people learn visually, some people learn hands-on. We run a cloud infrastructure in Fedora infrastructure, where we can spin up test instances, we can spin up all sorts of VMs and go all sorts of cloudy stuff. What if we use this as a way to give contributors a kind of playground test environment that looks somewhat like a small-scale version of the infrastructure to do whatever they want and learn how our processes work? An idea that goes along with this is what if we give newcomers a list of tasks that they can go through, and they can step through the process, they can ask for help on individual processes. If they're not sure, if they get stuck at one step, they can ask, hey, I'm going through this particular checklist, I'm at this step, I'm not sure what to do, can you help out? We can take a look and help out. The workflow would be, they would get into FI apprentice, similar to how they do now, and then we would launch a couple of pre-configured cloud instances, depending on where their interest lies. I have a couple of examples. One example, they say they want to learn how to deploy a new web app. We run a playbook, a couple of cloud instances get launched. One of them is like our Ansible hosts, they would learn how our Ansible templates work. One of them is like an app server, so like a wiki server or FedoCal or any other web app. One might be like a small database server and one would be like a proxy. They would learn how all of these different kinds of servers and services that we run interact with each other. They would learn how the proxy communicates to the app server, where Ansible fits into all of this. It would basically be a playground for people to learn our processes in a small scale way, where they can't actually break anything because the cloud is separate from our production networks and it would provide a way for people to learn how things work without actually potentially breaking anything. Another example, say somebody comes in and they want to learn how our monitoring system works. We use Ansible for our Nagios templates, it's all automated nowadays. What if we had something like that? We run a playbook, it spins up a couple of cloud instances. One of them is like our Ansible host that actually does the Nagios template generation. One of them is like a Nagios server and one is a host that gets monitored by Nagios. They can see how does the monitoring protocol work, how do I add things to Nagios, how do I add new checks to Nagios. It's another idea that I have. So again, the workflow, they would get added to Fi Apprentice, we would launch these instances and then we could give them a list, going back to what I said, we could give them a list of of tasks to go down. If they want to learn about monitoring, we tell them this is how you get into these instances. Here's how you can start playing with the configuration system and exploring. And they go down the list and at the end, or if they have questions, of course they can ask us questions, but at the end, we could look at their instances, see how they did, give them feedback if they want. So this encourages feedback, this encourages communication with people who are already active in the infrastructure and with people who are looking to become more active. It also lets us see kind of us visually what they are focusing on, what they want to learn. And it also provides a clear promotion for, when people want to get promoted out of the Fi Apprentice group and actually focus on one area of the infrastructure, we can easily say, okay, you completed the monitoring checklist, so we'll add you to the sysadmin knock group, the monitoring group. You completed the web app, the proxy checklist, so we're going to add you to sysadmin web. So it provides kind of these clear cut levels of how to get added to the groups beyond Fi Apprentice. This idea, of course, does have some problems. We have to maintain all of that infrastructure, we have to maintain the playbooks that spin up these instances. It's never going to fully demonstrate how the pieces of the infrastructure interact with each other. There's always going to be other pieces that interact. In the examples I gave, maybe some of those apps talk to Fed Message, but we don't demonstrate that in our checklist. There's always going to be other pieces that are not covered by giving people a small-scale playground to work in. There's also the question of, if we start doing these task lists and have people go through these tasks, what should be on those lists? How do we bike shed and figure out what should be on those lists? So motivation, keeping newcomers motivated. So what if we had a way to award something like badges for hard work? Because they aren't going with us. Oh, wait. So we do have this infrastructure, this thing called Fedora badges. And they are easy to make. They're cheap to make. They require a tiny little bit of graphic design, but that's about it. They're free to award. They don't cost anything. They make contributors feel good when they help us out. They do something. We want to say thanks. It's a virtual token of appreciation. And I claim that we should award them. And the infrastructure for doing that is already there. We have this system in place already. So some examples. Going back to, you know, the cloudy idea that I had. You completed a checklist on your cloud instances. It looks good. Here's a badge for that. You know, thank you for putting in this work, showing us, you know, that you want to be involved in this area of infrastructure. Here's the badge for that. Thank you. You submitted a patch to our main configuration system, our repository of our configuration system. Ansible. Thank you. Here's a badge. You know, maybe you said five or ten patches. Here's another badge. Thank you. You know, maybe somebody comes in and they run interference during an outage. So, you know, all the people who can do something to fix an outage are working, but we have a lot of people coming in and saying, hey, this thing is down. But, you know, maybe an apprentice, you know, is kind of running interference and saying, yeah, we already know about this. People are working on it. Thanks for letting us know. We're on it. And we can give them a badge for that. So my idea is, like, we already have the system in place. It makes people feel good. Let's use more of it. Yeah, that's what I just said. So another problem is that being a newcomer can kind of be scary, right? We have this Fedora admin channel that kind of, you know, people come in and ask questions and everyone hangs out in. But we average like 275 to 300 people in there at any given time. And, you know, we want to encourage community involvement and people coming in and people who don't know how things work. We want to encourage them to be able to ask questions. But doing that in front of 300 people can be quite scary, right? It's a big crowd to get up and basically say, I have no idea how things work, like help. So the idea here is, and I haven't talked to too many people about this, it's just an idea that I'm throwing out. What if there were a smaller channel, a smaller place where people can come in and ask questions until they're more comfortable, you know, with how things work and interacting with the community at large? So I had this idea of Fedora for Newcomers. I have no idea if this will work. I haven't mentioned this channel to anybody before, but the idea would be, you know, limit the channel to infrastructure veterans who can answer questions and people who are new within the last couple of months who are still learning how things work and want a small place, probably under, say, 50 people, to ask questions rather than a place that has 300 people. So there are, you know, a couple of problems here. How do we enforce the end month rule? How do we say, you know, hey, you're not a newcomer anymore, like, get out, right? And also, does this isolate newcomers? Like, this is a community-focused project. We want the community to interact with each other. Does this kind of isolate and segregate the community too much? We want to kind of avoid that problem. So those are my ideas, and I'm going to kind of just, I don't know, I guess we can have a discussion. I don't know where I am on time right now, but those are my big ideas that I have. I don't know what people think of them. Again, I've talked to some people about these ideas, and people seem to like them overall. Another thing that I didn't mention, and I was going to add a slide, and I forgot, is Clemont has been doing infrastructure office hours. I guess I did kind of mention them in passing every week, and I don't really know how the turnout has been on them. I think we've gotten some people coming in and asking questions, but... We got on two apprentice that have been coming almost every week since we started, and they're working together on tickets, so I think that's quite nice, collaborating. And yeah, that's pretty much all. We haven't had too many other people coming, so... But for me, it doesn't cost too much to just keep it running. I'm there on ISC, and I have to answer questions, so I just want to... I think it's better to have it there and see if somebody comes or not. So, I'm happy to take any questions on my ideas, or if other people have other ideas that they want to talk about. So, I'm really interested in the idea of having these little growing environments for running. Do you have any sort of proof of concept, or can you say more? Okay, so the question is about the throwaway cloud instances for a small-scale learning environment. I do not have a proof of concept, and I was... new cloud. So, we are currently redoing our cloud infrastructure right now, and that's been my focus lately, and my thought was once that eventually happens, I can... I have no idea what I just did here. I can kind of start playing with the proof of concept once that new cloud is deployed. But no, my idea, I guess, would be we already have ways through Ansible playbooks to create instances that do certain things. So, we would just write a bunch of playbooks that kind of configure a small-scale environment for each area that somebody might want to learn. It seems like we might be able to just limit the size and scope of the deployment to make it more functional for a specific space, right? So, I only need one server that launches the proxy. Right, yeah. And then, configuration would be all true. So, there's another thing that I'm thinking here, which is that the opens... So, the SUSE community has something kind of similar to this in the sense that they allow their users or their participants, anyone who wants to learn more about the environment, they allow them a checkout service. So, they're given an access management role, and that role lasts for up to two weeks. And then, everything that they create is then shut down after that specific checkout time. And the tools that they use to do that are obviously open source. Well, it's... I mean, the one that I know is built on EC2, but it allows for them to... allows for their users to create an environment, do with the environment what they expect to do. They identify the resources that are expected to be deployed and give them sort of a top budget, right, in terms of how much spend is associated with that. And then, after a specific duration, all of that is cleaned up. That sounds something like what we're looking for. Can you... Yeah, point, yeah, maybe even just something on the mailing list, even? Since we're talking about a space for newcomers, so I have a solution to put a part in discourse where newcomers could put their questions and stuff. Yeah, we have a discourse instance in Fedora. So, like this admin part, like Fedora infrastructure, could have a section over there where newcomers could put their questions effortlessly, and don't have the fear anymore of actually asking a question in the IRC channel. I personally have not played with discourse much. I don't know if anybody on the team has done much with it yet, but yeah, that's an option. Any other comments? Anybody want to throw things at me because of my horrible ideas? Is there a way for a person, you know, one of the apprentices or anyone really, to just provide a question in an anonymous fashion? All right, so could they message a bot and have that bot put that question back in the IRC channel as a general? Right, so the question is, can newcomers ask questions anonymously? Right now, I don't really think we have a system like that. You can use any ISC. Sure, that makes total sense, but at a certain point you're kind of getting tracked. It's okay to be tracked by a lot, but you know, if your name shows up in your interview, can your reporter find out if you asked a stupid question? So, just by experience, I think, so we see on the mailing list a lot of people sending in their introductions, oh, I would like to get better in ensemble, I would like to learn Python or things like that. And for me, I think what's very difficult for them to get involved in the infra, I don't really think it's the tools, but it's more the process. And how, pretty much you can be interested in learning Python and say, oh, the guys in Fedora Infra, they're doing a lot of Python applications, so I'm just gonna try to join them. But you have no idea of what is it to actually create a distribution and what is the process involved to actually create the distro. And I think this is very, very difficult to actually get. Because if you want to start to help on projects, you have to kind of understand what those projects are doing. So this is maybe a bit more on the development side of Fedora Infra, but I think it also impacts the CIS admin and and things like that. So maybe we, we have a few applications that are not so much tight to actually building Fedora, but more like for the community. So maybe those, we should maybe try to take care of a bit more, but usually that's the application, we spend not so much time because they are not that critical for like releasing Fedora. But I think they are the good application for an apprentice to, to get started like badges or Piger is another good one because based on Git, I think it's like maybe a bit more generic application, maybe a good idea to try to say, oh, you want to help us, you can, you can look at those applications first and then, and then slowly learn about the process of building Fedora. I think we were trying to do that with the, the easy fix system that we created. And we've had varying degrees of success for that. But yeah, I, I see where you're coming from. I think that's a good point. What would you say is the bigger problem that people have to get used to general concepts of administration and developing the infrastructure software or more that people get involved into it, like, oh, they are already sysadmins at companies or whatever, and then just go in and make something out of it? Or is it really people who are often try to learn new tools and say, oh, I can do this in the infrastructure team? That's kind of a tough question. I personally, I think there's two kind of, kind of go hand in hand to some extent. I don't know. Again, I think there are things we can do to specifically help people become more familiar with our processes, which like the things I talked about. And, you know, through doing that, they'll, they'll also gain general sysadmin experience as they go if they don't have it already. I, you know, it's hard to say which the quote unquote bigger problem is. I think they kind of go hand in hand. Well, I'm being an apprentice, like forever, because I used to do sysadmin the whole fashion way. And now, right now, I'm starting to learn Ansible. And I think the, even when you are really helpful and you answer everyone questions, there is no, no really easy task. I mean, there are market as easy fix, but normally you say, okay, this is easy fix, but how? So I think the, the, the documentation should improve in, in that matter. I mean, this is, the problem is this, you call, you should go this way at least to, to start a research by yourself, at least pointing you in the right direction. It's a good point also that it might look as an easy fix for us because we do it every day and but for an apprentice to say, oh, it's tagged as an easy fix, but it's actually no clue of what needs to be done. And I think on GitHub, they, they recommend to use like, it's like help, help quantity or a good first issue of something that there is no, the connotation, it should be easy for you. And if you, if you can't do it, you're just like, yeah, so might be something we can try to change the tag there and make something. Is it better to say that it's an easy fix in the sense that no matter what, someone else can help me walk through this without, without issue, right? So someone more experienced could help me walk through it. And, and that means guidance may be necessary, but, but it's a fix that we, we all can do. Right. So one, so one of the things that we can do is while creating the easy fix. So whenever somebody new comes in, problem they face it, like where to fix the issue and where to find the file. So in that case, the, the, this problems comes in like, I have to go and talk over IRC or over publicly over the, like on the GitHub issue, right? So somebody who is very shy or introvert. So they find that issue there to even approach someone to ask how to go ahead. So in that case, we can like, at least for the easy fixes or the ghost first issues, we can actually write the steps on like, this is the way you need to find the file. This is the method you need to edit. And this is how you need to approach the complete steps when creating the good first issue. And so that it's very easy for them to, and it should be like maybe like five to 10 lines of fix that need to do, maybe, or less, even like one, two line of, so, and that can be a good start. So, so one more thing. So, so maybe with something that you put in that category, you could give maybe, if it languages for too long, you can add an extra step, like add first steps. When describing the issue, you can write that this is the issue. And because this is a easy fix, so these are the, if you're trying, like, somebody new is trying to attempt that issue. So that person can go through this file and there's the issue there. And this is what we want. And this is how you can proceed. Or these are the line of, this is the, maybe this is the module which is used or this is a module that needs to be used. So you can mention all those things in there very clear so that he, the person who is going to attempt that particular problem, does not need to like ask somebody or directly can. So it's basically learning the barrier as much as we can. Right. And I would say we do that over time, right. So, so as that, as, as this easy fix is identified, but then, but then it languages, now let's get more, let's get more details on the requirements. But I think in some cases the easy fixes are not described very well maybe. So you say, I want to do this as an easy fix, but you don't describe it enough to know where you need to fix it or, but I see what you're saying, yeah, I'm a requirement developer and not, not at great development. So if there's going to be something that's done and it's an easy fix, then they're actually sort of defined. Any other comments, questions? I will give you all 15 minutes back of your life.