 Welcome to our talk about the Kernel Development Learning Pipeline Program that Joel and I have started at Red Hat, KDLP for short. I'm Julia Denham. My name is Joel Savitz, and today we will give you an introduction to our program. So first, a little agenda. So we'll begin by describing what this is, what KDLP is, why we would want to do such a thing. Then we'll give a brief overview of the program, which includes an introduction to Linux Kernel Development course, some various things related to internships and mentoring, and this new kernel devs program group that's part of some initiatives that we have to get people involved in kernel and grow the kernel groups at Red Hat. Then we'll talk about some of the program growth, some of the exciting things that have been happening, internal and external connections, connection with the Linux Foundation and their mentorship program. Then we'll talk about some of the success stats, a little bit of the numbers, and finally wrap up with some resources so you guys can get involved if you like. So first of all, just to make it abundantly clear, what is KDLP? The main idea is that we are trying to build a comprehensive pipeline for Linux kernel talent. And it's difficult to get people into the kernel, and so we want to build a pipeline. So okay, first a little bit of history, where did this come from? So I went to UMass Lowell, and that about halfway through I did a co-op with Red Hat. Just completely by chance, I ran into this manager who was hiring at an event he wasn't really supposed to be recruiting at, and I ended up in this co-op. And if it weren't for that, I never would have encountered Linux kernel. I never would have known where to look to get started. I wouldn't have even really known about it. So anyways, as sort of a side thing, I became involved in an initiative to improve Fedora usability, stability, and performance on the Raspberry Pi platform. And then along the way, so I kind of focused more on the low level stuff and there are some gaps between what I had learned in the co-op and then what people generally are familiar with from their computer science education. So long story short, we ended up developing this into a bit of a study group to learn about kernel drivers for the Raspberry Pi. And then that was part of a directed study, like credit-bearing directed study type of course, sort of independent project, that eventually became more of a class-like type of thing with a small group. We ended up making some slides through the help of the Red Hat Research Department. Bringing in some interns to help work on the project and further work on the Raspberry Pi stuff. The course became more developed. We got more interns. You know, eventually we pitched the program to the Red Hat Enterprise Linux, the RELS Shark Tank program. And we got the sponsorship of Mike McGrath, big thanks to him for sending us here. And from there, we continued to develop the program until it became a full-on credit-bearing course, or a regular course, academic course that we created. All the stuff was open source. And we will continue to talk about that later on. But for now, I'll turn it over to Julia to talk a little bit about why we're doing this. So why did we start this program? So we noticed that many Linux kernel engineers are getting close to the retirement age. I started at Red Hat three years ago full-time, and I've already seen a couple retired. We've also noticed that it's really hard to find qualified candidates for kernel engineering. And most colleges nowadays don't even teach C programming, which is hard to do kernel without C programming. And kernel onboarding also takes a lot longer than the average onboarding time. So our solution to this was to create the qualified candidates for kernel engineering. So there are three main pillars to the program, the course, the internship, and then full-time work. So students will learn concepts in the course. They will practice them with weekly assignments that are similar to things that they would be doing full-time at a company. Depending on how the class goes, they could get an internship at Red Hat, interning on kernel. Using all the skills they learned in class, helping with the program, and then depending on how that goes, they could be full-time kernel engineer at Red Hat. And Joel is going to go into the specifics of each of the pillars. Thank you, Julia. So first of all, I'll talk a little bit about the Introduction to Linux Kernel Development course. So the general idea with this is to kind of give a meta-overview of the kernel. Because as you know, it's a gigantic project. There's not a single developer who knows how to rewrite the entire thing. No one is a master of any of the subsystems. So what subsystems do we really focus on? And the answer is pretty much none of them. We go into a couple, but the general idea is to teach people how to teach themselves, right? Teach people how they can get involved in the community, how they can properly interact with the community, while using the community standards, which we know about those, right? And so the prerequisites for this course are just a general background knowledge of C, an advanced knowledge of C mainly, and a basic Linux experience. I mean, generally programming experience as well is extremely helpful. But the overall idea is to get sort of a tour of the different parts of the kernel, some of the different tools that you can use to interact with the kernel, how to do, you know, how to navigate the code, you know, things like CScope, Elixir, some basic tracing, you know, BPF, you know, specifically the BPF trace implementation front-end, and, you know, some basic F-trace concepts. All right, which are things that, you know, once you're pointed to, they're very useful, but you wouldn't even know really where to look unless you spent a lot of time running around in circles or you got lucky enough to find a mentor or just, you know, find yourself somehow at Red Hat in, you know, a particular department. So everything we do is open source, of course, you know, that's open source for open course. And we've developed a credit-bearing undergraduate and graduate university-level course that we teach at University of Massachusetts Lowell in Boston or in the Boston area. And that, you know, that we've been running in its current form for two semesters now. We just finished the last one back in May. And recently, we've been piloting a mentorship program on the Linux Foundation's LFX platform thanks to partnership with Shua Khan, which we'll talk about a little bit later. The main idea and our main philosophy with this course is that we want to teach difficult material from first principles. So then, the idea is, you know, with internships generally, the idea is to bring people in who have some idea, well, to test them out in an environment where, you know, you're not making a huge commitment to them, right? And they, you know, they're not necessarily, you know, siding onto the whole thing. They're trying something out that they, you know, they may not know whether they like it, right? So you can opportunity to kind of vet them and do a three to six months or, you know, a longer intern, basically interview, right? And see if they are A, qualified to work in this particular area, and B, that they actually like it, that they actually enjoy it, right? That it's something they want to spend their time doing. So with that, what if we thought, what if we went back a step further, right? What if we take, you know, we're working already at the university level, kind of at a semi, you know, training wheels type of internship, and, you know, we're already thinking, like, do they like the kernel? Do they like, you know, are they competent at working on the kernel? Do they, you know, do they enjoy working on it? And from there, we can bring them in and, you know, internships already. I mean, I know a manager who was saying he doesn't even like doing three month internships in the kernel, right? Because by the time they have the absolute basics, it's time for them to go. I mean, even the six months is not very long, you know, I stand to mind eight months. I still barely knew what I was doing at the end of it. So, the idea is if we can get people set up with some of the tools, the ecosystem, right? Some of the, you know, the Git email patches. I didn't mention that earlier, but that's a very big important component of our course. Getting extremely familiar with Git and email patches, right? Because if you can do that, you know, that's a big part of the hurdle into getting involved in the community. But if you can do those things, if you have this competency, then when you get to your internship, you can actually hit the ground running and accomplish a lot more in three to six months than you otherwise would, right? And so by the time that we actually recruit them to full-time, ideally, they can hopefully shorten that two-year mentor or two-year onboarding window that Julia mentioned to, you know, a lot shorter and then, you know, saving everyone involved time and money, which is great. So at the same time, we want to support the key initiatives within Red Hat, right? And target people towards areas that, you know, that are growth areas and train new talent, right, while focusing on areas that people otherwise wouldn't really be exposed to. And now I'll hand it over to Julia to talk about some of the initiatives that we're doing within Red Hat to bring people involved and bring new people and interns, you know, together to learn more about the curl. So one aspect of the program is this group called the New Colonel Devs Group. It's a way to kind of connect all the different pillars of the program. It's an internal group at Red Hat where Colonel Engineers of any level can come. So we have a chat, so you can always send messages in chat or on the email list or you can come to our reoccurring monthly meeting and ask questions, give presentations, like if you're new and you want to practice giving a presentation, it's the perfect space to do it. Or if you're nervous about asking a question, you can come and ask in this group. So program growth. Throughout starting this program, we have partnered with different organizations within Red Hat, including Red Hat Academy. They have sponsored us to go to a few conferences. They've helped us out learn how to work with colleges. And we've also met with their curriculum team to potentially get our curriculum up on their platform. But that will take a lot of time. But they were very interested in it, which is awesome. We've also worked with the intern program in Waterford, Ireland. We're with Julia Blay. We ran a workshop, a Colonel Engineering workshop, once every two weeks for, I think, four months for a group in Ireland. It was virtual. And one of the interns in that workshop actually just accepted a full-time offer, which we're super excited about. And we've also been working with a few other intern programs at Red Hat, and we're hoping that we'll partner maybe with Burno or Israel in the future. And then our newest program integration is with the Linux Foundation that Joel will talk about right now. Thank you, Julia. Yes, so to build on that point about the Linux Foundation. So last year at LPC, we were talking to different people about the program. I ended up talking to Greg Goe Hardman, telling him about what we're doing. He was saying, oh, we already have something like that. We have the kernel bug fixing thing. He said, you should talk to Shua Khan. So I was introduced to her, and we started talking about her mentorship platform, kind of comparing some of what we're doing and what she is doing. And we found that by working together and by offering our course on their platform, we can provide some complimentary content and strengthen the value of the service that we're providing while also adding something to the Linux Foundation. So she was very helpful in working with us on this. And so we piloted having a Linux Foundation mentorship group cohort in our course this past semester. And we learned a lot from that, but it was a very good pilot program. And at the same time, we got UMass Lowell students involved with the Linux Foundation's mentorship platform and got them kind of a nice get their names on there for being involved in Linux. It's a nice thing that they can have. They can show hiring managers, put it on their resume. It also actually unlocks some Linux Foundation programs that are only open to graduated mentees. So that's very cool. It also allowed us to have probably the most diverse group of mentees or students, different terms. That we've ever had or that we could have possibly had if we were only working in Boston. We had people from all continents, pretty much not Antarctica, right? But we had people in a few African countries, I think Nigeria and I don't know, there's another one. Some people in Central and Eastern Europe. I think we had three or four in India who were interested, a couple in New York and Texas. There's one guy in Peru, really all over the world, which was very cool. So next I'll talk a little bit about some of the numbers, some of the technicals here, right? So this three actually was two until like a week ago, right? We've had three full-time hires come through the program in various forms and through our efforts we were able to get them interested in kernel, teach them a little bit about kernel, and then place them within various kernel groups, right? So we have some relatively new talent there, which is exciting. There were two people who did co-ops with us that really liked Red Hat, but it was a difficult time for Red Hat hiring and they were both credited to Amazon and Microsoft, because they were good at what they were doing. One of them, and this is an interesting thing about our program, he liked working at Red Hat so much and he liked the environment. He decided the Amazon juice really wasn't worth the squeeze and he wants to come back. He really liked the Red Hat environment and yeah, so he actually left Amazon recently, fun fact. But overall, we've had seven interns and co-ops who have been vetted and trained by our program and then come through and do internships with our group in various forms. Some of them less connected, working with other teams, some of them working a little more closely with us, working on the program. And overall, it's approximately 30 students and mentees completed introduction to Linux kernel development during the past academic year, which we're happy to announce is the most diverse group ever by gender and location. So a little bit of program information. So who runs this? Joel and I, we have Charlie who started out as a student, then he was an intern and now he's full-time and he is our course content lead. And then we have Dennis who has interned with us, we took the class, then he interned for a year. He unfortunately has to go back to college but we're hoping he will come back to Red Hat when he graduates. So one thing is this program is not part of our full-time job. We do a lot of this work on the side. And we work closely with our executive sponsor who we'd like to thank, which is Mike McGrath. And we'd also like to thank Heidi Dempsey who's been helping us out a lot with dealing with the interns. And here are a few resources to leave everyone with. We have a mailing list that we send a quarterly update newsletter to. So if you just want to be informed about the program, that is something that you can join. Low traffic. Or you, if you have a specific question, you can reach out to Joel or I. And then we have a website and that has all the course information and like we said, it's all open source. So if you're just interested in looking through some kernel slides, that is the perfect place to do it. So any questions? We have about shifting things around to make the kernel solution work closer to what people usually see after they let the, you know, after they start working computing today, now they like start to make your first decisions using like GitLab, and then you need them to, or the actual kernel workflow like, ah, not only do you know how kernel works, and you send your first patch through GitLab, try to send a patch through the mailing list and all that mess. And so how many people would react to that? So our philosophy with the Git email patch stuff that we do is that, well, it's very tricky to learn, right? It's very finicky. And so we, the first few assignments are, you know, we provide a couple of chances where the real challenge is just doing the email patches, really. Like you send some email patches, you get it going. And the idea is, you know, it's really, it's a stripped down version of Git where you really understand the fundamentals of like what Git is, right? Because there's a lot of extra abstraction that's added on GitLab, that's added on GitHub. Those are very useful tools, right? But they kind of build on this abstraction that, you know, it was originally created by Linus Torvalds to work with his other project, the Linux kernel, right? And so by making people do it many, many times, they get really good at it and they get a really solid understanding and a firm foundation about what Git itself is that would allow them to go to GitLab and then in their minds have a more clear separation of the abstraction of GitLab and, you know, pull requests. Like what is a pull request? Well, on the mailing list, it's someone sends an email requesting a maintainer to pull from their branch, right? It was a little confusing. I remember when I started out to say like, well, pull requests, like how does that relate? But I think by learning, you know, the original workflow and really hammer it again in so they really have it well trained in their mind. They get a good understanding of Git. They get a solid foundation for working with the kernel and then they're able to better understand the other abstractions and I think work with them even better. Well, we probably could get more sub-maintenors involved. That's an interesting idea. But what we have them do is, you know, how do you, you know, describe how to connect someone with a sub-maintenor on the part of the kernel or do you, you know, describe how to present yourself, how to interact, how to submit your patch, how to have that conversation or how does that part work? Well, we probably could get more sub-maintenors involved. That's an interesting idea. But what we have them do is so they're not, you know, they're submitting email patches against a base repository that's like our assignments repository. There is a point where they do generate patches right now. I mean, we're making some changes, but they generate patches against the kernel and they include that patch as a part of another patch, which is like a little confusing for them. And then they send it to mailing lists where, you know, other students have, as a part of their, you know, their grade, they review each other's patches and they get feedback from each other. It would be interesting to integrate more of the sub-maintenor kind of workflow, but I think that's an interesting point. That's not something that we've integrated yet, but it's a good idea. Yeah, it's, it's, it's, yeah. Repeat it. Repeat the question. Oh, yes, yes. So we partnered with, and you lift the challenge, I think is what it's called, is the question and the answer is it's been shut down. They're not letting other people in. Apparently, it's a bunch of bad scripts that people don't want to maintain anymore. And, I mean, we probably, you know, I'll talk to Greg at some point if I see him, maybe I'll mention it, but I think, I think they, they want to move on from that at this point, unfortunately, because it was sort of, yeah. Oh, really? Yeah, so, yeah, the response is, you know, that maybe there's a new model going on. Yeah, I mean, we've discussed this with Shua Khan a little bit and she was saying like she, you know, it's very, they're very protective also about the code, you know, because they use it to, you know, the source is easy to evaluate people. And they really don't want this stuff getting out there. You know, some of the evaluation stuff, you know, people all around the world trying to, trying to break this stuff and, you know, show that they can, I mean, unfortunately, you know, kind of falsify that they've actually completed some of these challenges. But, you know, it's interesting. I mean, definitely, it's definitely a interesting model, funny story. Someone actually mentioned they came up to me after I did this talk at Boston, right? And they're like, hey, you know, do you remember, I do lift a challenge? You know, I was like, oh, when was that from? You know, like, what? And it was like, oh, like, back in like 2015, 14, it's like, well, yeah, I was in high school. Like, I didn't know what was going on with Linux at the time. So it's a bit before my time. But it would be interesting to try and work with Greg if he's interested in that. But, I don't know. From what we heard from Shua, it's no longer operative. But there's no other questions. Thank you for attending our presentation.