 Hello, everybody. Welcome to our talk. My name is Don Sickus, a kernel workflow lead at Red Hat, and with me is Prere Bargaba, a Distinguished Engineer at Red Hat. We're going to talk about how we incorporated GitLab into the REL kernel workflow. Next slide. The REL kernel has a workflow that was established over 10 years ago to handle large volume of backwards. Using emails, a mailing list, and patchwork, we created processes to guarantee that change was tracked and reviewed correctly. Worked well for a number of years, and over time, like most workflows, we added more rules, processes, and volume. This led to the workflow slowly becoming the bottleneck. With the new requirement to publicize our workflow for CentOS Tram, an opportunity came to modernize this workflow. This presentation will talk about what we wanted to accomplish, how GitLab helps us, what we implemented. The hope is you can walk away, understand the difficulties of an email-based workflow, understand the new technologies we introduced, and hopefully be encouraged to migrate your email-based workflow to a Git forage-based workflow. Next slide. All workflows start with requirements. Ours was simple. Upstream first, all change is reviewed and tested, and that there is an audit trail of how the change lands in the REL kernel. What does that mean? Upstream first means all changes push the upstream community for review and acceptance, and then backport the final version. This allows us to leverage upstream expertise as much as possible. And all changes reviewed and tested kind of breaks out into two pieces. We have the review, which has two parts, technical review, and a business review, which is basically bugzilla. All kernel changes are gated on the acceptance of both those reviews. Each review process has complicated rules that the workflow needed to accommodate. And our testing is broken into two parts, automated functional testing, performance testing, and QE verification, with QE verifying changes that automated testing could not. And finally, audit trail means we need to ensure that we can understand why and how a change entered our kernel. This is necessary for various security certifications. We also track all posted changes to verify nothing was accidentally dropped or ignored. Next slide. Putting this all together, our old workflow revolved around a patchwork server and a mailing list. The maintainer spent most of the time tagging emails and patches with information that determined whether the patches were reviewed and ready to be committed or not. When automated testing happened on the side, it was manually coordinated to block the patches based on its results. A lot of manual work, but it worked great for a long time. Next slide. However, it had its struggles with the increase of volume, the amount of manual work was overwhelming. Every email was triaged and labeled. Lots of scripts run to generate the correct tags and move the patches along. Scratch builds and tests were not integrated into the tooling and developers had to wait for an email report that may or may not come to find out if their patches could get integrated. All those work was manually coordinated by the various kernel maintainers and after a while, the maintainers became the bottleneck. This had to be addressed. Also, the developer contribution involved emailing backported patches, following multiple threads for feedback review, verifying the various bugzilla settings and checking a status page to see if you did everything right. Explaining this to new hires was a challenge. The workflow was outdated compared to popular services like GitHub and GitLab. We had to address that too. Next slide. As I said earlier, the straw that broke the camel's back was new requirements. Red Hat shifted strategies and wanted Red Hat to have a proper public contribution path where non Red Hat contributor interacted with Red Hat employees using the same workflow. Red Hat calls this sent all stream. Taking a workflow and hardening it was going to be a challenge, explaining how to contribute to the real kernel was going to be more challenging. There are a lot of core nuances that we did not want to share internally. So we use this opportunity to rethink this. Modernize the workflow and make contributing easier. Next slide. What do we want this new workflow to look like? First, we agreed that it had to be a merge-based workflow. We needed a robust API to build CLI tools and interact with it. Kernel developers love their CLI tools. Having a web UI was okay, but not higher on our priority list. Having a built-in automated testing tool was a plus along with open-source friendliness and being easy for developers to use. Most importantly, it had it to drastically reduce the manual burden of the maintainer. Next slide. After a few weeks reviewing the tools and services out there, we concluded that GitLab met our needs. The service had merge requests at its core and CLI tooling built into its architecture. With its huge API list, it was easy to find and customize tooling around it. The service was open-source and already popular in the community. We spent some time verifying it could meet our needs. Our eternal CI testing service, CKI, already leveraged GitLab CI tools. So we knew that piece was solid. A few merge requests helped us understand what a developer and review contribution workflow looked like. We learned about web hooks and how to leverage them to customize the GitLab workflow to our needs. Finally, GitLab had defined structured rules for contributing, reviewing, and maintaining. These rules made us rethink our workflow and strengthen them. We felt confident this would be a good fit. We finally saw a path for Red Hat and non-Red Hat developers could easily contribute to the rail kernel where viewers had faster feedback on their contributions and maintainers could finally monitor the process and not be the process. Next slide. After months of work, we did it. The transition was not easy. Chained among developers is hard. Mapping all processes and new ones is hard. The world we better trust is not an easy sell. We started small. We educated a small team of developers and added another team and incorporated all their feedback and then rolled us out to everyone. We made it. At this point, I'm going to hand this over to Prar to talk about the technical stuff we had to implement to accomplish this huge task. Prar. Thanks, Don. Excuse me. As everybody just heard from Don, we had a lot of issues that needed adjusting or resolving in our existing workflow. As a result, we're attempting to completely overhaul the way we contribute to the rail kernel. We've changed the process based on input from the maintainers, developers, and QE. I think the best way to review the changes is to look at them from a point of view of a contributor to the rail kernel and what they now have to do. So I'm going to focus on the developer point of view for this presentation rather than look at the entire process. To start, our main user-facing bug and feature request tool is still bugzilla.redhead.com. We use this for all of our rail projects and across all of our OSs. We allow open bugzilla reports against rail and our other OSs like Fedora and CentOS. So typically we start with a bugzilla filed by someone for an issue they have with the kernel. An engineer is assigned to that bugzilla and debugs that issue. It's easy if there's a fixed upstream, they just backport that fix and get it into rail. If there isn't a fixed upstream, there's a little bit more work involved. The assignee has to create code and post upstream for inclusion into the upstream kernel before considering a fixed for rail. Ultimately, and in either case, the code is upstream and the bugzilla assignee needs to get it into the rail kernel. This process of debugging an analysis hasn't changed over the 15 years I've been doing kernel development at Red Hat. And it's after this analysis step where we've introduced a new process. So as I just asked, how does that code now get from upstream into the rail kernel? And I'm going to also throw in a new question. What are the requirements for providing a backported patch for submission into rail? We have a document called commit rules that details exactly what a backported patch must look like. It has guidelines for referencing a bugzilla, the format of the backported patches title description and tags like the CC signed off buys tested buys, etc. We also have a tool called get backport that formats an upstream patch referenced by its commit hash in a matter that will meet the standard standard set by the commit rules. There are obviously some things that get backport cannot do. It can't know what you did for a test. It can't know about patch conflicts. For example, the newly backported code may conflict with some other code that already is in route. And it can't know about the dependencies on code found in other bugzillas. But the format does conform to the general commit metadata layout specified by the commit rules. After backporting and testing contributors now must create a merge request. This can be done through get or get and the get lab web UI. Red Hat Kernel Engineering, however, recommends that the lab utility be used on the command line in place of those options. It has an intuitive and easy to use interface. For example, to push and create your merge request, you simply do get push your fork and a branch with your changes in it. Now you can follow that with a simple lab MR create. The lab project is open source is an open source project hosted at this link. It is not a unique red hat project and we've noticed other large groups and communities using it. There are two red hat engineers that currently have commit access to the repository and both have been active in making and reviewing enhancements. When the when the merge request is submitted, several things happen at once. The web hooks run on the merge request, CKI starts and the email bridge sends out patch emails. We're going to go through each of these in the next few slides, but first I'm going to turn it back over to Don to give us a deeper look into the engineering behind the web hooks. Don, are you there? Go ahead. So thanks for web hooks. One of the key features about GitLab for us was its use of web hooks to deliver events. These events are in response to merge request comments, CI pipelines, issues, etc. Using these events allowed us to tailor the workflow to our needs. However, web hooks are a one-time event and with the volume we anticipated we need to make the service more robust. In addition, we need to be able to access data behind our firewall securely. So we created a RABDM2 service on AWS that queued up all our events and delivered them to registered containers on our internal OpenShift platform. These containers processed the event, had safe access to internal data. Once processed using an API token, we could then update the GitLab merge request with comments labels or status. This allowed developers, reviewers and maintainers immediate feedback that once was a manual triaging job. Back to your prayer. Thanks, Don. So as Don just said, we've created a sophisticated set of web hooks that run on merge request creation and other merge request events. These include the bugzilla web hook, which validates the bugzilla state and flags, the commit comparison web hook that compares the commit against the upstream commit, the DCO web hook that confirms each commit and the merge request description contains a signed off by that acknowledges the developer certificate of origin. The subsystems web hook that tags each merge request with a specific subsystem tag like PCI, ACPI, networking, individual drivers, etc. And lastly, the kernel watch web hook. This is a brand new feature. This project allows users to uniquely identify areas of the kernel they are interested in. For example, you might not be interested in being a reviewer for PCI, but you might be interested in the high level PCI driver changes that are reflected in the include Linux PCI dot h header file. On the other hand, you might be someone who wants to get experience and learning and reviewing an area of code and adding an entry to kernel watch allows you to watch that area of code. Kernel watch adds a comment to a merge request so that you'll be notified via get lab at notification on the merge request that changes the files that you're interested in. Continuous integration also runs when a merge request is submitted or updated. We have a sophisticated test suite that now runs part of the get as part of the get lab process and starts exactly when that merge request is submitted. Previous to this, it was run separately and current developers had to watch their email for results from the CKI subsystem. As can be seen on this slide, CKI results can be queried directly in the merge request. CKI will build RPMs and test changes across all supported architectures with a few minor limitations. For example, we don't build the debug kernel for all our architectures. Users can download the artifacts from the tests, the RPMs that are built, the logs and tests and review and use those locally. To be clear, however, CKI must have a successful run in order for the code to be included into the rel source. The last project that runs on a merge request submission is the email bridge. It is a utility that converts the commits from a merge request into an email patch set and sends them to a list. Right now, the current rel kernel development process provides for email reviews of patch sets, just like it's done upstream. So it's necessary to have these emails for review. The email bridge will use the merge request title and description to create a cover letter for the change set and send it to a preconfigured email address. Reviewers can then act, positively acknowledge the changes or knack, reject or request changes. This review process is currently all done in email. Since that email review is done outside of GitLab, we do have an additional web hook to track data. It's different from the others in that it runs on all comments made on the merge request. We call it the act knack web hook. This web hook processes acts and acts from reviewers in email. The email bridge applies them as comments to the merge request and ensures that the appropriate people from RH maintainers have reviewed the code. At least two reviews are required. If a merge request spans multiple subsystems, then at least one reviewer from each of the affected subsystems must provide a review. We have some additional changes that we need to make as we continue through this transition. We're currently using, as both Don and I just explained, email reviews of patches. The email review process needs to be converted to using GitLab's built-in approve and blocking threads model. An approve would replace that email acknowledgement, and in order to block or reject code from inclusion, a reviewer would now open up a blocking thread on the GitLab merge request. And also, to aid in reviews, Red Hat has identified an open source tool, which happens to be maintained by Red Hat or Beeshawn. I won't be able to demo it today, but here's a screenshot to show that it's also command line based and takes design hints for month. We are going to be using this process for the CentOS string kernel project. Contributions will be a bit limited at first as we figure out the process and make changes to it. Watch centos.org for announcements about the kernel. If you're interested in trying things out today, you can also try the Fedora kernel project, also known as kernel arc, at this location. We're planning on rolling out additional changes every few weeks. We have other problems to solve, dependencies between merge requests that might be solved by merge trains, data retention issues, et cetera. We also have several feature requests that need to be resolved by GitLab. For example, restricting editing comments and restricting those who can block a comment. With that, I'll hand it back over to Don for closing comments. Thanks, Pratt. Thanks, Pratt. As Pratt said, we still have kinks to work out with our workflow in GitLab. But overall, while a transition was difficult, the pros clearly outweighed the cons. We have a faster workflow, cleaner rules and more tightly integrated processes. Hopefully, others who had thought to transition to GitLab have a better understanding of what they will get themselves into. And for folks who would like to get involved with contributing to the real kernel, we are now finally ready for your contributions later this year. Thanks. And any questions? Oh, actually, we have one slide. We have links here for all the projects for folks who want to get involved or understand our technologies. And with that, we're open to questions. Yeah, so there is one question. Will GitLab's issue be used? If so, how does it cooperate with Bugzilla? Don, do you want to answer that or do you want me to answer that? So for the way we've decided all source content will use Bugzilla. That's how Red Hat tracks all of its issues and problems for various reviews. However, for workflow issues, we decided we're using GitLab issues for that. The reason being is that workflow issues are, it's easier to track in GitLab because that's where all that code is. And it's kind of a little bit outside the business purview of what we need to do. Perr, do you want to add anything to that? Yeah, we do acknowledge that there's going to be some confusion about that reporting issues on workflow because people tend to think of the workflow in terms of their code. But we hope as time goes on, it'll be a pretty smooth and natural process to separate actual code issues as it were from process issues. Thank you. There are other questions. We have still a lot of time. So guys, feel free to post other questions as well. So here it is. Do you plan to go away from the email-based review system for that as well? Yes. That's ultimately our goal. Sorry to cut you off, Don. That is ultimately our goal to drop email reviews. We recognize that this is a difference from the upstream community. But the time has come to make this leap forward. The upstream community for two years, maybe even three, has had discussions on migrating to a Gitforge-based workflow as well. And just to add, we recognize this is a difficult transition, especially for developers. Therefore, we have this email bridge as a crutch to kind of allow us to develop the necessary tools and mature them such that we can transition successfully. I think the name bridge is very interesting in the email bridge. It's meant to be a bridge between a mailing list and GitLab. But it's also going to be used as a bridge for us to migrate people from their old email-based workflows to a GitLab workflow. Thank you. There is one more question from Denise. What was the most difficult part of all this? There was a lot of chat. The mind shift. How about that? Getting the developers to understand what GitLab can do. I was a component of this with my experience with various projects in the CTI team and convincing the kernel developers that we can use something besides an email framework was a big mind shift. And as I said before, the transition was hard, but you take it step by step. I grabbed a small group of kernel developers. I convinced them. They started like prayer and a bunch of red hats at Denise knows. We started there, convinced them. They started to understand the GitLab power, the power of GitLab. And they brought it out to some more people and brought them in to now we have this critical mass of people that understand how great this workflow is going to be. And then it's easy to spread the word from there. So for me, that transition was hard, harder than a lot of the technical challenges. Excuse me. When Zika's first approached me about moving to a Gitforge-based workflow for the kernel, I didn't exactly laugh in his face, but I came close. I am a true convert to it. I think that Don's right that it's mind shift. And one of the problems that we faced and we're still facing as we're rolling it out is that kernel engineers are used to a very simplistic use of Git. They check out, they clone a tree, they do some stuff, they create patches, they send it to an email list. Explaining to people how forks and clones and local copies work. You know, explaining what the GitLab process really is, is actually the hurdle that the biggest hurdle, I think that we've had to cross to these. My biggest hurdle is the weekly meeting with Zika's. I couldn't resist on, sorry. You're the expert. Yeah. There is one more question. Have you looked at tying commits to tickets, bugzilla, to enforce traceability of commits back to the origin ticket? We do that today. Every commit has a bugzilla metadata in the change log. That's part of our commit rules requirement. And for kind of that reason, and that ties into our audit trail, too, is we need to verify why we have every commit in our tree and where it came from. Bugzilla provides why it came into the tree and that it was approved. I mean, otherwise anything that comes in without a bugzilla is easy to argue that it wasn't approved and it slipped in and that's a red flag in our case. Now, there are some red, there's some process, there's some workflow stuff that may come in without a bugzilla, but that's just behind the scenes it doesn't impact customers. You're a big fan of bugzilla. Do you have anything else to add to that? No, that's exactly it. Perfect answer though. Okay, there is one more. What was github considered as well? FAA open shift uses github. Yeah, we did look at other github forages. Garrett github. We had already started some work with CKI on github and that was one of the reasons we went, we ended up moving towards github, but there were other reasons. That github, github generally as a company has more of an open source view, I think, than some of the other corporate proprietary gith services. So we really, I really kind of latched on to that, that when I found out that github is really truly an open source company, it mattered a lot. We could have certainly gone with github. I don't think there's any huge reason overall. So the kernel workflow wasn't the only thing that's using github. We were also at the time working with, we're trying to bring out this sent Austrian project, and they were trying to evaluate how they want to deliver their content. Pajor, GitLab, github, and they stumbled upon, GitLab was something that made sense to them, sense to them and their community. And we're actually trying to work with the Fedora community too, to see if GitLab would be a good fit to update their workflows too, or if they wanted to continue using Pajor. There's some concerns that Pajor wasn't able to keep up with the future requests that Fedora infrastructure needed. So GitLab seemed to make a good sense for all those three projects. And as Parat said, CKI is in GitLab already. It seemed to be a better fit than github. Do you think that this approach is going to be adapted by kernel maintainers outside of Red Hat? So, let me answer that. Let's both answer this one. As I said just a few minutes ago, we do know that the upstream kernel community has been looking, has been investigating using a Gitforge approach. And what we discovered when we went through that, those emails and saw some of the discussion on that mailing list about using a Gitforge, they had a lot of problems that we have. How do you handle reviews? Like how do you acknowledge reviews? How do you reject reviews? How do you make comments? How do you comment on a specific commit, etc.? We have a lot of the same things. And what we decided to do was maybe step out for once a little bit more forward than the upstream community, solve those problems. And now we'll be able to go back to them and say, well, this is what we did. And you have pretty much the identical problems we have. Will they adopt GitLab per se? That's a question for the upstream community. I can't answer that. Will they adopt, maybe now be more open to adopting a Gitforge-based workflow? Yeah, that's all right. That's definitely one of the intentions of this project to show them the way. I wanted to expand on that. Part of my role is I'm on the board of the Kernel CI project at Linux Foundation. And the mission there is to stabilize the Linux kernel, Linux stable in particular. What we're trying to do is we're trying to get the upstream maintainers on board with using CI systems, either Red Hat One or just 10 other systems out there. And try to use them to gate their commits before they propagate up the Linux history. And so the question, the struggles they have there is how do they gate all the changes there? We developed technologies. There's a talk about KCIDB by Nikolai that talks about what we're doing there. But you start to think about these maintainers is that how do they really start structuring all this stuff? It's like Pirates says, do you want to start tying in CI systems? We went down that path with an email-based CI system reviews. And you still have to manually triage and coordinate everything. Is there a way to integrate that and make it a faster and smoother process? And we want to show these maintainers that we're doing that with GitLab. And they should look at that. And there's various upstream communities that do that. There's a DRM is a popular choice. They're exploring GitLab and using that successfully in their workflow. So that's the point of this effort here is to show these sub-maintenors that there's opportunities for moving forward. Patchwork came a long years ago and that was a great step forward for these maintainers organizing their stuff. And now we're at another point, an inflection point saying, hey, you know what? Maybe it's time to take another step forward to either Git Hub or GitLab or another source forage that really truly starts to integrate some of these CI CD ideologies into your workflow.