 Welcome So yeah, I Hope you are all here to contribute to Cento Vestin Colonel And maybe not maybe you just like get inspiration Or maybe get scared away by what we have and what we have built Anyway today, I will explain how people external to to the rail or Red Hat Colonel teams can contribute to the Cento Vestin and by means of Rail taking the Cento Vestin to the rail colonel So first I will spend few words about what is the Colonel package in Cento Vestin It's a package as any other And right Except it has some specifics So it's it's rpm. It's basically it's actually set of rpms But the specifics are that the source repository for the Colonel the the The repository where the work is done is not the repository from which the rpm is built This is different to most packages in in Cento Vestin So it has separate Repository more than that later second difference is that the work on the Linux or the Cento Vestin colonel is Parallel so imagine like bunch of well number of surgeons Work in operating on the same body at the same time One of them is like maybe updating the circular system replacing arteries with something better While another one is replacing a leg At the same time. So there are some challenges. Yeah, especially when the arteries go to the leg They have to coordinate somehow So yeah, just to just to give you an idea how massive it is I checked yesterday So currently we have 149 Merch requests opened for Cento Vestin Colonel 9 Yeah Some of those have several hundreds of commits in in them Of course to have all of this to be manageable The the Colonel teams have to establish some some processes around that because yeah, it wouldn't work otherwise So there are custom rules Custom processes that needs to be adhered to if you want to contribute on the other hand the the benefit of all of this is that We basically need a dedicated person to maintain all of that to merge the stuff to take care of the pack of building the packages pushing that to the to the district to the to the repository from which the Packages are built. So we have a dedicated person or maybe even persons for that And they deal with the Cento Stream Processes themselves. Yeah, so the distribution have some processes But the contributors to the Colonel don't have to deal with those So we basically exchange one set of processes once of complex processes for another set of complex processes Yeah So this is the introduction in order to understand what What the work entails what are We basically constrained by let's first talk about the expectations So let's take a user user of Cento Stream user of rail. Yeah, what they expect from the Colonel Of course stability. What does that mean? That means no regressions Stop that work before should work after a colonel update, right? Well, it's more it's performance No performance regressions The user probably wants After update for this for his regular for their workloads to work as fast as they did before Yeah, so performance regressions. No, no No API changes, of course if you have a program if and then update the colonel You still expect your application or program to work. Yeah, and basically every application interacts with the colonel So the API is that the colonel provides the application should not change at all. Yeah, right? This is what you expect Well, then we have kernel modules You may be using third-party kernel modules and we the driver for example for your graphic card or whatever And you want those drivers to work after the update, right? So to the kernel update So the problem is that those modules those kernel modules the current drivers they are using the internal API of the kernel So you want this? Application binary interface between the modules and the current to not change at all, right? Well, it turns out that some people Want more not just the binary interface. They want the also the internal API Which means I take the driver source code From the old version update the colonel and compile it against the new canal and it should still compile Sounds seems sounds like the same, but it's like it's actually very different thing No behavior changes, of course. I want the stuff to run exactly as it was running before the update So yeah in shirt do not touch the stuff. I'm using Yeah, redhead, please keep your hands off the stuff my stuff and also Please bring me new features Yeah, we want the newest Chinese. Yeah, so please like yeah, give us what we need There's that new hardware we bought we want it to work in your next release So please update the drivers for us. Oh And we also heard about this cool new feature that upstream kernel has so please bring it to yeah Update the core update like yeah, dude, please update it. Please update everything you can Except for the stuff I'm using Don't touch that stuff Yeah, now do you see the problem that my stuff is different thing for different users So everyone wants to update everything except for this stuff. So this is the situation we're in We need a compromise. Of course, we cannot deliver all of that at once So the compromise that we arrived to It's a compromise. Nobody is happy about it But apparently people are still buying rail and using CentOS stream. So it's not that bad So the compromise is no functional regressions if something worked before it should work after the update Performance yeah, it's now we should not regress performance But imagine cases when you do something and you improve performance for 90% of users by a lot Didn't touch performance for 9% of user and for 1% of users you slightly slowed down There there were clothes. Is that okay? Is it not who knows? Yeah, so there's some kind of balance So yeah, but Probably slowing down networking for 90% of people or slowing down this story treats. That's probably not acceptable Then yeah, of course so the the user space API that the kernel provides to application that should not Not change, but it should not break Yeah, so we can change it if we maintain backwards compatibility probably or if nobody notices. That's okay as well, right? the Colonel modules Yeah, they should probably work but Here is a problem the the the kernel level of land upstream and the the community kernel it doesn't Doesn't care about internal API stability. It changes all the time the interfaces inside kernel and to the drivers They are constantly changing so if we are to Bring in some new features. We cannot keep this interface completely stable We can try to maintain some backwards compatibility, but it's like a lot of work and nobody's enjoying that work Cost a lot. So yeah, so we have to limit somewhere again. It's it's about finding a balance We also Are updating drivers Not all drivers obviously we don't have infinite people working on on the kernel But we are updating drivers, but again Sometimes updating a driver means bringing in a new kernel core feature that the driver is using So there are dependencies And it's not always easy because sometimes bringing in some new feature from a steam. It's it's it's risky Yeah, we risk stability. We risk introducing regressions. So again, it's kind of balancing act So we might update drivers, but maybe not Enable all the features that the hardware support supports and of course we updating the the kernel core features again Balancing that with stability not everything is possible without compromising the other stuff So yeah, as I said nobody is happy, but hopefully it's good enough for everyone How Does this work in practice? When we started and now I've been talking mostly about or basically only about so CentOS stream 9 But it applies to also the other other streams So for CentOS stream 9 we we had when when the development started there was a kernel was 5.14 and The CentOS stream 9 is still on this kernel Yeah, even those years later. So we have kernel 5.14 But we're back porting stuff for upstream. We are taking comets from upstream and applying them to the to to the kernel And that ranges from single bug fixes for like simple stuff single features to like a large basis of whole all drivers and whole subsystems So for example for you to know what I'm talking about the XFS file system is up to the upstream kernel 6.0 the USB subsystem all drivers the core like everything is Up is on par with upstream 6.2. You know the latest kernel in the upstream is 6.3 So this is pretty new BPF subsystem. Yeah, this is pretty core stuff That's like connected to really memory management and all of that It's 6.2 in the kernel in the current CentOS stream 9 wireless including all drivers 6.3 multi-party CP 6.4 that kernel is not even yet released upstream Yeah, so you see all the different stuff is Kind of suits into the kernel. You have replaced different version and yeah So maybe you get the Frankel kernel Temminology now or technical term now. This is what we have It works it works but oh One thing one thing I forgot and I will stress its multiple times We are taking stuff for upstream. That means everything almost everything We strive for everything that we have in CentOS stream to be upstream. You're taking stuff from upstream Everything is to be upstream first so as I said all of this can Stick together and work reasonably only thanks to Thanks to Several things first is the processes we have around that I will talk about in a minute and the second and very important thing is testing So this is the testing that happens Whenever a single change Be it a single bug fix or that large rebase whenever it is applied to CentOS stream kernel first testing Happens when a developer does the work. So when we back port stop from upstream Yeah, or when we develop stop upstream and then back for it. So this is where the first testing happens Yeah, obviously nobody wants to just push over the fence untested stuff Then Automatic testing kicks in. Yeah, we have CKI There were talks about CKI in previous devcoms. If you are interested you can find the recordings online. So this is Continuous kernel integration testing large test with a lot of infrastructure around that. So this is this is run We have an a Steve Linux network stack testing Which is specifically for testing networking perform both functional performance and We're adding more over time So this is run automatically then there's pre verification which means before the change can be merged to the kernel a human and Hopefully or maybe automation if it's there if there is automation for the particle feature or particle bug But someone someone else then the original outer and it's this is important Tests this and if it passes only then This is merged to the CentOS stream kernel Then there's integration testing because it's nice that you tested one feature or one backfist in isolation But it can interact with those other 150 Parallel changes to the kernel in weird ways. So once this is merged We need to test again to see whether it did not conflict or did not like good influence by other stuff in negative way and last thing is proper QA testing done by QE engineering. That's like really comprehensive testing of the whole kernel and all the features and I think so that's a lot of testing Okay, so bit that good. Let's Let's Look how it looks from the point of view of the contributor to CentOS stream kernel The centerpiece is the merge request We're using githlap for CentOS stream not just CentOS stream kernel, but the whole CentOS stream development So this is the URL of the CentOS stream 9 kernel source repository. So this is where the development happens It's completely public you can go there You can watch what we are working on and of course contribute. So merge request Against this is repository. That's that's basically the the the the centerpiece Which around which everything is built however before You can Open a merge request you need to do one important thing and that is to file a bug or issue Every change we do needs to be tracked So you go to bugzilla or jr. Doesn't really matter File a bug or open an issue there Be sure to select the proper product component subcomponent Product Red Hat Enterprise Linux 9 Component obviously kernel subcomponent in bugzilla. Whatever you think is the best fit The reason for that is you want your change your merge request to reach the the proper team and In the bug or in the issue explain Why you are doing that explain the benefits of it Why it's important If you if you come as a contributor to CentOS stream kernel, it's not The the amount of work That is connected to the computer contribution. It doesn't end with Submitting patches submitting commits There's more just testing I talked about someone has to do it and there's maintenance Someone needs to maintain that stuff for the all the whole period where Well, that's my edit edit Enterprise Linux is maintained. Which is many years Yeah, so Submissions don't come for free for red hat So explain benefits explain why the change is good and Also, because it means work for other people Try to make it easier for them provide detailed testing steps Yeah, so the so the QQ QQ quality engineering Is okay with the change because they know okay. This won't be that much work for us Don't forget to to note that you will be submitting the Mange request So you are not asking the developer to do the work Yeah, make it easier for them and Be mentally prepared for the option that it might be rejected So we have a bug or issue Then now let's do the work We clone the repository as usual from good lab Starting create our own branch based on the main branch This is where the like the main branches where the where the development happens Apply our patches Now let's stop at that The and now this is the first the first thing about the processes the The commits the commit messages They have some mandatory format. I will explain it in next on the slide So it's not like freeform commit message So that's first thing second thing apply upstream commits one to one So one upstream commit is one CentOS stream commit do not squash commit together and of course It's a Frankenkernel, you know, so it's unlikely that commit from upstream will apply as is to send to a stream kernel Most likely or some commits do if you're lucky, but most likely you will have to do some changes Adjust to the old APIs that are there or something Those changes need to be explained. Yeah, there should be like reasons why Your why the the the backport differs from the upstream Let's look at the example. This is a real commit from CentOS stream 9 and Let's see It's first the mandatory fields mandatory format. There must be a bugzilla or Jira link in Every commit every single commit must have this line and In this format. Yeah, it's it's parsed by machines. So even like training white space is a problem. Yeah You will be taught so it's not like It's not entirely black box more on that later. So this is the first thing second thing every as I said The the police there is upstream first policy So everything that we bring to a CentOS stream kernel must be upstream this line actually says This is the hash of the upstream commit. This one is backport in Yeah, so and again exactly this format. It is parsed by machines Third thing that's mandatory is the signed-off byline which has some legal implication If you're not familiar with the certificate of certificate of origin see the documentation must be there and Here I will point out this section. This is not passed by machine. This is for human consumption for reviewers Yeah, so this is free form except the conflicts line that like must be in this form Explains what was changed compared to upstream? Yeah, so this is this is an example how it looks like So you apply patches and then you can finally submit submit Merch request. Yeah, the usual GitLab workflow nothing you Are not familiar with so basically you fork So at remote push to your remote open match request using web UI or API and you're done almost In the match request description, you have to repeat two things the bugzilla or gilaring and your sign-off byline when you do it then the automation Kicks in on GitLab A Colonel is built and it's tested if you are an external contributor Then then only a limited limited tests are run It's not not full test. It's for security reasons Once your submission receive reviewed by someone from Red Hat They will start the the complete pipeline We don't want the the code compromised by a External contributions obviously Then also some some automation is is run which Will do checks on the submission. It will check better the fields I describe like all that bugzilla entry and so on but it's present. If not, it will it will tell you in a comment in the Merch request It will check whether the bugzilla or JIRA issue filed whether it's in proper state There's something you cannot influence someone that has to put that in the in the correct state the bug or issue Another check is for missing fixes So if you backported something from upstream and there was a follow-up upstream that fix that you will get and it's not there It's not in your submission. You will get notified It will also check for conflicts between submitted much requests It is quite important given the amount of magic quest we have then There will be some labels at it Actually quite a lot of labels. So this is an example of one magic quest There are Subsystem labels here basically those indicate which Subsystem were touched or very modified by your submission. This is a large magic quest. Yeah It has I don't know like Three or four hundred commits. Yeah, so it touches a lot of subsystems There are labels with the current state like exnet needs review or CKI and so on It will also add approvals to to to git lab to the magic quest meaning The list people who should review the magic quest and it will notify them Then a review then review happens. This is important part But there's nothing surprising so someone from redhead or from community Will review the commits you submit it Will make comments Now it's important to to respond to the command if someone has objections or ask for ask for explanation Provide that the discussion happens in the in the magic quest. Yeah in the in the comments there and When finally when it's reviewed and I think I was ready The merge request is ready for QE testing. That means you get all of these labels in the okay or warning state Adversed so your commit description is okay. No merge conflicts. This the CI testing succeeded and The somebody or like all the approvals were given the all the needed reviews were done Then and bugzilla and JIRA is like in Prepared for testing state you get ready for QA Then the testing happens The pre-verification again done by someone else than you You will probably agree in the bugzilla or JIRA who will do the testing whether will be redhead or whether Someone from your company or your friend provides or your friends provides the testing whatever Then if that then is done, then you get bugzilla or JIRA. Okay ready for merge and boom It's merged or the main tender will match it One short notice Sometimes if the for example if there is a driver update it might depend on other changes in the kernel yeah, so if you for example update the Networking driver your favorite networking driver to the newest version It might need some changes in networking core to support new protocol perhaps or something So that's a different merge request. Of course. It's not the same stuff so different merge request needs to be filed for that and To save us time you can actually submit those merge request in parallel You can depend on other merge request, which is not yet merged by pulling the code Applying your purchase on top and Submitting that the only thing you have to do is put this line to the to the merge request Description that you are depending on this particle and other merge request and then The the automation we have will be able to figure it out Find out that like what is the first commit that is really part of your merge request So that's like top of the dependencies will put this label there and then you're good The only problem is good lap is not aware of that. Yeah, this is a feature that is missing in the lab So we out we implemented it ourselves and as a consequence. It's really hard to impossible to to use the web UI I Will yeah, okay 15 seconds We have tools that will help you with that lab is a CLI tool that helps with submitting merge requests And we have review metric, which is a custom tool that helps with reviews that understands the workflow we have So that's it Here's a link for full documentation process if you're interested in details and now questions Yeah, so the question was that by doing the backports We are repeating the work that upstream does with the with the LTS kernels And better it would not be easier to use them Yeah the problem is that the That the the quality of the upstream stable kernels is not satisfactory or a distance we found There are things like machine learning involved which selects which commits are back ported and also basically whatever someone thinks is Decisible for those table kernels strewn over the wall and applied and It does not mean this does not meet the stability and no regression warranty guarantees we need So we really want all the stuff that's going into that to get reviewed by us Yeah, and to be tracked to to have someone who's responsible that's like we can go back to say okay Yeah, look you broke it. You fix it. Yeah So all of it together You found out it's like for stability. It's it's yeah, it is basically no go Yeah, there's question Yeah, so the question was what? What is what the integration testing consists of what what what it is? Basically right now, it's it's sick. It's done it by CKI again, but it's like a larger battery of tests There's a lot of different tests with that I don't remember of the top of my head, but they are there's a TP and like many more The question was there whether they are open source. I believe so I'm not 100% sure of everything, but I think so And Please don't like yeah, I'm not sure I'm not sure. Thank you