 Hello, my name is Siaku Prešek. I'm Iñaki Malerba, we are both software engineers at the Continuous Kernel Integration as CKI, oh, I'm changing. It's actually our first time presenting at the DiveComp, so it's going to be well. Yeah, we are the CKI team, as we also want to call it, Cookie. We work testing kernels and kernel patches on our daily basis. As you might know or not if you've ever been on the kernel team, this is not an easy task. So I would like to start with a nice graph, or that it should be working. Things happen. Okay, so Internet problems. So I was saying, we do testing kernel patches and kernel trees. This is what kernel development used to be previously. On one side you had the developer making a patch. It used to take some time until the patch was ready, it was reviewed, and was sent to the QA side. On the right you have the QA side, which has the patch for a few days, this week in the base case, and checks and finds if there's any issue with that. If it happens to find an issue, it returns the patch to the developer's side. This probably a few weeks later, and the developer had already moved on with the development, was working on another patch, so we had to spend a lot of time trying to remember what the patch was about, what the bug was about, what the new feature was about, and find the new issue that the QA team found. So for this reason we realized that we wanted to push CI into the kernel development. I know probably most of you already know what CI means, but just in case, continuous integration is the practice of building and testing each change automatically as early as possible. This means that using a Git workflow, every time you push a commit to a repository, a pipeline gets created when your changes go through a build, some tests, and whatever other checks you have to do to check that your changes match the criteria. But how? The kernel is a huge project. You cannot move kernel development to GitLab or GitHub in the base case. You need to find out a way to put these changes on big communities. Usually it takes time. You have to prove that the things you are trying to put are really worth it. You have to prove that your ideas have value, basically. So this is our talk. It's not going to be about the details of what we did. We are going to give our opinion of the things we think were great, of the work we did, what we found really valuable. And if you get curious and want to know more about what we do and how we do it, I would recommend you to go watch the talk. Red Hat joins CI Party, brings cookies that was given by our partners at Linux Plumbers the past year. Yeah, so a lot of this is going to be based on our experience with various people from different companies, from the community. There was a hackfest last year in Lisbon that was organized by our colleague, Konika, and we think we learned something from it. This is kind of our view on what we learned from that and how it kind of plays with our experience of working on the project. So, prepare for cookie analogies. This is the first one. You have to provide the value as Senyaki just mentioned. So imagine a big package of cookies for a couple of cents. So what do you get? What do we have? How do we provide value? So we have the hardware. Some of you know this. We have Watt of Enterprise hardware, hardware that you cannot buy anymore. Sometimes just one piece of hardware or something. It's very special. And ideally we want to use that and test on all of it because there's probably a driver in the car somewhere, there's probably a bug in it, and probably someone's going to be very happy when they hear it. So we have a lot of VMs, GPUs, different peripherals, sound guides, you know. And also the value is, that's the fine box. There's a shared list here that wasn't updated at all. And obviously there's a lot of test suits that have been in Red Hat for a long, long time. And they haven't been onboarded, haven't been stabilized, but can still provide some interesting value. And another way to bring those values is subject matter experts, the developers, the maintainers of the tests that actually somehow help with the issue. They know this stuff. So the value, here's some kind of a culture, I don't even know what it is. Basically it's just about, like, we send a report to a mailing list and then there's an interaction. And then, you know, somehow realize as well, maybe it's a good idea to have our upstream kernel tree tested by you guys. So this is what we have been, we hope to continue improving this interaction, finding bugs, and doing good tests, I think real late. Oops, here's another example. This was a compiled error that was discovered very quickly. Thanks to the fact that our build took about, like, five to ten minutes on X86. So, you know, in ten minutes you can actually have that report. You push, you can get the tree somewhere, or there's a change in upstream, get repo. In ten minutes, you know, the maintainer has a kind of result. And our builders being, like, quite fast. So the example that I like the most is this one that was Greg, an upstream kernel maintainer, doping canary, and it got picked up by our CI system. So this one was just a big test that isn't supposed to be merged, that in kernel identity, any boot test should have that effect on the machine, and run the kernel on it. Some CI systems didn't pick it up, some ours did. So, you know, this kind of makes me, and that's just my personal opinion, but it makes me feel that really this good test cannot help us make a good CI, because, you know, with what we have in the infrastructure, we can run it every time. That's actually actually the basics of CI, I think. Here's an updated print-outable, or print-outable page for bugs we plan. It's being updated on GitLab. That's number one. So, it's another way, I think, that we can make the developer's, like, geese as well, use the appropriate code. So now I'd like to have my geese fault, you know, the tool, it's in a perfect shape, that's exactly what it should be. So there is a short list of tools, test suites to use, that are helpful, that work for us in some way. It may not work for the community always, it may not work for some people, but, for example, with the Linux test project, we, you know, we're advising, like, parallel people, when they don't know what to test, they put it there, or they can also do functional testing. And, you know, it takes time, learning takes time, changing your workflows is difficult, it's difficult to adapt, but eventually people get into that mindset, well, I know how to use it, I can actually have a testing diary if I need to. So that's good, that kind of works for us. You know, when interpret testing diaries, it's going to be around all sorts of people, three cells, all sorts of kernels, probably, and I kind of doubt it. We don't, there is a ground to you, we don't use OTP for. Another good tool is GitLab. It's all about issues that we have in Jenkins for automation. It's, yeah, that's where we have our pipelines, RCI, and also where all of our code is. But, yeah, there are others in the partnership that actually run the pipelines, generate the test recipes, and they submit that to Wicker. Wicker is our provisioning, basically, that's where our machines live. For those of you not familiar with it, it allows us to power on a few machines and how to manage them, use them in a somewhat responsible way, it's not perfect, but that's mostly what we need to do. And here's something to talk to you, you know, during the access, I think, cooperation with other people. There are multiple different ways to do this. There's lava, there's ironing, which does roughly the same thing. But, you know, to be able to cooperate, no one's going to just, you know, drop on their view, so maybe there's a way to at least make sure that we get some hardware labs to know how, have one way to submit what we need to test and to any kind of testing setup without, you know, just using that resource, just consuming it. There's Wicker, a test speaker, which is, you know, kind of works for us. So if we have our tests that we need now, I'm not sure you can also, and that you can actually do. I don't think you actually need Wicker for all of those tests. There's a link file. I know that those tests you can basically separate it, you can look at that it doesn't really need Wicker at all, if it does, well, yeah, so it's not too hard to use to produce those results, I think. Another thing which kind of works for us is KPDB, that's where our test material lives, and that's things like, you know, what hardware it does, what environment variables are we using, how do we constrain the test to, like, a specific runtime so we don't burn resources. Yeah, I'm kind of hoping that we can open source this because there's some hard-earned data that we, you know, gain as we kind of work in these tests as we move forward and add more of them, and, you know, having some kind of security to go in there and be nice. It would be helpful. Very interesting. Another tool which kind of works for us is Petro, it's a patch-tracking software to use, and, again, you know, there are some obstacles, for example, you kind of have difficulties merging the changes you need, working with the web suite containers, but, you know, it's good to get it sorted out somehow. The last tool I have mentioned is kind of Cajun built, which, in a traditional way, we built our kernels to be, you know, our internal runners are faster, so I'm kind of hoping to fix that up. So, under the way, it's two depth books, but myself a little bit. So, most of the interaction is a little bit like, hey, your kernel is testing, and they're saying, okay, what is this crap? And this sometimes happens because they don't write the tests, it's like, you need to find another person that's not already in the company, and because of that, it's kind of, it's hard to give them trust, you know, trust like we're doing, and because of that, you know, we try to improve that interaction. A way that happens is kind of having a solid foundation and being able to for people to log in somehow, and that's, oh, sorry, I forgot that. I don't have a good picture, so I get lost. And, yeah, this is no good picture, so you can look at it pretty quickly in that. The good way to plug in into that is PPPDB, which I already mentioned. It's, again, where we test the disabilities and where we decide, oh, we're going to use this test and this architecture because it makes almost no sense to run it on this. There's some description of the test suits that are being used. So there's a tiny bit of takeaway from this, from all this, from these two cookies, and basically, you know, there's a lot of like tools that we use that are already mentioned. Basically, I'm hoping that if you use something for kernel, kernel testing, like, you know, colleagues, friends, that you can do, or acquaintances, then, you know, just think about, like, what you're building, how it interacts with the community, what are the inputs, what are the outputs, and if someone's going to actually be able to use that. You know, we're trying to make it so that this is something that obviously we use as well all the time. Okay, so, well, I don't know if any of you have ever seen a kernel mailing list, but those places are really crowded. There's a place that hundreds of developers use to communicate, to share patches, to review, to accept patches. So if you want the people to take your work into account, you need to be very careful with the mails you sent. Spamming a mailing list is not anything that it's going to be good for anyone. So we developed a few tools and used a few strategies to keep ourselves from pushing spam onto the mailing lists. For example, one really nice thing that we have, it's like, we have web tests. A web test is something, it's a test we know it's not ready or we don't really trust, so we waived it, so we ran it on the end of the test so we prevented it to affect other tests or the stable tests in some way. But also, we don't use those tests to calculate the final result of the stage. For example, you had 20 tests that passed but one web test that failed, that failed test, it's going to be reported, but it's not going to affect the result overall. The stage is going to be passed anyway. Also, when using a beaker pool, you have hundreds of computers and it's really common to find a broken machine or something that's going to affect your tests. So, well, someone has to go and manually fix it or open a ticket to the team that it's in charge of fixing those machines, but we need something to act quickly. So, we maintain a dynamic list of what are the most likely to fail hosts, so I try to avoid running tests on those hosts specifically. But also, for example, when putting a new kernel tree to test, you don't know how it's going to behave if your parameters are well-tuned or what. It's easy to put a new tree but every tree is kind of different, every kernel has some differences. So, we first want to have these trees under review. That means that every failure that the pipeline generates first has to be manually reviewed by a person, a member of our team, and then if the failure is a real kernel failure, a bug, or whatever, the report gets approved and it sends the mailing list. This way we avoid sending infrastructure problems or our internal problems to the mailing lists. But, well, anyone eating cookies doesn't want to have to open many packages and layers of packaging to get to the cookies. The kernel maintainers doesn't want to have to open 10 tabs to get to the test results or having to dig through emails. So, we try to keep our reports as short as possible, provide only the results that matter to the developer. For example, it's amazing. I mean, we really like to tell people, yeah, we run these thousand tests on these many machines and architectures and everything went well, but the developer doesn't care about that. They need what failed and where to look the failure, they want the tests, they want the logs, they want something that will help them to reproduce these errors. So, keeping the reports short and concise is not something easy for everyone, but it's really something that we got from our previous times that we get together with the kernel maintainers and that was one of the feedbacks, that they just want results, not bloated emails. But the work doesn't end in there. We send the reports, but as Jacob said, these testing is new and the kernel developers, the maintainers, don't trust the tests. So, they always come back asking for questions. How did we test the stuff? How they can reproduce it? Where are the logs? So, we're always trying to have a big interaction with the mailing lists. There we have people made in Veronica, always replies to the mails, to the maintainers explaining, showing the logs, showing the failures, trying to point to the code or the patch that introduced that failure, that it's really important because the maintainers feel this feedback from ours. It's not like we only say, your patch failed, go look why. We try to be with them and to give closure to the issue. Okay, it's a bug. Okay, let's go. Let's open a report. Let's follow this report and try to get it fixed and make sure that the bug is fixed and if it's an infrastructure problem, we will obviously go and fix it. Also, in the Linux Plumbers, we had a hackfest and we realized that many people were cooking the same cookies as we were. There are a lot of people that love their cookies and they want to share. So, now you see on the mailing lists a lot of different groups posting the results to the mailing list of their own tests. So, we realized that we needed a common place to put all this stuff. So, we're constantly generating a lot of data. We have tests, we have builds, we have merged revisions, we have passed and skipped and failed. So, we needed a place to put everything, all the information we generate. So, internally, we developed what we call the Data Warehouse. It's a place where we put all our stuff, all the information we generate. It's there. It's a place that we hope our developers in Red Hat can go and check if their patches were tested, if they failed, why they failed, get the relevant links and logs. Also, get some statistics that help us identify failures, for example, tag these failures, link these failures with upstream failures or with infrastructure problems or whatever, it's like a big overview for the work we are doing. But also, we are working with Kernel CI, which is part of Linux Foundation. We are making this big Kernel CDB, which is a place where everyone who's running tests for the kernel can go and push their data. So, hopefully, someone would take good care of this data. Someone needs to make metrics or statistics or get information about how the kernel is doing. So, this is the place where we want everyone to push the data on a standardized format, which is not easy. Everyone is testing different on different hardware, use different terms for different tests. So, we are trying to push forward to unifying these standards and getting the best information possible. And, well, someday, probably, only send one report to the mailing list showing the results of everyone who's testing the kernel, which is what kernel maintainers need, one result that has information about the test. So, well, what's this about? What does this pipeline look like? We said we weren't going to dig too much on the pipeline, but we wanted to show you what the pipeline looks like. This is a really short description on what we are doing. On top, you can see different sources like Patchwork, Git, Koji, Copper, Git Hub, Git Lab. Patchwork is what we, as Jacob said, what we use to track the mailing lists. Tracking mails is not easy. It's a text formatted for humans. So, we use a tool that is called Patchwork for that. Git are the Git trees. For upstream, we are only testing some maintainer's trees, official Git trees. Koji and Copper are internal tools, Git Hub and Git Lab. It's because as we have all our, most of our code on Git Lab and Git Hub, we want to test every change. If you hopefully submit a change to our pipeline, we have a way to run the whole pipeline on our infrastructure and to test if the changes are correct or not. All this generates what we call triggers. The big hack we have around Git Lab is that Git Lab CI is planned to run CI on the source code that is hosted on the project. But that's not the case, that's not the thing we're doing. We're using Git Lab to host our pipeline, but we are testing upstream code that is on another tree or on another source. So, we have to get the sources and do that. All that makes magic, it's on the triggers that generate a pipeline depending on the source of the change. For example, we have the Lint, which runs a few tools that check that the patch is correctly formatted according to Red Hat standards. So, that's only executed when a patchwork trigger is created. Also Merge, Build, Publish. Publish is because we want the maintainers to be able to check what we did. So, we also publish the kernels we built. It's like we merge all these, we built it, but just saying it is not enough so we are publishing the kernels so the maintainers can't really test what we did. And the test, well, the test stage here, it's in Git Lab, but in fact, the test stage, what it does is uses kernel and K-Pet and K-PetDB to create a list of hosts where it wants to run in Beaker and send the tasks to Beaker. When the test is finished running, the results come back to the Git Lab and the pipeline finishes with the result depending on what happened on Beaker. As I said, we sometimes have a review stage depending on the configuration of the tree. If the review stage is there, it's going to wait for a member of our team to review the result and approve it or not. And when everything is ready, it triggers a report. The reporter grabs information from all the stages, some information about the merge, about the build, about the tests and generates this big email that we send to the maintainers with the information that they need. So here's a short slide about collaboration. The first one being like what we can do for you for the community, for us, for the community. Well, we can test more of the trees in tiering. Actually, it's a much more simple process than I thought. You basically just open a VR in the lab and in tiering, that can get merged. I don't know if official approval is necessary, but most of it can go pretty easy, especially if those changes from that need to get merged into something like main lab, where we're going to hit the issues anyway eventually. So it's really better to grab that load and go into some of the problems as soon as possible by doing just that. The other thing you can do is play around with tests with each build. See, I don't think it's ever truly appreciated until something goes good. And the last thing, yeah, the data that's been generated, that's been pushed free into KCIDB, that's something also you can build applications on the day at any reason to process for current and all statistics like that. So how can you help? Well, you can help improve the tests if you like, and the speaker, you can help report bugs or report issues if they have to. And that's it. Now it's time for a Q&A session, but we're going to be asking questions. No, just one. I'm curious to know who of you have never heard of KCIDB before and could use it in theory in some way or know a person that would benefit from this? Please raise your hand. So everyone here actually already knows KCIDB. That's great. So that kind of renders my full-up question kind of invalid because I was curious to see what the barrier of entry is, how difficult it is for people to interact with us in any way, you know, to make a PR and some other projects, whether it would be for dust, and back on definitions, I think. So, now I'm certain if you have any questions for me, I would really have to answer them. That's the Laura. There are a few challenges in staging the kernel builds because it takes about, like, I have a 24-4 machine. It takes about, I think, seven or eight minutes or so. But I imagine because it takes that much CPU time to build, like, one kernel, is there any, how is that staged? Well, so I'm going to try to repeat the question. The question was how is the kernel building staged? To answer that, basically, we have builders in OpenShift and we can add CPUs to it. It's basically virtual, so OpenShift really helps us to be able to tune how much power we need to make that build. And there's a lot of questions being done with the builders. Thank you, Laura, you can make your rest of the team before getting into, like, five minutes. Yeah, luckily there's more people here of the team that will be happy to provide more information about that, but basically it's, I think, the secret is scaling as much in parallel as we can. We don't use one machine for all the building and testing, we just use OpenShift to get virtual machines and run the stuff in there. So, okay. Yeah, this is one of the parts that I think is a pleasure to talk very about how it works for us. I didn't catch all that, could you please speak up? I heard something about Koji and it's possible to run the tests because Koji's Koji are wearable or already testing Koji builds. So, that's already being done. There's a build that we're missing, but you would be worth listening to Koji. Any more questions? I'd like to build upstream different instance so we can see the results. That's a very interesting question that I think haven't been asked for that, so thank you. I'm going to try to repeat that, I'm not sure if the microphone caught it. So, the question was where are we going to build a bigger instance upstream? Well, I can't really speak to that because no promises about what we do, but it's a unique question and currently I don't think it was considered. There are... It would create quite a lot of problems, I think. Bicker has a big downside that setting up a bigger instance and a bigger pool, it requires quite a work on the setup and also overhead of a machine that needs to be dedicated to the bigger orchestration. But we are trying to work on moving some of the builds on the tests to Amazon or Cloud provider so we could make it public. The main thing is that those machines actually be somehow accessible to upstream community that would be, I think, problematic, especially with some very precious hardware where we have one piece or something. Building a beaker. Oh, why would you go? Any more questions? Thank you.