 Hello everyone. Thanks for waking up early and attending this talk on Saturday morning. Can you hear me? Can you hear me at the back? Yes? Okay cool So there's no sound system. This is only for recording in case you're wondering So my name is Guillaume Taca. I work at Collabra and this talk is about kernel CI So I had to talk last year already about kernel CI for them. This is kind of an update on what things have happened in the past year, but also about the future really So what is the purpose of kernel CI in case you haven't in case you're not familiar with the project and also What it wants to achieve in the longer term. So it's kind of climbing a big tall mountain really without trying to reach the point where kernels are released without bugs and You know have at least quality control. So we know which bugs are in the kernel and extending test coverage This and also having reusable tools. So the idea is to test the upstream kernel But anyone using the kernel in their own downstream environment should be able to reuse these tools We now have an official mission statement As written here. So that's basically what I'm trying to explain here But this is like the official word we have now. So people spend some time making it making this stuff So they deserve this at least a slide here So we have Last year I said maybe it will become a Linux foundation project and this has happened. So in October kernel CI has become a New project of the Linux foundation with all the members listed here So Colabra, Lady Brie Redhead, Google, Microsoft, Fender, Result, IO and Civil Infrastructure platform, which is another Linux foundation project Now we're still in the early days of we know what does that actually mean for the project So, you know, we have a mission statement We have a budget and some stickers, but there's a lot of other things coming soon, hopefully There's a website also a part of the website explaining what that means for the project So although it has become a Linux foundation project the the aim is still to be really community oriented So of course there are things that are done by the members for the members But it's really to facilitate Things to happen for the kernel community at large So we're still focused on upstream Completely, I mean the tool again can be reused for other things So if Redhead wants to test a federal kernel, they can do that Well, maybe the project will help make it happen as an example, but it's still about testing the main one kernel upstream kernel and it's still about sending email reports to mailing lists and Engaging engaging more with subsystem maintainers and developers in general To basically help people Add their own tests to kernel CI a lot of well every subsystem have different workflows and some Maintainers have created their own small CI, which works perfectly well for them probably But then they might be reinventing some wheel and also not have access to all the hardware that kernel CI provides So anyway, that's the kind of consolidation this That that is the main objective we're trying to achieve so one Kind of philosophy behind kernel CI Is what we've called the open testing philosophy because you know, there's only one main on kernels So in the same way that people from many different origins for many different reasons contribute to the same code based on Linux kernel You know when kernel 5.4 is released That's for 5.4 is going to run on a on a super computer on an Android phone or on anything It's the same kernel the same code base, although it does a lot of different things So the reason why there are so many different test systems is because of these different use cases But we could also have a Common way of at least sharing the results because some tests are relevant to all of these cases If we want to like test for memory leaks in the kernel You can do that in it, you know wherever the kernel is running So everybody is doing their own product tests But this this is the idea is to have Have a test system that allows people to contribute their tests and contribute their hardware in the same way that they contribute the code So yeah, like I said, so this large number of contributors So yeah, basically that's that One of explained here The image is trying to say that we should try to like have the same surface of between tests and the code So it's a bit like test coverage if you want to like align the two things like the whole Idea of having open source development and having open source open source testing open testing on top of it basically Now how do you achieve that so this is like an abstract Pipeline explaining how kernel CI works for all of these blocks. There is At least one implementation, but you could have different things for each of them different instances So let's start from the beginning. So you have a developer in the top left corner. So someone writing some code So the code ends up as patches kernel CI doesn't test patches directly at the moment But it can be done other systems do it Or get branches then of course it gets built The purple arrows are like when files and results are pushed to then you have build artifacts and after that When you have a build you can run tests and then process the results to detect when there was a regression or basically See what needs to be reported from the results And then your results are stored and analyzed to be able to then provide a report to the people and The developers and this is when the loop is closed. It's like a It's a bit like any in automation when you the objective is to have a working kernel, you know, like if you try to automate An arm for example Like a mechanical army want to know where in which position is going to be if it's completely open And you just move something and you don't really look at it You can guess where it's going But if you have some sensors that will actually know where it is and you know the angle or everything then you can have You know closed loop or to make a system and this is kind of what we're doing here Except that it's doing this around developers and not the round machines But it this is it's a similar kind of thing Yeah, sorry I'll speak louder. Thanks. There's there's a link here you can now so these lights are variable as PDF there's a link on the On the page for this talk and on this slide seven there's also a link to a shared document that has the more detailed description of the Pipeline design modular pipeline So this is like the theory now in practice. What is that? How does that work? So right now we have the currency idol dog There is one database that's being used that's being used for a long time But we have a new one using Google BigQuery and the idea is to have to use that as a prototyping database But with access to more more testing systems, so All the tests currently the test results currently available on currency idol dog also sent to this Database and now we have also red hat submitting test results to it And we would like to see maybe other people so typically typically member companies would be the first one So maybe to contribute to that So we're kind of refining the database schema for like storing bill results and test results And the the first outcome we would want to have is have a unified test email report to based on all that collective data We could then send an email say for you know a stable branch release or something It's that I don't if you've noticed on stable branch mailing lists Or stable mailing lists you have a lot of test results from coming from different test systems So that's one of them The targets for having a common upstream kernel testing test system is to have to have only one report that's gathering all the data So having this database is the first step. So if you look on the previous slide, that's basically the The The store bit so yes toys artifacts and database, but this is at least the database part Yeah, and then we can build kernels in different places So that's you know the block block number to bill how do you build way if we if you have different people providing different like cloud compute And different ways of doing it if you're modular enough, then you can include them. So That makes the project easier to scale and right now we're using Jenkins, but We've made it and we've changed it in a way so that it can run anywhere So you can run all the kernel CI steps in the terminal and some people have tried also using it in GitLab already So Kevin is not here Kevin Hillman has done this in a as a GitLab CI way So he's using the kernel CI command line tools in his in a branch in a kernel branch So automatically when there's one new changes are being pushed to the branch It uses GitLab CI to build and test things like in other GitLab CI projects But this could be used in other test systems as well And also obstructing the lab API so you know test labs are where hardware is available to test kernels So by default we use lava, which is you know the linear project for automating tests But it's not the I mean there are plenty of other things So we've made an abstraction in the kernel CI core tools so that it can submit things to any lab So right now it's only sending things to lava, but we can send it to any any other lab that has a web API And can accept a test definition So these are things we were working on So yeah, the development goals a bit more specific So the command line tools are like KCI be able to do To reproduce a kernel CI build from scratch KCI tests are to generate test definitions So these are the things we're trying to improve so people so they become more portable Also we want to make it easier like I said a bit before to Make it easier for people to contribute tests So if you're a maintainer or developer and you have a test or maybe you work on the Part of the kernel and you make some tests for it. It should be easy for you to enable this in kernel CI and right now this kind of A steep learning curve if you want to do this at the moment You need to be you need to know too much about the internals of kernel CI to be able to add a Add a test to the list, but that's one of the things we try to like lower the power basically and We need some good interaction with that so we need some collaboration from people who make the test as well so We started with like a key people to try to try to do that So maybe next year I'll come back and see if you follow the project you'll see maybe things won't become easier But right now it is not very easy It's it's still a good thing, you know if you have a test you really want it to be in kernel CI You can send it to the mailing list or an IRC and kernel CI on pre-nodes or kernel CI groups.io mailing list Or just on the on the subsystem mailing list people know about kernel CI now Also, yeah, we're trying to improve the web dashboard. So I don't know if you've seen the current kernel CI dashboard But here we're showing mostly Bills and boot results. It's not really there is a test tab, but it's not really good at showing test results Because kernel CI was originally about billing kernels like a variety of different arm-depth configs and then about Boot testing and it's still kind of stuck in that phase So we've we're starting running tests, but the results are not shown properly on the web front end on the web dashboard So that's that's one thing we're fixing actually right now to unlock a lot of things there's many tests that we kind of held back because maybe we could run them and then Email reports have some limitations and if the front if the web dashboard doesn't show all the results then the test Kind of get hidden and that's that's a waste of results But we still also need to improve the Email reports. That's also a two-way process because it's hard to design something From like from the beginning it's more like you know sending email reports and seeing how people react to it and whether they You know what is useful what is not useful and different subsystems or different mailing lists need different types of reports actually So the key message I guess here is we need feedback and If the email you get is not perfect Often people say if you send another one like that, I'll just you know put your put a filter to put all you cannot see I emails in Spam or in junk Well, that's maybe a if it keeps getting bad all the time, but normally we try to like improve things as we go. So That's the key messages, you know, we can't get to perfect thing from from day one basically And also a slightly more longer term projects goal is to improve test bisections So we have bisections for boot tests for boot, you know Just plain boots at the moment. So like booting to a login if you can log in there to boot And that's very easy to bisect because it's only one pass fail result per test per run But as we're running more test suites that have a lot of tests Then it becomes harder to bisect things when you have different test cases passing or failing at different rates So that's we have I'll come back to this in a in another slide, but we have some some ideas on how to do this as well So reusable tools. Yes. So we have KCI build KCI test KCI root FS and there's two to create root file systems. There's a Wiki page that might That might be written in a not too distant future But right now there's a wiki page kind of work in progress that gives you at least a starting point if you want to start if you Want to see what these tools do We also have the interface to the BigQuery Database so there's a common line tool for submitting kernel CI results to the common kernel CI So if you have a test system and you want any testing the upstream kernel you want to submit results You can get in touch and use that tool and you know, we can give you a token and then you submit your results Then also Docker containers. I haven't worked on it myself Recently so I'm not an expert in the state of all of these but I know that there's some good progress that has been done to have containers with a lava instance with a you know Jenkins configured for kernel CI and the back end that's the back end currently used on kernel CI.org with MongoDB and front-end which is here on kernel CI.org So there are containers to make it easy for someone to recreate the whole system in their own environment And again, if you try to find some issues, it's all on GitHub on the kernel CI project on GitHub But we need we need more people to use it basically to get some get some feedback on it So yeah about advanced by sections So there's this there's this tool I worked on called scalpel which is inspired by easy bench Which was written by Martin Perez at Intel. So we've been talking because he made this tool for for graphics testing It's actually not really used in production easy bench at the moment But there it had a much more advanced by section feature than the normal git Bicycle one you have because this can deal with different were several test cases at the same time So when you run one test you can provide the results for many test cases It can also request test to be run several times if they are not stable So the scalpel project is basically extracting this and boiling it down to a more portable thing that can be used And there's a if you look at the project on on on this git lab Project you can find as a demo that will run like create a git history with like dummy pass fail things and it will find a problem in it And it's kind of ready to be used in kernel CI But it will come a bit later I think but that I hope that will really help For some things so people say maybe we should be testing patches as people send them because when you have one patch on the mailing list You can test it on its own and you know if it's good or bad before it gets merged and yes That's ideal, but it means there's a lot of patches So adding you know supporting all of it and you know building if you want to build all the kernel Config's we build or even a subset of that safe You want to be a 10 kernels for each patch that is sent and tested with 200 different? Platforms and you know that doesn't scale very well I think so maybe in some cases if we do it wisely We can only test a subset based on which subsystem the patch was sent to but I can take a lot of tweaking So the idea of the whole Advantage of having bisection is when you test index next for example where all the new patches that have been applied are merged together And if something break So if something breaks at this point Then you only need maybe 10 iterations to to bisect from yesterday's next to today's next So you'll be able 10 kernels basically to find when when you know that there is actually a problem So you can be very efficient So that's why bisection is such an important thing for currency I Okay, so tests wish list so these are the things that have been identified so far But again, maybe some people would think that Network keeps failing all the time all the things maybe are also important to test But these are kind of obvious things that are directly So the other the ones at the top are the ones that kind of come with the kernel And we're not really doing them too well right now, but that's I Think that's like an obvious list of things we should be doing and the next test project I know that LTP that was mentioned a bit earlier. That's At least some parts of it because some parts may not be completely relevant to the kernel Maybe more like use is space oriented, but there are definitely some things in there that we That you would expect to be run on on the kernel test system and then some some are more Subsystem centric like you know for media subsystem you have visual to compliance and other things IGT and XFS tests These there's a lot of them basically And this is where more integration with well more communication with the subsystem Maintainers and developers needs to happen We're trying to be we're trying to go There's a lot more that can be said it's a very big project It's kind of because it has been around for five years But it's new at the same time because now we have a budget and we have some ideas and we have a lot of limitations Have been removed Since we've joined the Linux Foundation So I'm hoping it's like a rebirth for the project We need also some engagement from the people when people to realize that now, you know This has been selected by by the Linux kernel community basically as the main project for testing testing upstream And you can reuse it for you and your own products as well So does anybody have any questions? Yeah Okay, so the question is about the technologies used for testing in general so There are many different things so the tests that come with the kernel are normally written in C like a self-test Or you have some shell scripts as well and Well, the static analysis coxinell is a separate language And K-unit is also written in I mean Cassan and these are like built-in things so the kernel configs you enable basically and K-unit is a bit the same It's part of the kernel so it's all written in C. Now you can have There are also some test rates a lot of basically the quick answer is a lot of it is written in C because it talks to the kernel So it does a lot of system calls and it makes sense to have a low level Sorry to so I can So the question is whether there's a high level language to define the tests and then run them So yeah for for example the the tests run in lava There's a YAML definition that explains what needs to be run and then you will have you'll have some commands there That might be binaries or maybe shell scripts or maybe Python so it depends each test suite has a different way of being run Yeah, I think you were second Okay, so the question is for one run for one test run what is the usual run time and what's the expected time for that Would be reasonable Okay, so Of course each so the question can be answered in different ways So if you want to test everything then you can add you know Time how long it takes of course some things run in parallel So it's going to be like until you get the whole result is going to be what all takes the longest that can also be done by Splitting test suite into smaller things to run more in parallel. So there isn't a simple answer to that but what is what is What is desirable basically from a time duration, I think it depends on where in the process you are where in the Can all workflow you are what you know, so if you're saying if you're sending a patch for example You want the result to be quick enough? If it's if it's Linux next because there's a version every day You expect to have the results maybe within like six hours that wouldn't be to that or maybe 12 hours Even if it's running a lot of things If it takes more than that then you have more kernel versions to test then at times it take to actually run the test if it's for Then a stable release maybe maybe you can run more tests that will take two days and it will be valuable so it depends on which Version of the kernel at which stage you are in the development Okay, so the question was about integrating it in Travis CI to Yeah Okay, so it was a comment about if you wanted to integrate it in Travis CI with a limit to 50 minutes Basically, you need to accommodate things around you can also like build on the one kernel version if you know the DevConfig the architecture you care about I guess so the question is about is it makes sense to run Docker You could use Docker to build a kernel and run things that will We can use Docker to build a kernel at least yes to have an environment to build it Anyway, yes Yes So right now there's a list of good branches that are monitored every hour So the question is how I cannot see I test triggered and the answer is right now There's a list of good branches that are being monitored you can see them on you know on the Website here you have trees and branches. So all these are monitored every hour And the system knows the last version that was tested And if there's a new version then it triggers the whole thing and it builds a bunch of tests Public best master It's even if one more even if one more commit was pushed to the branch That's a different revision at the head of the branch and that's enough now like I said also Which we've proved that it can work in a GitLab CI environment where it's when you know It's typical GitLab CI workflow and that can trigger things as well So you can reuse the same tools in a different setting if you want to I Think maybe you were next yeah, and then okay Okay, so two more questions So the question is Yeah, so the question is if we start testing patches, how do we determine which tests to run basically? So first, you know which mailing list it was sent to that that could be a clue And then of course you can do some stats on the the files that it's touching and You can also what right now we have some But so we have a YAML based configuration at the moment to For each git branch with tests to run we could have something like that for patches So do you know depending on which mailing list it was sent and maybe some people will care more about some things like you know Some maintainers will say every time I push something or well, maybe my tennis don't send that many patches But we could have any kind of arbitrary rules to I think there was a question here If we're monitoring the cross So we are yeah, we're monitoring a lot of different branches So the question is can we are we monitoring any random branches? So we're monitoring the main ones of course like main line and next and stable And we're doing as much testing as we can on them Then we're monitoring subsystem ones and we do a lot of testing But maybe a bit less and then the further away you are from the master main line master branch the less testing you get Basically, so it scales in that way so you could have an individual Branch, you know from like maintain our individual developer Maybe if you have like only one or two kernel builds and you don't you do only a few tests on that then it will you know For one Linux next if you do 200 bills, that's like 200 people doing one bill basically So that's the way kind of scales If the community think it's useful to have a branch like when someone wants to have a branch edit this and Request basically if people think it's a good idea and if we have in a bill and test resources then it can be added As long well the main rule is I guess that it needs to be upstream oriented So if it's a downstream branch for your own product, that's not something that would be unconvinced at all normally Okay, so the question was about Generic Way to talk to labs So right now so there's the there's a command line tool called KCI test Before I show you so this is basically it's about the The bit when when we have a bill then you go to the run box basically that's when you start to start to test so Right now we're using so this command line tool KCI test is in Python And there's like an abstract class with some methods like there's one like to generate test definition another one to submit a test definition and we could add other things like receive the results right now the Results are sent directly to the back end in some cases. We could like I have the The test labs and the result back to a place where KCI test would be used as well to receive the results and Process it and store it in a database so these are kind of the main primitive functions and To really have it work in practice. We need to test it with more labs right now It's kind of only working with lava so it's still not very mature That's basically where it's going So I think we've run out of time. Yep. Thank you very much