 Hello, thanks for joining for this case of test running in test rings talk today. My name is Shua Khan. I'm a Linux Cardinal Fellow at the Linux Foundation. I want to talk through a quick overview on what we're going to cover today. I'm going to give you a quick overview of the case of test and goals and challenges we have with this case of test. I'll walk you through some use cases. I'm hoping to do some demos as well. I'll show you a few use cases to use cases and how individual tests can be run, how you can run all of the tests and so on. I'll hope to do that. And then we will also talk about what does it take to supporting fully case of test running in test rings. And then I will also talk about briefly Cardinal CI use cases, related use cases, which are applicable to generally all the test rings. But you're kind of focusing on Cardinal CI getting case of test running in Cardinal CI well so that we can continue to test, run case of test to do regression tests on all of our trees as we are doing integration and our development. And then that's kind of the agenda for today. Let me jump right into it. Case of test is a developer test rate. It is written by Cardinal developers. Cardinal developers and Cardinal users can both use this test suite. So what does it contain? It is a collection of black box, test white and black box test, user space with some Cardinal assist in terms of modules to test the Cardinal side of the code. From the user space. And then it's a functional and a feature test as well. You will see, for example, huge pages test related to test, to test huge page features, and then some of the virtual memory features that go in and then timers, different various kinds of timer tests and so on. So we also have a mix of hardware and driver tests. We have a few GPU tests and then drivers, some driver tests. This is where we can, there are some challenges to adding tests for hardware. So it's a work in progress. We keep adding tests, but drivers is continuing to grow. We also have a small percentage of our stress and performance tests. Examples are CPU hot plug, for example, and memory and CPU hot plug. And what happens when we were to add and remove a large number of CPUs and remove memory modules, memory. So we have a few stress-related tests, but it is by no means it doesn't have too many performance and stress tests. So where does the Cardinal self-test get? I gave a link here. Let me quickly show you, share a screen to show you what it looks like. Let me, sorry, I'm playing with sharing. Okay. So this is what it looks like. And you have K-Unit fixed also funnels through this repo. So you will see K-Unit fixes, for example, K-Unit next and next and fixes. This is what the self-test looks like. And then there's a patch or project project as well. So patches are maintained there. I'll quickly show you what that looks like. So you'll see patchwork. It's all, you can go look at the patches that come in through the patchwork. And let's see next. And so testing focus is, what is the focus for this testing? Features, functionality, regressions, and then we want to cover subsystems. So we do have a large number of subsystems. Every new release, we have several new tests and test cases getting added. Every release, we see the number of subsystem coverage going up continuously. Not just the subsystem coverage, it's also individual test cases for existing subsystems. So you see constant growth in this area. And let's talk a little bit about what it's not for. It's not for testing workloads or an application focus testing. When I say that, that's just not the focus. Like, for example, you were wanting to test a specific workload, then this case of test is not suitable for that. And who are the authors? Who writes these tests? Cardinal developers and cardinal users as well. We have a large number of test writers as they use the test. They send in improvements and then they send fixes when the tests don't work very well. So who are the users? Again, we, our cardinal developers use cardinal cell tests heavily during their workflow. And cardinal users, of course, use them to test the cardinal and test their specific subsystems, specific systems that they are interested in. Patch flow. So this is a cardinal cell test patch flow is a mix. We have the reason the patches funnel go through the cardinal cell test kit, which I maintain. And then patches also go through MMGit and then x86Git. I'm not listing all of them. Various freeze maintainers send tests up to Linux's mainline. Networking also another. The reason for that this patch flow model is because we want to make it easy to write tests and make those tests available for users and developers to run. If a feature, networking feature, a new test depends on a networking feature that is in networking Git. It doesn't make any sense to hold the test stop to send it through the next cardinal cell test kit. So same, same goes for MM, same goes for x86. So our, the idea is make it easy for and tests and features go together, make it easy for, for enable, make it easy to test, find a test. And that's, that's the goal really to make it easy to test. It doesn't matter which tree the test actually goes through. So I'll give you a quick overview on what tests look like. At this point, I'll take you to sharing my terminal and I'll show you what it looks like. Let's take a look. All right. So here you will see I'm, I have a, this is a 5.3 latest, 5.8 latest. I'm at 5.8 RC3. So let's take a look and see. I wanted to show you, this is the main self-test directory that we are in, tools testing self-test. You will see a lots of directories with the tests. You'll see all each of the, like for example, you have KCMP, for example. That is a one test that tests a certain KCMP feature. And then you have a test for VM test right here. This VM test has multiple tests underneath. So I'll quickly show you, go into an example of breakpoints test, for example, when I talk about this is, this is a test, high level test that gets run. And when you run this test, you will see multiple test cases being run. Let me quickly run this test for you. So let me run it from the main. I want to show you the use case of being able to run this from the main directory. How can you, case of test can be run as a individual tests or all tests together. Right now we are just going to run one single test. So I will run targets variable. Let's you select which test you want to run. Like for example, I will quickly run site test. So it will also install the required necessary headers because the idea is case of tests are designed to test the headers and the kernel together. So it will build all the tests. First install the headers. Go ahead and install the headers and then run the test. And you will see it built this size test and it ran the test. And then it will, that's the output of the test. And you will see this tap version 13. That is a test anything protocol reporting output. Results are reported in this format. So you will see when, okay, one indicates here that this test is successful. You really run and then you'll have a test is skipped. You will see test skipped. So that's the idea of then this output can be used by parsers that are tests that can parse the tap version 13 compliant output. And then parsing becomes easier to keep checking your results are the same. You want to make sure that each time you run the test, you're seeing consistent results, right? So it becomes easier to test consistency of those results. And then I'm going to run one more here to show you the test cases. And I'm talking about test cases. You will see I'm running a breakpoint steps here. It will build it and run a lot different tests. So this particular test, when I talk about test cases, this is what I'm talking about this test breakpoint tests consist of two tests. One is step after suspend test, which I don't run because it'll I don't. This is a more of an invasive test. I'm not running that. So that's why it needs to be run as root. It tells you that it cannot run the test. And then it says I am skipping this test and because you need to be root. And then you have about 110 test cases in this individual test cases in this particular test. So you will see that all of the tests are run. So it'll print output saying you have so many errors and failures. There is two failures in this particular case. And you can keep checking this test, running the test and figure out which ones fail and which ones pass. Let's go back to the presentation. So now you can see how the break if you map the breakpoint test here. You will see that the main test breakpoints here. And then there is a two top sub test that we had. We looked at suspend test and the regular breakpoint test. And then breakpoint test has about 110 individual test cases. So we are constantly compare adding to the individual test cases here. And then as well as adding to the, we are increasing targets. So we'll keep increasing targets and we keep adding sub test and then individual test cases. So we are working on all of these four different areas. Let's go to the next one. What are the goals and challenges for this as the Cardinal Self-Test keeps growing? What are the challenges? We have reasonably good framework right now to be able to test and customize building, running and so on. That's the focus. We keep evolving the common framework. The common framework includes reporting, building, running, install, and all of those fall into the framework. And then we keep increasing coverage in terms of new configurations, new features, and the drivers fitting into the targets model for each subsystem. We keep adding those. If you go back to looking at the previous slide I showed you. So we are constantly adding, working on adding these. So it's a continuous process. We do add regression tests for fixed bugs. When some of the drivers I maintain, if I find problems, I go add tests. When I am fixing bugs, I want to make sure they don't resurface. And they also testing for various conditions. So a lot of the Cardinal Developers keep doing that. So we have also, you have seen quickly that it shows pass, fail, and skip. This reporting is part of the common framework. We keep looking for areas where tests report. If they cannot run, they report to skip that they are skipping the test instead of failing because it will result in even skipped tests are reported as fail. That would be false reporting failures. So we continuously monitor looking for any bugs in test areas where they are not correctly reporting pass, fail, and skip reports. And reporting also, I simply thought, I showed you what test anything protocol output looks like that we are compliant. We keep making changes as the test anything protocol makes, keeps advancing. And recently we are actually working on a RFC patch set right now that we found it's not totally compliant. We are fixing those. So we keep making changes there as to keep up with the test anything protocol. And also balancing case of test runtime and coverage. Originally, I hoped that we could run all these tests in a certain amount of time. However, that is not the goal at the moment because we want to be able to run. The subsystem has grown quite a bit. We have a large number of tests and holding it to a certain time is no longer advantageous. So I am looking at you can still select if you want to time it. You want to keep the time. You can pick and choose the number of tests using the targets variable. You can run only the test you want to run. So that way time, total time, balancing total time and coverage. I am leaning towards coverage now as opposed to saying, hey, we want to hold to a certain time. So what are the other challenges? It is heavily used by kernel developers. The primary goal of this test suite is it is a developer test suite and we continue to maintain that spirit. Because we want to be able to use it during our kernel workflow to be able to use it when we add new features. We want to make sure that our other features don't regress previous behavior. No regressions, no new bugs introduced. So we want the goal has always been developer focused and it will continue to be developer focused. And additional goal is to being able to support in test strings. Because it is important to do so. The challenge here is a framework that has evolved and well suited for manual testing. Meaning a developer going and being able to test their subsystem, specifically just run their subsystem specific tests to support extending it to support auto test environments. That has been a continuous challenge since its beginning. So what are, it supports native and cross builds because if you want to build a kernel self-test to run on a ARM64 system, for example, you can do that on an exciting system and then take those tests. And then other use cases include individual tests. You can run, I showed you being able to run individual tests just now. Running size and breakpoints test, you can pick and choose the tests you want to run. And you can also run a subset of tests as well. And then you can do relocation. That feature, that support is evolving in some sense. Currently it does not support relative paths, but it's a work in progress. So let's see, I do want to show you a few more runs of this, but I'm going to go through a few more slides before I switch to a demo mode. So case self-test use cases. Running tests, you have, so the previous slide, this is like building use cases. You can just build them and then run them later, right? So and then you have the way I showed you the case self-test command from the top level, make file by running make case self-test, it'll build, run all the tests. That's what it does. I have shown you examples of running individual tests. And then also installing. If you are building a kernel self-test to run on a different target, you have to do install. The reason for that is all of these tests, they have shell scripts attached to them in some of them. Some tests are driven by the executables get called from the shell scripts. And then also a lot of the running part of it, like for example, if you have a dependence on a module and you want to load the module before you run it, there is all of that, it generates a run script at the install time. Using make, it generates the run script and then you will need the run script to be able to run it on a target. So I'll show you an example of run script in just a bit. So this is a good time for me to show a bit of a demo on what I am talking about in terms of install. Let me go share again. So let's quickly do. I'm going to do a case of test install of one test and show you what I mean by run script. Oh, okay. If I spelled it right, it would work. So what you're seeing is it is going to install the headers in the target. So that's kind of what you're seeing. So once that is done, we can go take a look what it installed. So it generates a case of test directory so you can easily R sync it. And you will see this size directory is where it puts all the objects it builds executable. So you'll see get size over there. And then install is the one that you can just take and copy. And you will see the run script that I was talking about. And this run script will run it and then echo when it provides all of the environment you need to run this. Like for example, if you were to run this run script. So this output should be familiar to you. We ran it from my Git repo a little while ago. And then the same test, you are running it as an installed test. So let me stop sharing here. So, okay. So that talks about a little bit of how to run these tests. And then what is the current status of case of test and test range? That's really the focus, some of the focus of this conversation today. Kernel self-test is run in Linux Kernel LK, LKFT testing. So let's go look at that. I want to show you, okay, I'm sorry. I'm going to share that screen Chrome tab. Let's go looking at LKFT. So this is the Linux kernel functional testing ring. Linaro runs a lot of tests, kernel self-test on various rings. They test not just kernel self-test, they run other tests as well. But this is what the dashboard looks like in terms of currently they are testing. They show the stable RSI 5.7 being run. And then they also have mainline tests. They also tested the next mainline right here. You will see the Linux mainline test right there. So let's go back to my presentation here. And then they run tests on Linux Next, various trees on Linux Next. And then also Linux mainline, stable, and they will cover all of these active kernel releases. So let me share that again with you quickly to get an idea on what are the active kernel releases. Currently the kernel active releases are here. 5.7 needs to be added to this as well. I don't think it's supported quite, but you can see all of the releases that get supported. 5.4, 4.19, 4.14, 4.9, 4.4. So this LKFT ring runs kernel self-test in all of those. So let's go take a look at another one. So a case of test also gets run in zero-day service. So the way they run is they run mainline on several trees in kernel countries for their coverage. Kernel CI test rings, this is where we're working on. We are working on adding more and more tests to be able to run them. The focus has been that, okay, it's running on LKFT. We are coding there, but we also want to run it in kernel CI. So that's kind of what we are working on at the moment. And I have been working on fixing some of the framework to be able to have all of the kernel self-test built in a cross-build environment and then any bugs we may have. Most of the things that sometimes it show up is because the framework is flexible. It allows tests to be able to build and run. They can customize. For example, BPF requires Clang to be built. So customization is possible. Because of that, once in a while, we find that when we make new changes to the test, it sometimes will have to look at the framework. Does it the use cases? There are two separate use cases. Use cases for developers use cases that get test more often than ring use cases. So we have to make sure that all of these use cases don't break. So that has been somewhat a challenge and it's a work in progress. We are continuously making sure all of the use cases are work and they continue to work. And so dependency checks. There is another requirement that in kernel CI, you might have a system that doesn't support all of the configurations. You might have to tune the configurations the way you need. And then when you're building it on a development system, you want to make sure a kernel self-test can build what tests don't build. A couple of things. You need some tests require dependencies packages installed. So when a test depends on external packages, meaning it uses the kernel heros and everything, but you might need a few libraries loaded. So kernel dependent case of test dependency.sh, you can run that to figure out if your development system can build all the kernel self-tests or not. So you might have missing packages. You can do that. You can run that and figure it out. And you also print targets that can be built. It will give you a list of targets. So you can pass it to the targets variable I just talked about. You can write a wrapper script using this kernel test dependency. You can do two things. One is figure out what you want. You need to install on your development system to be able to build, cross build, and then also get a list of tests that can be built. Let me go to the next one. So the more use cases being able to build, cross build as tests. And then one of the other things that we are working on is to be able to, if a particular hardware on kernel self-test kernel CI ring wants to be able to certain configurations enabled, then we should be able to pick and choose. So automating, it can be done manually right now, but automating that is what needs to happen for kernel case self-test to run seamlessly on kernel CI, without fully automating as much automation as needed without manual intervention. Make case self-test merge. I'll quickly show you this particular part a little bit, what the configuration files look like, and then what can you do with case self-test merge. When you run case self-test merge, take all the individual test configuration dependencies and it'll generate a kernel configuration for you that you can build the kernel with, so to get the coverage for the tests. Let's take a quick look at what... Okay, I think I lost my window. Sorry about that. I lost my window that I was demoing things for you. Okay, I'm back there. So let me share that with you. And then you will see config files. So you can see several config files under each of the tests. So let's take example of maybe... Well, let's go ahead and try what's underneath. So it'll show you that this particular test requires all of these kernel configuration options enabled for it to run. It has its dependencies. So if you were to do kernel case self-test merge, what it does is it takes all of these config files here. It'll generate a kernel configuration file for you to build the kernel with. Let's look at what ZRAM. I mean, it's easy to guess. I guess it needs ZRAM and ZS malloc. It'll enable that in the configuration file. So if you can just do case self-test merge, I won't run this, but yeah, it'll take a bit of time to run. If you were to run this, what it does is it takes all of these config options and enables them in your .config. So it'll generate a config for you that has a kernel, and then you can build the kernel with all of those options enabled. Let's get back to the stock sharing here. I'm getting back to the presentation. And let's look at, so what is tricky? This configuration is work in progress. It can be done. The tricky part is being able to enable specific features. For example, you have a VM. Just say that just a multiple options to a system call. That would be something being able to select that those specific test cases can be difficult. It can be done, but that's a level of complexity. These are the use cases that we are focusing on to begin with being able to build and install and being able to select the specific configurations that we want to test, tailor the configuration, test configuration to the system that is being tested. So I talked through these a little bit. K self-test all will build all the tests for you. Make K self-test will run them as well. If you are natively building it, make K self-test install will install the targets. I showed you what that install looks like with the size test as an example. And then you can, for kernel self-test, kernel CI workflow, you have to first build, you're doing it, you look at build because you're building in a different directory. You want to be able to, maybe you are using a development system to build ARM 64 tests for the K self-test for a ARM system or maybe you are building it for a PowerPC or you could be building it for Spark, for example, you could be building it for any system. So cross build, you cross build that, you build a kernel first and make sure you, then after that you go ahead and build your tests because we have seen the link between kernel configuration and kernel self-test with the configuration files we have available. You need to tailor that you have to match those. So a little bit about rev matching. I didn't talk about that earlier when I was talking about rev matching kernel and kernel self-test. In general, you want to, in some cases, in a stable, older stable releases, you want to rev match the kernel. That's desired outcome. And in some cases for coverage, if you want to be to run 5.8 mainline kernel self-test on 5.7, which is okay. They all run, you have to choose between rev matching versus coverage, right? In some cases, it's necessary to rev match because especially BPF tests, you want to rev match that. You don't want to take a kernel self-test from a newer release into an older kernel. So we talked about this is how you can install the test. And going forward, next steps, like next steps, I kind of, I talked about next steps already to some extent, talking about how we can evolving this manual framework, a framework that's well suited for manual testing, running tests into a automated environment where you are able to tailor your configurations to the system, target system you are testing, and then also being able to select the specific features. So that's work in progress. And I'm working with kernel CI admins. And Kevin and I have been collaborating trying to get this all working so we can run it in the kernel CI on a consistent basis. All of the, when they pick up a new release to test, they will be able to run kernel self-test as well. So with that, I want to spend some time answering questions. If you have any. And let's see, I'm seeing some questions here. How does K self-test compare with p-test? I have to, I do not. So okay, so I will answer from a different perspective. Kernel self-test is part of the kernel repository. So this falls into a different category of tests. We have LTP tests, we have several other tests outside the kernel repository. They're all very valuable tests. So the difference between kernel self-test and other tests is we are able to evolve kernel self-test as we are writing features. So we are able to develop tests as we write kernel code at the same time. Very often kernel self-test and kernel features go together. So that is the advantage we find as kernel developers that we are able to test them. Without these two efforts being separate, they become a joint effort that they go together. So I do not make that answer to your question. That's that one difference that I see from outside tests. I'm not, I don't specifically know about p-test, but so that's the answer to that question. And let me see a second question. Do these kernel self-test systems use Sysbot reproducers of kernel crushes? No. So the reason for that is Dimitri and I went back and forth on how we can, if we want to bring Sysbot reproducers into the kernel self-test. General recommendation is reproducers aren't, they're auto-generated very often. They don't have the same level of code quality. I mean, it's auto-generated code. It might not meet the coding guidelines of kernel also. And then it takes a lot of work to turn them into, to add them to the existing kernel self-test suite. Because the kernel self-test suite has to worry about testing protocol, for example, or being able to build cross-build and so on. So we have chosen to keep that separately. And that, I maintain a Linux art skit. Let me show you. So I talked about that in the previous one. That is a Linux art skit that I maintain on kernel.org. So I periodically pull from Dimitri's Sysbot reproducers. So they're all preserved. We don't want to lose them. So we, and it's being used. There is a run script that runs it. And then LTP is, started using this as well. Pull it from the Linux art skit. And then, by the way, Linux art stands for auto-generated test suite. So we are preserving it. It's there, but we are not planning to make it part of the kernel self-test. For reasons of, we evaluated it and we decided we want to keep it separate. Will kernel self-test work with any kernel version? Yes. The goal is you should be able to, for the most part, take the kernel self-test from the main line and be able to run on the stable kernels. So there is, I say that I ask people to use their judgment because on one hand, we are continuously adding new tests in some cases to cover existing features. Those existing features might be present in the stable kernels. And then you might get a better coverage. However, you will also have a lot of noise with that because stable kernels only take fixes and new features and new configurations don't go in there. So you will have to balance that coverage versus noise. So I leave it up to individual users as making a call on that because you will have a larger, depending on the release, you might have more new tests, newer tests that won't be able to test the, won't be effective on a stable release because you do not have those features present or configurations present on that, on those kernels. Let me see if I have one more question here. Is it worth slash recommend kernel self-test when writing custom device drivers? So driver coverage is, let me think about that a little bit. So if you are looking for, if you are looking for being able to test a custom device driver with the case self-test, obviously that's not possible. Unless you write a test specific to your device driver. We have a few device driver tests in the kernel. We are increasing them. We do not have as many. However, you could, if there are areas of the kernel, your device driver depends on a feature. Just one feature that comes to my mind right now is say Android Ion memory. So if you say your driver depends on that, you could run the Android, that particular test, that test is specific features, kernel features that your device driver might depend on. So DMA, you know, any of the DMA stuff, Sync is another one, DRM Sync protocol. So there are some sync tests in the kernel self-test. If your driver depends on that, so I would say, hey, include that in your testing. So you have to kind of take a look at what features make sense for your kernel features that you want to make sure they don't API and kernel features don't rigorous for that you are interested in for your specific driver. Other than that, if you are writing a custom device driver and you want to be able to test your driver, then you have to write a test for that. So it looks like I have a bit of time. Let me see. I can show you the Linux Arts get here. Okay. So I will show you where I'm collecting the Linux Arts, all of the Sysbot reproducers. You should be able to, I maintain this. I pull this, I just seven weeks ago, I think I did a recent pull. And then I pull them from Dimitri's SysColor repo and then bring those tests in and then commit them. I don't make any changes other than just no changes necessary. Usually I will add a commit log and then I'll pull, I actually pull all his commits and then just with the, and just apply them. So you can find all of the Sysbot reproducers here for you to run. So that is that, that you will, I keep periodically pulling them. And then let me see any other questions. So maybe P test is more appropriate to test a custom driver potentially. I do not know P test. I need to understand it before I can answer the question, but yes, there are, we do have various ways to test the drivers. Meaning if you have driver specific tests you want, you need to run tests on, then you P test probably is more appropriate for that. But we are adding more to driver tests. The problem with driver tests always is that you want to make sure, being able to run it in all environments, right? Like for example, you're doing auto testing, the kind of coverage you would get for your driver depends on whether that hardware is available on the kernel rings or not. So we have about three more minutes left. I also have an AMA session tomorrow in the afternoon. I am going, feel free to stop by and ask any questions you might have, more questions you may have, or feel free to send me email. You can reach me on kernel IRC channels as well as the newbies channel. So let's see what else is left. I guess there are no questions, I guess. Thanks for attending definitely and then look me up for the tomorrow session. And I did a talk on security minor development right before this. And I'm going to address both. If you have any questions about kernel self-test and kernel CI efforts, what we are doing, plan to attend the session. Okay, thank you for attending everybody. I hope to see you at the AMA session tomorrow. Thanks.