 Hello, good morning or good evening wherever you are. My name is Shua Khan We're going to talk about case self-test Running in Testings and where the status where we are and so on have given this talk At LPC as well. We had a good discussion there about that And then I'll also go over What we have done the recent release that came up and what feature that went into that feature So quickly, what are we going to talk about today? We're going to talk about general overview of case self-test and then KTL case of test goals and some of the challenges that we encounter in supporting case of test across all of current subsystems and use cases and then we'll also talk about running case of testing test strings and We'll talk about How what is involved in supporting Cardinal CI in As well, and then some of the next steps will identify So what is case of test the case of test Cardinal self-test is a subsist cardinal subsystem that Contains developer test suites for various cardinal subsystems spanning from memory management architecture Tests and so on it is a Cardinal developer test suite and it is a and then Users it's a developer test suite for developers as well as users cardinal users. So What is it it is a collection of Collection of Tests open and closed box tests. So when I say open it is the Since the developers write their own test, they know the code and they end up writing tests To test their own code in that respect. It is an open test suite And a closed box test suite meaning we do have a collection of user tests that run So they become you test written by People that don't know the code as much So that that in itself it's a closed boxes We have both of those in there and then we have some functional tests that test the functionality of System calls and so on or driver functionality in some cases some drivers and then it's also feature specific tests And then we have a mix of hardware and drivers test and we also have a few stress and performance in there Where you can find all of these tests is maintained in a Linux case health test git on the cardinal.org and There is a patchwork part project as well. If you look on the patchwork, you will see Cardinal self-test there and You can also reach me and you can reach the Developers that and that and users that use self-test on Testings on a three node the next case self-test channel So what is the focus of this testing? We the primary focus is testing features functionality and Regression tests and also subsystem focused. So when a new API gets added to say memory management subsystem or or Seccom for example Anything any new system calls get added the tests get added to specifically to to regression tests in the first place and then also Verify the new functionality So what is it not for let's talk about we talked I So far talked about what is it what it is for and now we'll talk about what it's not for it is not for testing a specific workload or specific application so who are the authors of these Who writes these tests Cardinal developers and Cardinal users write these tests and Who are the users Cardinal developers as well as Cardinal Cardinal users that is Testings for example So there is a overlap between Developers and users right because it spans multiple subsystems for example Somebody that is not actively working on the networking subsystem for example. They might use it to test Routinely if they are testing if they are running some tests They want to make sure networking is okay. They might use the networking tests as users So there there is a overlap between developers and users. So let's talk a little bit about patch flow If you are contributing to Cardinal self-test or you are just using them It would be a good to know how the patches flow. It is a little bit different from rest of the subsystems It is in some ways it is similar to documentation How documentation patches are handled? Cardinal because of because of the dependency on tests that have on the Cardinal API it The patch flow is Tune to that specific situation. So if for example a new API gets added or new Change gets made in a particular subsystem MM or networking or x86 or any any subsystem And then the developers Send a patch series. They will send a patch series Tests with the changes that are being made to the subsystem as a result They are going to be dependencies between the new code that's getting added to that Particular subsystem get and a self-test the self-test new tests or changes to the test in that area So as a result, they are all will be combined and will be will go through that subsystem Cree they might they will they not all tests go through the next case of district So you will see that happening during the merge window You will see that a lot of tests go through different subsystems They get added different subsystems and the way we manage it is we run We integrate them in Linux next We catch any problems for the most part to try to catch the problems in test breakages During that time or we'll catch it in after Rc1 comes up. So just just know that that's how the patch flow works So let's take a look at the individual What does a test look like I mean, what what do we mean by a test? So individual each The test directory is if you go into tools In into the kernel get and you'll go to tools testing self-tests directly you under that directory You'll see multiple sub-directories each sub-directory covers a specific subsystem. You will see In that you will see timers for example all of the timers Tests will be under that and then you will see m vm and then under that you'll see vm test And there will be each specific Area subsystem you will see test underneath and then each each directory Under each of those subsystems you will have you will find multiple tests for the most part if it's a substantial the subsystem has Large number of tests you will see several tests and each test will have Individual test cases sometimes and then it is a sub test and an individual test. So when you run a Test in For a subsystem or a specific test you might be exercising several test cases So we talked now that's kind of covers the overview and how patches flow and what is the test Looks like what what what what do we mean by a test and then test individual test cases now? Let's talk about the challenges and goals and Goals and challenges in that order The goal has always been evolving a common Framework flexible to customizing tests because we want to be we want to the goal is to Spend more time writing tests and not worry about common framework We do spend we do end up spending time in fixing framework problems, but the ratio of new tests versus The effort spent in framework is In that order we want to spend more time writing tests The idea is How increasing coverage for drivers and the various configurations in the kernel and also features we Have regression tests for fixed bugs as we find Bugs and we fix them We end up adding tests to Regression tests add regression test for those specifically so that we don't miss that next time around and The what does Some of these are all the challenges and then the goals at the same time We we have common interfaces for pass fail Skip reporting so when we new tests we write new tests We want to make sure those new tests fit into that framework and We are also support we also support being able to run Individual tests meaning not the entire test suite, but being able to select and run a particular subsystem or a particular test So we'll get into that a little more later on We and then also reporting reporting is a common framework thing We support reporting Results in using tap test anything protocol. We're currently I think 13 and we are We keep updating it to be compliant as the protocol moves and another goal we have is Balancing kernel stuff just runtime and coverage Right now we when we started we originally we were thinking will hold the time to a specific You know 20 minutes or whatever we had some goals, but right now we have lots of tests So right now the goal has changed a little bit that we do we We are tuned to more towards coverage than runtime Back when we started for five years ago This is a I Quickly talked about a little bit of a challenge that we have is because of the nature of kernel self-test itself It is primarily for kernel developers to make sure that they are subsystems or They are not Introducing regressions when they develop new features and they continue to look for the health of their subsystem it is tilted towards kernel developer use cases We do keep adding user K use cases, but it is I have to say that it is more tilted. So we kind of try to balance both But that is in itself is it's a little bit of a challenge Because of that the framework tends to There is a push and pull between the common framework where we are trying to address the needs of kernel developers as well as Users so for example Users would like to just run the entire test suite whereas cardinal developers Don't really want to They want to focus on their own subsystem at times and they want to quickly run and make sure they haven't They haven't regressed anything so as a result We are continuously balancing those two use cases where We are trying to try to support both communities So framework is So we are evolving the common addressing user cases use user use cases We're evolving the common framework to support testing this because that is beneficial to all of us because if we Can run self-tests in test rings then we catch catch problems early In fact We'll get into the test rings a little bit later, but Linux next Leonardo testing LKFT they run Linux next on Case of this on they run stay on the Linux next branch and we do find a Problems desperate is breakages early on because of that so the framework to touch on the Framework is like I said framework is well suited for manual testing manual testing essentially is a cardinal developer testing and then It is it does need it. We are continuously working to make it better To support auto Test environments like test rings at the same time not sacrificing the manual ease of manual testing So let's see case of test use cases So some of the use cases native and cross build use cases And then you So if you look at the car kinds of things it can support Somebody should be able to just say for example a cardinal developer Built a new cardinal. They have they booted a new cardinal. They just want to test a Particular individual test. They might have touched that area They want to go make sure that that didn't break So they should be able to run that individual test and they might be able to they in some cases They might have spanned multiple Maybe they made changes to an area of the cardinal that could affect couple of subsystems they want to be able to isolate those subsystems and say let's run tests only those tests and They should also be able to just run all of them All of the tests that exist under the self test directory Relocating the other use cases being able to relocate Native and cross build objects So one use case is your development and test system is the same you are you Are running your test cardinal on on your development system? There is no relocation involved You're just rebooted a your system with the new cardinal and you are running tests The other case is you actually build tests and you want to you might build tests for You are natively working on a x86 64 for example and you are running Building tests for another access x86 system that is kind of native build But you are relocating the build to be able to copy the objects to a your target system and in another another case is You are working on a x86 system Development system, but you're cross building your tests and cardinal and so on for a on 64 for example in that case you're cross building and you're You in that case you want to relocate the objects and then copy them over to your targets so generating So there are two use cases runtime use cases where you are Generating Generating so what what case office does is when you are Running it in install mode when you're running case of test and using make case of test It'll just run it on your system, but when you are Installing it will go run it will go generate a run script because In some of the tests need to to have modules loaded certain cardinal modules loaded to be able to actually run the test So to give you an example Cardinals self-test they are executables that just run in some cases. They are shell scripts that call other executables and They in in in some cases. They also load cardinal modules and unload them afterwards So let's see Let me find myself here. I think I am a bit lost Okay, okay, so this is we covered this part And then a couple The case of test and testings so I quickly talked about Nilek's next Being run on how does it run Linux cardinal functional testing LKFT? ring it runs various Releases like for example stable release and Then then main line Linux main line and then active cardinal all of the active cardinal is releases essentially so Where there is Linux next in more all of the cases other than one exception case of test is rev matched With you take the stable releases Say in general kernels So there is a tie between cardinal self-test and then the cardinal release itself because of the nature of the testing It's done case of test from the same rapport used to rev match the cardinal One exception is Latest table is run on all the stable release for this particular test and then in general you want to rev match the Cardinal because You might have herald changes in the cardinal that the test might depend on so zero day Runs cardinal self-test from main line on several trees and cardinal configurations Cardinals see I Use cases so we have been working on this is one thing that we have been working on I think we are at the moment We are able to build them And then We're able to run some of the some tasks in Cardinal CI, but this has been a working process and now as of the fight 5.6 Cardinal which is a few releases ago We're able to build support Native and cross builds All of the tests except BPF BPF has special requirements as a result it is difficult for it to Build in a It is rematched with the cardinal So it is difficult for it to to build so we kind of not worry about that at the moment We are working on that but we We treat BPF as a special case because of the dependencies it has on the Runtime environment Let's see Cardinal we added a new Cardinal dependencies Shelf script which will Evaluate your development system to see if you have all of the packages installed for it to for it to be able to do Cross builds that's and Because when you are cross building Cardinal self-test will depend on certain libraries in some case in some cases you have a mount live mount you some Some tasks depend on a special library, so you want to make sure that all of your live your Development System is set up correctly to be able to do cross builds So that's the this particular shot this shelf script will go through and evaluate your Development system and make sure you have all of the dependencies Met and it'll also tell you which test can be built with the current Packages you have installed on there and what you need to install to build more So use cases are similar In the sense that you want to be able to build and cross build tests for specific configurations, so we do have a way to To pull in would specify dependencies for tests for example, some tests will depend on a certain Cardinal configuration to be enabled in in the Cardinal itself and then it will do if You do not have those Cardinal case of a make case of this much generates a kernel config to include individual test Configuration files, so you can run that and then it will make sure that your cardinal that you're building will have The right configurations enabled for you to be able to test Run the test that you want to run on that cardinal and then I'll quickly go over the case of this commands and you can On on say you are You are planning to in to build and The commands are tuned for three different cases Whereas you can just build all of them or you can build and run the second Make out if you were to run make case I'll test in on the cardinal directory main root directory What it will do is it'll build all the tests and then and Build a test that it can build you might see some failures build failures, but it'll go on It won't stop. It'll just build and then run them run all the tests it built on the same system Um miss make case I'll test all will just build them. It will not run it May case I'll just install will go will actually build and run as well as it will generate a run script that You can use to which is what is run when you run when you run that script you will run all of the tests So if you were to do you can isolate targets Targets is a variable make a variable that you can pass it to Case I'll test in all of them case I'll test install all of the above commands the three commands And then you can isolate a specific test that way you can say I only want to run breakpoints or timers or any other you can give a Multiple tests that you want to isolate and run So what it'll do is it'll generate run script for just the tests either the test if you isolated test It'll regenerate it for those targets Otherwise, if you don't have targets, then it'll do it'll generate It'll build and generate for all of the tests under the self test directly that are enables specified in the make file Self-test make file is the one that controls the targets variable So when in whenever a new test get added that new directory usually or it gets added It will be added to the case case helps us case of this make file as well. So Cardinal CI workflow because of the relocatable for example, you would have to build the Cardinal first and then I'm showing a def config, but you know, it depends on what if you have a specific configuration you can use that and then you will build the Cardinal and then you will then build the Build case of this. So if you were to do a case of this all you can relocate relocate your objects and then into a Build a directory and Then from there, you can take that and then you can install that on to a target machine And then you you'll be able to run you'll have a run case of test run underscore case of that SH generated in the directory for you So this is you this is another way for you to do Cross compile compile and relocate you can or equal to the build directory You can specify that and then you are you keep you are specifying the Cross compile Cross compiler here and then you're doing the install you're actually installing Install really means what it does is it'll create that where you under your build directory it will Generate all the objects you need Including the run script that you can copy over to the target system to run your tests. So, okay, so before 5.10 Run case of shit the case of this script You you pretty much have to run everything you built if you didn't Isolate a test at the time targets, then you ended up with all the tests. So There's a new Ability that we added featured we added that we can now go Isolate the test after building run you will have all the executables all of the install is done You can go run Specific tests you want from the run case of this Script the script that runs the whole thing you can say I only want Certain tests I want to run and in addition to that it can even isolate individual tests like for example, you can see here That it can isolate From the timers Test suite it can isolate one single test nano sleep for example So this is this is a new one new feature that went into The next 5.10 and it's an RC one now want to leave time for questions as well. So So next steps We are continuously working on these next steps next steps. These are next steps is being able to add support for specific configs and then Support for specific features, which is a bit tricky because of the single test could cover multiple Multiple features. So that's a bit tricky, but we're continuously working on that. We're trying to Get to a point where we can run It is a very active subsystem because So originally when we started out we started adding tests for The features that existed in the kernel and then we now we are in the mode where we are adding tests With the as we add kernel features new features and we we also go make changes to these tests. We're continuously We are continuously adding tests and changing existing tests as a result This is a moving target in some ways because at times we find a new test might require Might require a change to the framework sometimes or we find the framework It particular test doesn't build because of the number of use cases we support in some cases A test comes in and then that test might not might break a particular use case or it doesn't work with the particular use case So we end up fixing problems. So it's kind of a moving target in some ways we especially being able to support in the test strings So because of this nature of the flux and change the test Framework if a new change to a test to breaks the use case then At times it becomes tricky to tricky to until that gets fixed. Obviously test drinks We will have to so we are fine. We but on the On the good side we find all of these problem Early on every usually we try to find them in the Linux next and we are focusing on finding them and fixing them before See one comes out and if it if some breakages happen, we are trying to catch them Early on during the release cycle. So that is it for my presentation. I Can I'll open this up for questions you can post questions in the chat Channel, okay, I have a question from Okay So does that answer a question Tim? I guess so this there is a question Dash L flag emits a list of tests, which should be possible for a framework to loop over or filter I'll have to look into that. Um, I How to play with that test and that feature a little bit more before I can answer that question I Race my volume guy a little bit. So I hope this is working better. I didn't sorry. I didn't realize That volume is low. So, okay, let me see. I'm looking at the Questions here I have a question. I don't are there any other questions Can you hear me better now? So there is a question here. Are there any famous bugs during case of this packaging for the next table and beyond to run in other platforms and Selfless in general, is it a place where I can find this kind of information? Yes, and no we don't it's the bugs are reported on mailing lists like any other any other subsystem bugs for case of this as well and then I We don't have a It is reported on the mailing list, that's all I can say. Yeah, we don't have a place the bugzilla type place that we have We track these bugs. All right, let me see if I have any more questions Any other questions? Thanks everybody. There are no no questions. I'm not seeing any questions in the chat