 Okay, thank you everyone for coming over again for these lightning talks We are next having here Linux test project introduction by Cyril Herbis. Sorry for pronunciation of last time It was good enough Okay, so hello. I'll welcome to my introduction. So let's start There is some information about myself and actually it made me feel a bit old when I realized That I've been using Linux for 18 years and also working for SUSE for quite a lot of time And I'm mostly working on kernel automation and stuff and I've been LTP upstream developers for 2009 actually So let's start with where it begin the Linux test project was was founded by SGI IBM and these big companies in year 2000 and the picture there is a screenshot from web archive because I wasn't around obviously because I was trying to install my first Linux at that time and In 2001 apparently it contained 100 simple tests Cisco tests and few tests it's collected for from other sources Well, historically LTP had quite a big Problems and that have had but reputation until recently it started to turn much better Last few years, but basically there was no real manpower behind it There was no code review basically everything that compiled got in and Yeah, the build system was a mess and it was quite the problematic to even compile the stuff And even when you compiled it fair amount of the test cases was failing randomly But so it wasn't really useful and in SUSE in the early times we had to run it few times and then filter out all the random failures and Yeah, it was kind of put together for from different sources sometimes crazy ones because of the Well, what I found out from the comments in the source code that when IBM acquired some of the old proprietary unique sees they just took that their test cases and somehow ported it into the LTP it was quite messy code and They actually put inside everything they found out on the internet So it was kind of Frankenstein and sometimes even today the stitches are visible, but it's much better these days And also one of the problems was that IBM tend to hire junior developers for the task And it turns out if you want to test the kernel you really need guys with some background in the computer science and Operating systems and well students wouldn't do well good job So what we have done so far we spent last few years Working on on the stuff. So now it's much more boring. We have out of coding style of git repository We have code review and mailing list. We also well work with the github So we accept the pull request there as well and we do quarterly release is with these numbers of patches and developers the Chopper but release and we also use Travis for compile testing the stuff Even things like cross compilation So it it's quite nice because now if you take the latest LTP had it will just compile is fine and Also, we make sure that the latest LTP is working on maintain distros We basically before we make a release We try to compile and run it up on everything we care for is our redhead sue said a buy and stuff like that and We also have quite nice test library. So we do not duplicate code in the test cases so if you are writing test that needs something like block device or file system mounted at some point you just flip a flag in the Structure that is describing the test and it's done and we also have quite a lot of API documentation and tutorials These are targeted mostly for people who wants to write test cases though and Here is the goal. It has been the same from the beginning. We are trying to Kind of make sure that the Linux kernel and Lipsion related stuff works fine We do not Do any benchmarking if you're looking into that there is a extant test to it formal from male Gorman And there is also test to it. That's general test to it for file system Even if it's called XFS test that's kind of historical name and nobody bothers to rename it at this point So if you are looking for testing file system, you should combine the LTP with that one as well So what are the challenges? Well, the the biggest challenge is that the project goal is quite broad and there is obvious disparity between the number of developers that develop Linux kernel Lipsi and LTP with kind of problematic because They keep to add Cs calls and stuff into kernel and we are still trying to catch up on that Well, and it's even difficult to estimate how much User space API is there if you ever seen the extant boot book from Michael correct the linux programming Handbook or something. It's not I would say that it's not not even half of it. Nobody really knows what's there LTP is well kind of large and it's mostly complicated low level code So it's challenge as well and sometimes there is not even documentation for the kernel Interfaces and when it is it it may be wrong So I ended up with quite a few patches in manual pages as well when I was fixing the test cases Because sometimes I just found out that the main page is saying something different than the code in the kernel is doing and also Sometimes the kernel API really changes because if you ever seen linux linux shouting We do not break the user space on mailing list what it really means that they would not change any code that is used By user space programs, but it turns out sometimes There's some interface that is really used by just the test cases in such cases It does not make much sense to keep it if it's broken So sometimes they break out test and we have to deal with it Okay, what's really inside we do have quite a lot of Cs calls test cases These are mostly unit test for the Cisco So they call the Cisco and check that is doing something reasonable We also feed it invalid inputs and check that it returns some reasonable errors And then there are some more complicated stuff But these are mostly unit test then we have fork of the open POSIX conformance test suite It was developed separately, but the upstream is dead. So we carried on and we have Started to add various regression tests for linux CVs This is quite red sand recent and we have well not that much but quite a number of these now Even the most important ones and then we have IOS stress test network related test cases that needs two machines and Real-time test suites and various containers and names paints and stuff Okay, what what are the design goals? We are really trying to keep it simple so the languages of choice are C and Each test is an executable itself contained and runs automatically and the status is past an exit value It's we are just trying to make stuff sane and easy to use Well, and this is the question I'm getting quite a lot and people are asking Well, do we really have to test our kernels with when the upstream is tested quite a lot And it turns out well when you take the upstream kernel and you start applying patches You will break it similar or later. So we really should test that well, and there is Short how to I'm just trying to tell you that is really easy to use LTP these days So if you want to run an LTP test case, you just have to download the code and then just compile it Then you can run most of the test cases directly from the source core directory Some of them needs path to the current Directory in the path so that can execute sub executables and the shell test cases need Paths to the shell library and path obviously, but that should be that should be it for most of the tests or you can really install it and use the Script that sets up the variables and stuff itself well the Actual the solution that we have in the LTP is quite dated it works, but I want to work in Work on the replacement. So it's much easier to integrate it into the modern Frameworks because this one just it's a shell script that runs kind of old Binary and it produces well text files that are Parcable, but I want to have something that outputs JSON and XML or stuff. So you just plug in That thing into modern CI. It just works well, we also have scripts that run the network test cases and These are some of them are basic some of the error are quite advanced. You can either set up two machines with The same LTP installation and one of them will SSH to the second one and run the network stuff between them Or it can also fall back to namespaces. So you can run the network stuff on single machine as well Well, and that's about it. There are links to the repository mailing list wiki and I see so Let's go to the question if anyone has some Basically today for run it Okay, they are asking about the stability of the test cases these days when when you execute it sometimes one of the Test may fail and it may be false positive, but we have thousands of test cases. So I would say we are pretty good We spend something like last six years working on that well You can write to the mailing list people do that from time to time asking about hey This fails and what I'm supposed to do and somebody will mostly me will look into that Actually, it's these days it's used on the zero-day thingy in Intel. So you will probably not catch that much of the bugs because The automatic framework usually catches them before it gets even to the main line But yes, you can always talk to us. So on the mailing list on IRC I'm on the IRC at least during my work time So well the the question is if we will be flooded with messages and the answer is We do not test much of the driver specific stuff yet And I'm not sure we will have manpower for that. So I'm pretty sure that we will not be flooded that much because most of the generic problems will show up everywhere and We are running it also into automatic frameworks in Seusset We are using open create to run the stuff and it's executed well daily