 Okay, it's about time and since we're quite tight in schedule, so today we're going to see some less basic examples. We're going to see how we can further customize the test. I'm going to still showcase everything live here. And yeah, that's the plan of the day, there are several examples along the way. You can also try them and follow me. Let me just check if, okay, this is still defined. Okay, so we've seen in the previous tutorial the test parameterization. In this one, I'm just showing it in another different way, also another way that you can write and generate long sequences of parametrized tests. So again, I'm getting the stream benchmark and now it's parametrized in this way. So essentially here we want to parametrize on the array size and then we have a bigger range. It shows us that we hard code everything, which we take different powers and we powers of two and then we have the array size, we do some math here, what's the number of times and then we set it up here when we compile. So again, but again, you see it's just a single test and we're going to see that we're going to generate several. I think no, we don't need anything else here is we want just the GNU, yeah, I'm gonna be thread here as far as I remember, nothing new here, I'm gonna also add those, I think it's 16. And another thing is here is another little trick when you don't want to have your reference values but you want to have units because you just when this is reported or when this is logged, it's better if you have units rather than not. All you have to do is put here a star which means that reference is valid for any system and it's like a follow back and then list your variables and you put the reference value of zero and no bound, no bounds and just put the unit, yeah, we do that also as a follow back in some cases or when you don't want to publish your numbers but you still want to log them with units. So that's it, let's try this thing and let's see how it, what it gives us, I'm not gonna run it first, I'm just gonna list it just to show you. So you see, so we had like 11 versions and if I want, because this is also with the syntax here, it might be a bit strange but essentially what we want to produce is such a list because every argument of the parameterized test is a list of arguments to be passed to the constructor here. So here we have a generator that generates those lists and then we simply unpack that generator. So in case you wonder what is strange syntax and sometimes can be a bit of a problem to get it right. But that's one example, I'm not gonna run those but you can try them. Now there is a nice thing which, these I don't have it in the tutorial but I can do it right now on the fly. So there is a very nice function that generates, that finds for example all the different modules and generates tuples of valid systems, valid programming environments and module combinations which really allows you to test. So here I think we have products. So what am I going to do? I'm just gonna copy this into, okay so all, I'm not gonna create a full test but I'm just gonna show you this in order to see how it looks like. I forgot to import something. So to see how this gets parametrized, like tutorials, advanced, let's see, I hope I haven't messed. It loaded the test. It takes some time because behind the scenes it uses module of L, we could have a version that uses spider for L mod documents. Okay the output is a bit crumbled here but you can see for example the different combination. It's gonna load Gromax 2004 version. I think the module is named Gromax 2004, FOS 28, Python 382, that's the module name and then we converted the non-alpha numeric characters to underscores and you see it generated basically for every system, for every partition, a test and there is also one thing here that we can pick a test and paste it in more detail so that we, so here are the, I just picked one. So here is the Gromax test using that module and yeah on the built-in and then it can do it also in FOS, but in that particular case you can also, let me, I'm gonna go here because there is another parameter that's quite interesting. So here I think Victor you have flat module name scheme, right? Yes. And that's why we get all these unnecessary combinations because normally if you have a hierarchical name scheme what I need would just give you the right combinations. Now if you have a flat name scheme you can pass a mapping of between the tool chain that is inside the name and the tool chain or the programming environment of Reframe because for example here we essentially, we don't need all those combinations so, and this is gonna happen like in this example for example I'm matching whenever I see CrayGnue I'm matching it to the PRTNGnue and I'm gonna do exactly this and I'm gonna have some here so that's useful when you don't have, when you have flat schemes. And then I think okay let's call it FOS, all the versions of FOS and that we just map it to FOS, to the FOS internal environment I think we're done, let's see now what we are gonna get. Oh, I think I forgot something, oh yeah, just a second, two sparse, oh yeah, yeah let's do what I have here, do this, yeah of course, of course, of course, where am I doing that? Yes, of course, okay, yes, sorry for that, why doesn't like it, must be dict of strict and string and string, environment mapping, ah, okay so that means this example is wrong, okay we found that bug in the documentation and I was wondering why I had the, okay so now you see, I mean we didn't get all the unmeaningful combinations so we get, you see the three versions of Gromax that they are installed, 20.2.2.3, so practically this we can, with just a simple file we can parameterize it and test all the modules in the system, which is, yeah quite interesting, actually we don't have production tests with that, I don't know if you Victor has written anything that uses it yet, yes I have one test case, yes, okay so moving on, here we're going to show for example how the frame deals with make files and different build systems, so here we have a dot product example that resides here and again this small bug here, so you can see again as I said before now the SRC now in the directory there is a make file and then the code, so let's see the test, let's go to the first one, so here the change is, there are two changes, so you first you tell the frame that it's going to use make to build it, actually we wouldn't even need to do that, this thing, if we didn't have to pass any flags to make or any options, but now since we'd need to pass the cpp flags in order to select the element type, we just also define it here, now since we're whenever you use make, cmake or auto tools, the frame doesn't know the executable, so here you have to specify also the executable and other than that this test is not introduced anything new, but let's see, let's run it, just select the make file test and actually here if you, it's not an exact matching this one, so this is a regular expression, so it's going to select both float and double to run, I think to be in mind, okay we just had GNU but never mind, so now let's see the output, so the output is the frame AUM we run on let's get the CN, GNU double and the bit, so here is what it generates, so essentially as we said the loads, etc by default it does make minus j1 and then it passes all these variables based on the environment, so if there are cases that some make files or even if you're using cmake, in the cmake it passes those in the configure line, not in the make line, but there are same cases that some codes don't like that, so in that case you can tell the frame not to pass those flags and we can even try it right now and it will just generate only what he passed, nothing from the environment and of course then you can even set the concurrency, the amount, how many jobs you want and you can also have custom make files, okay so you see that it didn't put any of the variables, so the same you can do with cmake or auto tools, so you can there are other build systems that you can hear for example say cmake and then there's also specific options, again those tools are really not so sophisticated as easy build or because the purpose is completely different, it's just to be able to compile a test code and then there is another interesting thing, we haven't seen that yet, so of course you might need to just clone something, you've seen before that we in the stream test that we, so just a question I'm seeing is lack, could you make that HMS aware which means sorry I can understand that, so here we you see yeah please the HMNS is hierarchical module naming scheme, I need to check Victor, I'm not so much into the modules, sorry I didn't understand the question, are the module namings, ah what was the question, I think okay you can, we're gonna mute, ah okay yeah that's certainly, I mean that's something that can be done on the back end, so transparently to this, yes because now it's it's it's done outside any, it's not module specific, okay so here we've seen also the pre-built CMD, so essentially one you could do is say that the sources there is none, that means that don't try to copy any resources and just do the git clone here, but for convenience there is also you can put the the repo here in the sources there, so and here we accept url so it's not going to accept the SSH syntax, the scp style syntax, but you should use the SSH protocol, you can see the git documentation in case you want it. Now another thing is sometimes you might have a code that uses its own configure script, not CMake, not autotools, so in that case again you can do, you might use a build system make, not CMake or anything else and then just pre-built commands, your custom configure and so on and you can see that it generated. Another another category of tests is what we call the run only regression test, I don't think we've seen one so far. It's basically the test that you don't have to compile anything, so we just have to run something, it could be that you load the module and you just run executable or you just run a utility, so they're exactly the same as the standard ones, except that there's no build phase and the resources of the tests, again tests may have resources which again reside, let's see the test instead of just talking, so it doesn't have resources, so again the resources would be under an SRC directory, it's exactly the same as the other tests and whatever is there it's going to be copied and yeah and the copy is going to happen just before the run phase I think, no just after the setup. So this one is a yeah, so in all the run only tests you have to set the executable apparently, nothing special here, okay let's run it. Just to be complete and then we have the compile only test, those are the tests that they actually don't, they skip the run phase, they don't have any run phase but they do have sanity and performance phase and in this case the only, you can use basically the same idea with a test but and then you can use the sanity patterns to look into the output of the compilation or the standard error, so yeah then we have this make only test for example that essentially it's just a compile only and the sanity patterns they're only following the, yeah we're just saying checking there are no warnings. One thing is that I haven't mentioned is that the frame doesn't check the exit code so it will not mark a test as a failure if it exits one and that's in purpose, it doesn't happen this for the compilation phase in the compilation if anything in the build script fails it will mark the phase as a failure but for when you are running then exit codes are not taking into account because there might be several you might yeah just want to check an error condition or you might yeah there are several reasons we want the user to specify what is an error there is a there is a configuration parameter that you can set and make the frame explicitly checking the exit codes of the job scripts which is all not always so reliable it really depends on the back end that you are using for apparently if you're just running locally of course it is or if you're running slurring with accounting again it is reliable but not in all the cases and then also the script that gets generated you must be sure that the last command is the command that you want the command of which you want the exit code so for these reasons it's not the default. Now I'm going to see the comments later on so one more offline question okay I will take the questions later on so apply sign to function iteratively we've seen already a bit more on sign to functions now here we have a silicon test that it generates a set of random numbers and then in that test we make we want to make sure that all the numbers they are in they are in the set they are bounded so here what we do is the following so here we use this extract all thing we've seen that before as in its form extract single which we're just extracting a single element of the match but this one is going to extract all the matches so here we have a pattern the file to look at and here is the capturing group here we have named it and how to convert it so as soon as this returns here we have a list of loads so then it becomes much easier to do our math so first of all here we wanted to check several things first of all we want to check the number of the count of the number of numbers we got is actually 100 so then we use this function the count numbers this is a bit more generic because it also accepts generators and then we just make sure that this equals 100 now here is the tricky part so we want to check that each of the numbers we got has the its value is bounded between 90 and 100 now there is a trick here so we can iterate with a for loop and the reason for that that's simply a python restriction so no it's not a python restriction it's actually a decision we made so actually when you have a deferred expression if you try like those produced by those scientific functions if you try to iterate over the elements of those lists it will trigger its evaluation so you have to do that in a indirect way you know let's say in more functional programming style way so so here we say that we use the the map so we use the map and we check that all the elements are bounded are within those values and then all of this is deferred and that's that's a very nice thing because here it's a deep expression that is going to be lazily executed at the later stage and yeah that's that's how you can you can write you can iterate over elements recursively inside the patterns you could write your own function in which case you could yeah iterate in a for loop but if you want to do it in a single line or that's the way to do it you can't have like here we we can't have like something like that this this is gonna fail this is gonna fail so in case you want to do it another thing here is yeah let's let's run and then I just forgot something ah yeah the thing I forgot to tell is the following so there are three when you're working with sanity expressions there are three things that trigger the three evaluation rule everything is going to be deferred so you can assign you can add you can subtract you can do whatever you like you can combine any expression you want everything is going to be deferred for for evaluation in the future there are three conditions that they cannot be deferred when you try to iterate over the elements of signed expression it will trigger its evaluation when you try to get its truth value so either explicitly or implicitly call bool on it and the third is when you try to print it not to print it when you try to get the str function on it either explicitly or implicitly so whenever you you write your sanity patterns these are the three things you need to look at and yeah I'm not gonna look into the output but you can check that yeah all the values are apparently between nine and one hundred now another thing that keeps coming up usually as question is how I kind of customize the job script that gets generated and here we are going to see how several ways so we're let's start for the from the oh from the simplest one keep track of time oh we still have five minutes sorry we can continue with this tutorial tomorrow so here you can use the pre-runs in this and the post-run res in this these are going before the run and after the run command where the run command is actually the command that that gets s run for example or npi run so I'm gonna just quickly run it and then you will see uh this is what it generates uh some things before and he's here more or less the pattern that gets generated so first is the job scheduler pre preamble then commands about the test environment so you the things you see module loads etc or exports the pre-runs seem these and this parallel launcher the executable and executable options and then the post-runs seem these now how about if I want what additional options I'm just showing this example and then we're gonna finish it so sometimes you want what additional options to the to the to the job script and actually this is pretty easy so essentially you have to there is this job descriptor that is generated after the setup phase so anytime after that you can with a pipeline hook here just as I say run before run and set those options and this is going to get into the job preamble noting the ns run command with the job preamble and yeah if we run it yes of course because the current system is not in the supported ones so all the previous tests were pretty generic so so then you will see that you will see that we get this thing and those they go always at the end and there is also another trick that you can use temporarily so you can pass options from the command line yourself so you can say it's a minus map not minus minus j mem equals uh let's put it 500 and you can pass from the command line directly to the back end and in this case you're gonna see that's an additional thing going here at the end and this is going to override yeah you rely on the scheduler's logic here it's going to over it what's in the test so i'm finishing here and we're going to continue tomorrow i'm going to take any questions offline and in the chat yeah i think we best end the stream and the recording here there's time for questions in the zoom session as well but we need to avoid overlapping this stream so let me end the stream here okay yeah so i can i can stay here for yes any questions