 So welcome everyone. My name is Victor and I work at CSCS, the Supercomputer Center on Switzerland and I'm part of the user engagement team and support scientific support team. So I'm a PhD in Chemistry, so I'm a guy that does the support for scientific applications and we there we care a lot about doing guaranteeing that scientific applications are running perfectly in our system. So for that CSCS it spends a lot of time and money on getting regression testing of our scientific applications, right? So before I actually talk about what's reframing and what we what we have done as a tool as a framework to guarantee this quality of service for the applications that we run in our systems, I want to give you a bit of the background what was before at CSCS before we started to reframe. So CSCS is head by the time very monolithic script that would do all the testing of applications through shell scripting. So we had a huge it was well written for sure but was like it's a bit difficult to maintain scientific software in shell and the testing. So we wanted one day they decided to change the the scheduler on Cray from API to API run to this learn native's learn and then we had to change a lot of lines inside the shell scripting. So it was a bit maintenance of how to change and if there's a bug it won't script because we saw oh we are checking the the the way that we're doing the testing it's a bit wrong we had to change it then we had to go to all of the tests and end again. So it was a bit of tedious task and difficult to handle. So then we came up with a new solution that's called reframe and reframe is basically based on Python so it's a framework that we that we want to leverage all the software or scientific software methodology so I like OOP right object oriented programming and also the fact that we had to change this scheduler lines from all our tests we start to say you know what I think we should be abstracting away the system abstracting away the module systems abstracting a lot of things so we can actually focus on only our tests right we should not be focused on on details of the system and we do support different systems that are conventional linux boxes or from to create systems so highly customizable customizable systems until basic ones or generic ones standard ones so it's a bit difficult to actually maintain and if you don't abstract away the system in your regression suite right so we want as reframe to the user to be focused on all the logic right if I want to write a test for gromax for instance I want to be focused on my gromax run I don't want to be caring about how the the job will be scheduled how it will run the system which partition it will run I really don't have to care much how to submit that partition I just want to say I have a gromax that's gpu I want to run a gpu partition of pitch time if I don't have a if I cpu only version I want to run on the cpu partition of pitch time that's it I don't want to say how does I can achieve that the that partitions right so we started reframe reframe it starts to get momentum and was starts to be developed on march 2016 now we are on release 2.16 and we have like 18 forks and 32 star gazers and still counting right it's a bit difficult I changed the numbers a bit before because it was 31 in my slide now I have 32 so it's all the time changing and the the reason why I like so much the reframe is actually design goals right that's what drives me crazy I'm amazing design goals we have first the first one is productivity as a chemist right I want to write my test as productive as possible I don't really want to care about all these details of computer science I want to be really really fast on writing my tests and I want to run them as fast as possible so and I also don't want to care much about the system right is am I running is learn am I running pbs why do I have to care why why do I have to care if I'm running lmod or running tmod or tmod 4 what are the details I don't really care much about these details too so for me I was hired css and for first week I got the task to write jobs to write tests on reframe and then I start to write them and was really really fast to start using it and I could like the first two weeks I think I wrote like 132 tests was really really fast so that's what I really like about it so really fast to use so really fast to integrate and from that time it's we have been running reframe daily we run the reframe daily no productions we have never seen any let's say stress the stack race of reframe reframe is very robust right so we have a very very written code the design is really really well done and it's that the team lead for for the designers in vasiles and he's really keen on on guaranteeing that we have a good a good code for reframe so what does reframe as a framework support right the first thing that also Kenneth was talking about easy builds that we do not have python dependencies right I mean we have no external dependencies on python just if you have a bare python you can run reframe of course it's python 3 based so you do have to have a python 3 installations so if you want to run a unit test of course you have to style something for the unit test as byte test but if you don't want to run a unit test you can have your python and start running so it's really it's really good to have that and for the way that we as a perspective of the user who is running right in the test we have complete separation between what is a system what's a program environment and your test logic so we're going to see the api going to show you later how it's going to the division and this this thing makes your life very very easy because then if the user doesn't have to care about that then the user just just writes the I want to run a program environment gnu which means I want to run gcc and I want to run with slurm and then my life is is done so I'm going to load the appropriate gcc compiler and then run with slurm of course that requires some logic inside the framework so we leverage we take all the pain for guaranteeing that inside the framework and then you have to tell us what does it mean to have the the program environment gnu right where you can find the module what's the module there for that so that is also a user has to configure it so it's not something that you download and you start using it as was but you have to get for your system how to set up some some let's say mapping between the system and the what we expect right so we do have some sort of cycling so I can say my test for gromax if I have gromax installed with gnu intel pgi kray I can go and say I have this test and I say I support this program environments gnu blah blah blah and then reframe automatically generate different tests for all of this so we're going to cycle through all the program environments and generate tests and I can test all these installations of independent installation of gromax for instance and then the the fact that we not having like some json file to define your test but we leverage python it means that you can be really clever or how you generate the the the the tests so you can leverage the entire python infrastructure to do that so these allows us to do some easy customization so you can have really easy factory designs or you can spawn a lot of tests and you can also do a lot of flexible flexibility in organizing them because you can have a hierarchy of tests and all this kind of stuff for us on the user support perspective of reframe vocabulary let's say we do have something that's called sanity checking and performance checking right so when we talk about in general to the operations people and they say are the system saying but then the system saying means the system producing correct results and it's performing correctly in reframe perspective if the test is saying it means that the test is producing correct results and if the test is performing it means that the test is getting the correct performance expected performance on that system right so we do have this separation and then there we allow the user to customize the way they extract what is saying for them and what is performing for them right so if i have my code like cp2k that gives you time as as your measure of how much it will run so i can get go there and look at for the time if i have grow max that gives you nanoseconds per day for instance i can go there and look at the nanoseconds per day so we have this flexibility of looking what does performance means for your test for instance like we have a hello application i can actually see the performance what does performance mean if you're compiling hello world right it's time taken to compile hello world for instance we can decide what your what is the you can decide yourself is as a the guy writing the test what's the logic like what's the the real thing that you mean by performance and we as a when you run reframe it you can easily see the progress what's actually happening so you see what is i will not show you later what is being running you're going to get the performance uh progress of of your test and also we're going to get summary what failed what we didn't fail what passed we didn't pass and by us failing means you either failed the sanity or you either failed the performance right so that's the that's the our concept so we're gonna we're gonna see at the end of the test of reframe this summary what has failed we also have an amazing logging support so it's it's is incredible you just like write i want you configure to say i i want to be a fire or i want to print it to the stood out or i want to do gray log and then it just works so you're going to send your files to gray log you're going to write fire you're going to print the screen it's going to be uh boom yeah so so it's just it's simply it's it's it's it's amazing though the way we have done that it's it's really nice and then we have a really that's one thing that i like a lot to work in the internals of of the of reframe you get to have a really clean api so and we we tend to make the api very stable so to guarantee that we don't do a lot of changes and but we keep adding functionalities another thing we do is basically currently we support slurman pbs but reframe is a open source project if you feel free to contribute to other schedules if you if you like i'm gonna we will love to increase this this list we also have support for different launchers so s run npi run npi asak we also support no launcher you can run your your code directly without any launcher we already support t-mode for l-mode and t-mode so any any environment tools available in the market we do support so we are ready uh we have like build systems backends cmake auto tools and make we also planning to do easy build n-spack build system but we didn't do that yet because we need to figure out better the use cases so we want to make guarantee that we are covering as much use cases as possible in a simpler way because on reframe we do care about the users story right we when we we're gonna see the test and for us it's all about writing the test if the test is becoming difficult to write uh that's not a good design so we go back and then we we draw again and we draw again we draw again you get a really easy to use for the user to write tests right uh because as we had heard before or yesterday someone said well i'm not really good on python that's that that's what we want to to to guarantee here that you don't need to know a lot of python just need to know the basics you can copy uh a basic template and changes slightly small things and you get your test because you want to guarantee that everyone from scissor the means who don't are not really comfortable with python to to computer scientists who are developing their code and doing want to do ci with them can can do with reframe so we have a signal execution of tests which means that we take the test we submit a lot of tests to the to the queue up to a certain amount if you want or you can say that the amount is limited infinite and you can submit everything to the the scheduler and when it finishes we're going to reprocess so it's a really really good way of taking making use of the scheduler to submit different jobs and being doing parallel executions of the jobs and we claim like the open hpc people that we have an extensive documentation so we I think we have a covered everything we in vazidis is really really how can i say uh tough on that so we never make a release if the feature that we want is not documented if the feature is not documented we don't have a release so and we want to have leaders because we have the bosses asking for releases so you have to write documentation together with the with the the feature so I would say that a majority of all the features actually are covered by the documentation in in reframe so we have many more things that we support please take a look at the github the project is an open source project anyone can contribute so how do we how do we envision the this architecture how can you abstract away all all all these so we have clear defined a regression api so what's the user who's writing the test sees right and we also have seen a front end api so let's say the people who actually running that right running the reframe because on on cscs we have this separation we have the professional people who are actually writing the tests of people from the support in general and people who are running the tests on maintenance for instance other people from operations these are the means so we have these these two classes of people right the dreaming people they basically just have to care about the front end flags and in general we we try to make a very simple simple interface for the common line just the miners are and it's running if you do minus value list the tests so it's really simple you can do minus h get the help so where we support a lot of a lot of common lines so we try to get the the dreaming people really really keep smiling them and then for the guys who actually develop in the test that means me right we want the that they write as me as less code as possible and be as expressive as possible so we take advantage of of many abstractions from from python and also in practice what I have to do to write my test is write a simple python class and inherit informed regression test and after that I am already have my test so I just identify I can customize it to get actually what I want the modules that I want the program that I want if I have to compile what is the source code if I want to get a make file what's the make file so I can customize it and then voila it just compiles run submits and runs your test you can analyze it in performance I'm going to show you later how it looks like so here the writer sees this thing right so we have an api that you allow it to have some system abstractions and some environment abstractions right and what you call system abstractions is basically the schedulers and the lounges right so this is related to the system itself and the environment abstractions is module system and the build system by the build system is like make see make environment modules by the team model so we don't you you you have access to these abstractions but you don't really have to to how can I say define if I'm going to be pbs or not or really go deep into the api so it's a bit packed in general doesn't look like so packed they have to cut the lines I mean to remove some lines you make it fitness line but this is as I told you is just like mark some you decorate the test if your class python class and you inherit from regression test it means that you have a simple test you can change this you actually going to show in the next slide to get a factory of tests so you can spawn several tests and then basically you get a description you say I want to run on the gpu partition of pitsteins and then I can say I want to run this test with you know kray and pgi so we're going to compile my kuda code because it's not q right here with these three compilers and we frame automatically detects that this is a cu right the kuda so we're going to call nvcc on this and then we're going to say the system the build system single source we're going to take this single source take the flag I'm going to execute the the code with this parameter so minus so sorry if there's a kutable with four thousand ninety six and a thousand I have to load kuda two kids or pitsteins to have access to kuda and then I can say the number of gpus per node I can get the part and so I'm looking at all of the output of the of the program on the screen to say time for single if I have this thing it means it meant that it has passed the performance is get you get the performance in gigaflops and then I can say I'm expecting 50 for on that case I'm expecting 50 gigaflops for that particular particular application of course these numbers are it's just like a dummy number that we've given here right and you we also have allows you allows you to we added some other things to guarantee the maintainability or let's say the usage and report of problems so we have a maintenance list so you can put your your name here so whenever you get a failure the report you say hey please contact blah blah blah blah and then you can put your stuff here so you can contact the people so we also have what's called tags because on cscs we do write tests for different setups so we do tests for production cases so we run daily uh we have maintenance tests so you're doing maintenance before and after maintenance and you also have diagnostics so when a node goes bad we're going to run this test to guarantee that the node is back it's uh in a sane state so this is a really simple case but this a real case where we have the code called arbor that's also a code that's developed at cscs that's neural I think neural network analysis and this test requires cmake so you can write cmake and you give some parameters for cmake and then you're going to compile the code and run the code and but because this code actually has different compilation conditions so you can do a serial code npi code blah blah blah blah you can actually import from the base test the npi version of the code and then you basically just say you add in the missing flag for cmake and then what's going to happen you're going to take advantage of the entire oh inheritance right from python and you're going to get a new test that leverage the base test so you're going to have two tests now one with npi one without npi and of course we can do better I mean we can do way more within that so we can actually have different compilations of different architectures so we can have haswell broadwell native and you can start parametrized tests and do all the options so if you want to run different conditions or different cases so I can go there and leverage basically a factory so I don't have even to write a factory class or a fake factory function to spawn different different tests because I can just basically parametrize my test right so this actually it may look like yeah that's a bit silly but in fact when you have a lot of tests and you want to spawn different cases instead of writing a function that's pretty big but just giving like one line saves all the space and drops a lot the maintenance costs right so but really I wrote this but how does it really works right it makes no sense I mean this cannot be magic right so on reframe we do something like easy build dies we have a pipeline we go through a set of steps that's always the same so when you start the framework you're gonna basically get all the tests from gonna import all the tests like easy build dies it goes imports all the easy blocks right and then you're gonna start running the tests and then you pick one test and then you check is this test supported by this environment or not of this system or not if it's not supported then you're gonna pick the next one you're gonna continue right because you can say I am pitch died as the before the test but I can be running on my laptop if I'm running my laptop it's not pitch died so that test was written for pitch died will be skipped right so and then you can look at the program environments that are installed in your system and also available for your test then you go for setup phase what does it mean it means that we create the folders required to run the test we cop clone environment to guarantee that we can reproduce the environment of the test then we compile if it's necessary to be compiled and then we run it can be a synchronous as I told you can be running parallel taking leverage of the scheduler or not if you run to run serially you can also run serially then we do the the sanity checking to guarantee the the sanity the performance and then you basically if if you pass or not you just clean the results and then finishes you go to the next step and then you continue right so the the the pipeline is given like an easy build so the one thing that we we we have defined for this pipeline and the the idea of the pipeline is to get how can I say uh reproducible uh run right so we can actually guarantee that whenever you run you're gonna pass by all those steps and you can define what actions you can take on those on those uh steps but sometimes you really don't need to go through that right I have compiled grammar I provide grammar on the system why do I have to compile again right so I can skip the the the compile test so we have done some some I can say I help the users to skip this past so we have provided two other classes a compile only test or run only test that you can compile of the test only like hello world if you want just to compile and to guarantee it to produce binaries or you can run the test it like draw matches run application because you have compiled already with easy build for instance so uh another thing that we we allow because we're doing python right users can do anything we can allow the users to overwrite anything we have done right so you can go there and overwrite the setup class and do something different and then play around with with with uh all the setups so it's just like before my test was not working I didn't go there overwrite the setup do something different because I want to do something and it just works right so that's that you can you can do you can do these these how can I say customizations as much as you want right so by that we actually cover all the possibilities of tests that we have on our regression suite we have never encountered a condition that we could not write of course we have encountered conditions that writing the test was not that good so then we worked out the framework to allow us to to better deal with these cases so if you write in regression tests and you are thinking that it's being difficult share with us your tests maybe it's a missing feature that we have to support on on reframe right so I would just jump the front end stage but running the the front end is more or less like that you just run this reframe and minus r to guarantee that you actually run in the regression and you can customize your configuration which this configuration is the map between the system that you are running in your tests so I can say oh pitchdine has t-mode installed so you're going to run t-mode oh my laptop has l-mode installed so you're going to run l-mode on my laptop so this is the that's the mapping between the system oh my laptop I have a clang l-l-v-m and in gnu on pitchdine they have in t-r-p-g-i all those so this is this is the mapping and this is maybe saying oh this is a bit silly but actually this allows me to write my tests from a laptop through a t-r-v-c-i to Jenkins to pitchdine I can bring my tests from all these systems because I can actually just add to this path right just add to this folder and I can take my test that runs on pitchdine and I start to run on juby on juby also on on the eulish systems right and then I can take my tests and start running on exit systems so just by adding then adding this map final right of course when I am writing my tests at css they do have this dyne gpu dyne mc the program environment from from there so going from my program environment from kray to a different linux box it's not doesn't make much sense right so my tests are not sometimes kray specific right so I can also change the path of the folders that have my checks so I can write checks for I don't know a specific cluster or I can write the test for another cluster so I can put the different folders so reframe allows you also to play around with these things so these two these three common lines I would vote as the most important ones because allows you migrate the tests or migrate the workflow to different systems and one thing that we do a lot is trying to separate or report to the users the different steps right because when you're compiling the code you are you have to compile it in an isolated place so that it doesn't scramble your home folder for instance you're running on a home folder in your scratch so we actually go to different folder and if it fails it's it's good that you keep the files there keep the scripts that we have generated there to run them so that we can try to reproduce so we have a sub and if it passes maybe my gram max input file is like one terabyte one gigabyte and I don't want to save it to the output so you make shoes also to store some files or don't store some files after the test has passed so we have the separations of of concerns so we have the stage directory where you actually did your test runs the output directory where you actually save your selected files that you really want to save for your test and we have this performance log directory if you have chosen to write a performance a test which means you chose to have analysis of the performance then you can also save a log of the performance you can say well but why that I mean why why well in a production system like we do in pitch died we do production runs which means we run reframe every single day so we save on the same folder always the test so now we have this file that per test right we have one file per test that contains the performance of the the applications along the days right we have the date we have the time it was rent so we can actually do statistics that's what we do on pitch died at least so how does it look like when you run it you you get like a google test some sort of output that you say run okay or run failed and when it failed and then you're gonna get failed and the reason why oh this guy failed and on the performance phase right so you know that the performance didn't match and their performance reference was 70 but it didn't fall within the so the reference was 70 and the value that actually was getting was 50 so it didn't fall within the range that you wanted because I didn't say that but we we can also say a range because we're dealing with production systems right so they fluctuate the performance if I if I get nodes that's a real local I'm gonna get a better network performance but my nodes are all spread on the system my network would be not that that good if I don't have a fully connected system right so and one thing that I like a lot is this logging so this is the how the log file looks like but because we also sent your gray log right we can make graph on her graph so I mean we can make graphs out of this this performance so every day the guy his own roster has to look at the performance if I had failed and didn't fail so we can actually get this statistics and though you know things have fluctuated or didn't fluctuate the performance so how do we do actually use how do you leverage all these things at css right so we do have around 241 tests so which means that we have 241 files with different test cases so this thing is spawns into different test cases what's the test case is the combination between different programming environments different variables that you have produced your primary trial test so we have like just 241 files more or less and then this is sponsored to I think a thousand tests is not mistaken my test cases it's it's but it's it's yeah but I think today is around around thousand something like that so it's really really spawns several tests of course we don't run a thousand tests because there's split into different systems you see here but that's that's uh that's uh how it looks like on pizza today so we do we do have what's called production so we run daily this this this test and the maintenance for the for the maintenance when you have maintenance and we do also for no diagnosis and we it's nice to have a production run but if someone has to be in charge of running that it's not so reliable so what we do we have Jenkins running our production our production systems every day so we have Jenkins lives installed in all in our clusters and every day in a production reframe is executed so which gives us really good statistics because we have not only the performance log that we told you but we can also know if the test fail was the mini right if I have in my hello world has a performance failure and I'm in our case hello what means that performance failure hello what means that you took more than 60 seconds to compile hello world that it's a really good indicative that we have a huge file system issue right so because we was not able to read hello world through a simple file so we can kind of try to guess so we have network tests we have iot tests we have performance tests into our system and you may see okay this is a bit like over killing right yeah but once operations was installing the maintenance in the cluster and the cluster was supposed to be the gcc was supposed to be emitting hardware instructions and suddenly stops emitting hardware instructions because they just forgot to start something else in the system and then we detected it and we told them and then they suddenly start to go back again right so it's really important to guarantee quality of service to be running these frequently so this is more or less how it looks like for us we have a pipeline to do for instance in this case a cache production system and then we open the the open blue ocean and you see how the pollution looked like and then you look at all the failures for that day and you can see what has passed or doesn't pass so this is nice this is great but it doesn't cover all my problems as a support guy i have to give all the versions of gromex because the users ask for the versions right so whenever i am compiling a new version of gromex what i do i have to take the recipe that i have for gromex change the the version compile again with easy build and run reframe right reframe allows me to change the the modules and everything but wouldn't it better if i can run easy build directly and get reframe running my test because i do have a gromex test so that's what we want we want to integrate reframe with easy build right so the the integration that we have we have proof of concept integrations we want to propose allows us to optima or optatively or you can choose to run this integration or not and why because sometimes you don't need it sometimes we don't let let me say one case we do provide easy build recipes to the users right the user may run the style that version of easy build but he doesn't want to test my test right he wants to test his input file so in that case he will compile gromex and then test for himself the the his input case right he doesn't have to waste his compute resources with my test right so for him it's not not so nice to be using using this so you can option to use it and some for instance like we have some tests that are mpi based and some systems you cannot run mpi locally so also that may may not be optimal for you so what we have done is basically we have leverage easy build hooks as proof of concept to integrate reframe and we have seen really good advantage of doing that the advantage is because you because you're doing hooks you are actually inside the easy block of reframe of easy build so i can leverage the entire easy build infrastructure that's really good so i can program completely what i want but and i don't need to change reframe right that's easy so i just go and call reframe and i run reframe the bad thing about that is that to actually support the easy block you need to apis from easy build internal apis from easy build which may or may not be available in next release right so you're gonna break and these things are not even tested right so these hooks are not even tested so whenever you do not go do a release or reframe or easy build they didn't see this code so they didn't even test it right they didn't check if it's properly uh running and another thing that's really interesting is that because i'm running the internals api which are tied to a version of the of easy build i also have to come come with like common line arguments for reframe right minus r minus c capital c all this kind of stuff so it binds also to the version of reframe i'm using because if i change the component argument of reframe then suddenly i'm not being able to use that feature because you cannot pass common line argument to hooks only on easy build right so what we really want is that instead of having just a hook we this integration becomes a full step into a into the easy build pipeline and to give an idea how does this this thing looks like that's the integration right this is full of concept of the integration easy build does not support a function called prepend fake modules path right what it does has load fake module that's what it does it does this prepend at the end just loads the module so what i have to do i have to remove the load command make the function here and put internally to this post hook because the api is not there right the function is not there why because whenever you start the software we want the to prepend the path so i can actually load that path when i'm running reframe right and then you can see things like if not dry run right oops hooks executed if you have extended dry run right this is really nice i was talking to can yesterday he said maybe a bug i said no no no this is a really nice bug it's a feature why because i can now see when it's dry run and i can say oh reframe command was this one right so i can do proper uh say messages when you are in dry run mode so extended extended mode but that implies that i have to include a lot of things from israel for instance at the vsc utu is like he was talking yesterday with zappier in israel 4 so this is doomed to be to be broken right so we do need these steps to be integrated so how does this integration look like today i mean how does this is even a simple case right because you see there's some things like if i am doing grow max then call the test for grow max if i'm doing amber and kuda though the disorder test for amber for kuda right but wait a second this is the test that i have for reframe but it may change but this is tightly integrating the hook so if whenever i update or add a new test to reframe i have to come here and update this table and if you go to this production file you're gonna see this if it's not just this is small this is just enormous it just takes the majority part of the of the function so this is not optimal integration another thing that's really interesting is that because hooks do not cannot be passed common lines i actually have to hijack the build options from import the build options and to do dry run i have to know that there is also the silent option right if easy build changes tomorrow the common line doesn't require silent this is broken right so i think i have been stressed enough that this is not a good solution right we need a proper integration and how does it look like to run today i do hooks reframe and then runs and then run post sanity check hook and that that thing actually runs reframe so and the output looks like that running amber and then doing the checking running run the amber gpu because that one the gpu version and reframe was executed so let me review just a bit we did the normal sanity checking of amber so that comes with easy build and additionally after doing that which means that i guarantee that i have the binaries installed i can now run the regression right i can run this step i would do but i would do like a monkey right i would i would run this after it was run i would go there and run reframe myself so i'm just putting the monkey step that i was doing into integrating through into easy build right so but what we actually want is something like that we want something like uh minus minus regression framework and you can choose reframe or you can choose ub2 or you can choose build test or whatever you want and then give additional arguments because if i'm running on my on my on my laptop i may not be able to detect it because my dns changes if i'm on css and at home and everything so i may may want to give like a different command line option to detect my system or anything like that and then the integration that we want is that then it becomes a full step of the pipeline right i think i'm running a bit out of time but it's really good because i uh with that we finished the integration step and i just want to acknowledge the people who actually have actively contributed so as you can see is a lot of people with high big salaries working every day reframe so we do have a lot of efforts on reframe so if you're feeling that reframe has no future it'll be dropped or anything no no we invest a lot of money on that so it's going to continue so it's a really safe project to to to catch up and uh uh to give you some directions what we have been doing with reframe is that reframe now does not support what you call dependency so i can we could have something like my test compiles and then my my other test compiles something my other test compiles something and then i have like this multitude of run tests that will depend on these compilations right we don't support that that's something we want to do we won't get some dependencies so let's say this test depends on that other so if that guy fails i won't run all that so we want to have a better container support we do have containers we can run containers inside reframe not a problem but it for us is all about the user story right so you want the user story the guy who's writing a test written in container to be as simple as possible right so we are improving our container support and you want something like a benchmarking mode where where the output summary is a bit different when it gives you a bit more benchmark details so my final message or my tech hope message is that reframe is you can write in high-level python your your your tests to the regression tests that can encompass your from your laptop to a super computer system and it gives you comprehensive outputs comprehensive reports that you can integrate with your great log infrastructure your log infrastructure and it's a let's say as an extra point here it's highly maintainable in csss this has a big future for that so with that thank you very much and questions thank you very much victor any questions so it yeah it's working here yeah yeah it's mainly a curiosity i was thinking what were the design fault that led you to list as key feature we have no dependencies on whatsoever uh yeah okay the second one may be related how do you store uh historical test data how do you store store store okay so can i repeat can i answer for the second then i go to the first one so we for the historical data we do have three things the first thing is that we do save the file the log files on a shared file system so whenever uh you run on a pit site on cache or other system only for instance that we showed here if you all the the logs will be saved on that thing so on the on that parallel file system and save in specific folders per system so i can keep track of all of those log files the second thing we do is that we have a log infrastructure for a centralized infrastructure in css for logging gray log infrastructure so we send the data to this gray log and then we can probe these data into gray log directly or you can do the third one that i showed here we can get the graphs of these data to a graphana right so that's we we say this this is a graphana the access gray log so we do as infrastructure itself file system and or a gray log system a logging system three support so for the first question why we don't want to have independence well because starting python uh for some users is a bit difficult thing right so some people don't want to have like a bloated python or some versions of operation system like red hat seven comes set up tools at zero point nine point something and if you do update the setup tools is 44 or something so we don't want to be bound by this versioning also of the dependencies right we want to guarantee that we we we run independently and if there's a bug in one of these dependencies then we have to update it and guarantee that the version and then my user have to update it so we want you don't want to to have anything related to upgrading and was it is me so another reason is that we a key thing is to run the test in the user in an environment as close as possible to the user so even now when you run your fame you don't do any module parts uh you just do you start from the environment you are currently in and optional you say you even unload the module that has loaded the frame so if you if you even uh unload the module that you have loaded the frame and you had a lots of dependencies that um perhaps has a lot has changed you know compilers or whatever changed basically the environment that the user test will run so if you in order to run the frame you had a set of dependencies installed then perhaps that could be a problem wouldn't replicate exactly what the users do when they log in with the clean environment we wanted the environment to be as clean as possible but yeah it's it's not a strict mandate but so we can still keep it we do have a pie test dependency of course yeah it's only if you want to run the unit test any other questions do you have a repository somewhere where you can share tests with other people yeah so the on the repository actually on the on the reframe repository you have what a folder that's called cscs tests so we can take a look at all of the tests we provide versus like gromax tests you can see how it's written but actually you don't have the input files so you cannot run exactly the gromax we have there because you don't have the input files but you do see the tests right but other cases like hello world you have the files like any video test you have the files uh network file tests we have the files but because of the size of the input file sometimes we didn't put on the on our git repository but just a question of the size because we didn't want the people to go git clone and take forever cloning the reframe right that's the that's one of the the ideas behind we're investigating to do git lfs like large file support like large file system but it's still being uninvestigated so we haven't come up with a solution yet that can share tests but we're investing a lot now on on sharing our tests and get people sharing that test because we we from from those stargazes and people that actually have worked the test so many people many of them are super computer centers so they do have the the set of tests right and it would be nice that they share our files and we share with them and then people can download from their side also if they if they're willing to do that right so we are trying to find a way of how to share between sites the same problem that easy build has that people have their own local repositories and want to share stuff and yeah but you can you can take all the css tests i have a question as well yeah one of the the things i've been struggling with an easy build a bit is the regression and i really want to look into reframe to run the easy build regression test to reframe so what would that mean do i just have to define a single python module that instructs reframe how to run an easy build test is that it or so if if you are if you want to do the the api integration right python to python or you want to do it doesn't even have to be with the api i'm happy to call the eb command from reframe as a command for inside reframe yeah you can do that you can actually write a test and say uh my build test would be uh if you don't have to do the build test preview command will be eb and then you're gonna get eb your file and then that's it can i run and you can parse the output so we do support easy build today that's not i mean we also support spark today but the user story is not optimal so we don't say that we support it but you can support it how exactly so we so we we do have this this uh build system right and uh so i can say build system is a single source which means that you're gonna do nvcc on this file right so there'll be a build command to do that and the build command will be composed of these these lines nvcc and blah blah with minus oh three all this kind of stuff but there's also a pre build command so you can self dot pre build command equal to something and then you can say eb blah blah blah blah blah and then you're gonna do the eb eb command and then it's gonna skip i mean you don't have to do the actual build so there will be to be none and then that's it you can just i mean no build command but this is you see this is it feels like a hack right so this is not a really nice support so we were figuring about how to leverage these into getting easy build because there's there's there's something we need to do with easy build to make easy build work out of the box right so and that's what we are struggling with because okay the stage directory i can point to the build directory all this kind of stuff but if my system's l mode does it guarantee that this easy build will be if my easy build was misconfigured will it work or not do i change the configuration to make it work or do i take the configuration from the system itself so these are things we are trying to answer ourselves before actually writing the full the full reply here or do i always pass the modules to command here to easy build so this is things that we are investigating let's say to see what's the best user story final thing yeah i mean coming back again to your question is mostly i mean we should see the exact test case how what you want to run because you can write a monolithic reframe that does eb something but remember talking with you about the dependencies so yeah let's let's see the exact test case because in any case you can run because essentially it will generate a state so you can run whatever you like and now it depends on how it is what exactly you want to do another thing just one on on the users actually we have already from us several centers that they're trying it or actually using it i know a higher state super computing center is currently having introduction nurse can l l n l uh they have again production rewrite they run some tests um and then we have other centers as well it's from universities and it's a route here to university we have i mean starting building up also i mean so the idea is now how we start sharing tests we it's also the posi center in australia niva in new zealand the i i know they're running on production there's also a couple of uh private companies in hbc that they're using also pbs but can then contribute with this as well so yeah so we hope that it will expand for that aspect yeah and if you want to do the integration of easy build and reframe internally we have to get easy build to be three part of the but that's uh but it's also possible to do that any other questions nope yep thank you very much victor thank you