 Okay, we'll continue, but the next presentation at the Easy Build User Meeting will be Shazab Siddiqui talking about his Build Test Project, take it away Shazab. Thank you, Kenneth. And I'd like to thank the Easy Build Community organizing this event, especially Kenneth. This has been an interesting experience doing this virtual. So anyhow, if you don't know me, my name is Shazab Siddiqui. I work at Lawrence Berkeley as an HPC consultant. I'm part of the user engagement group. So we take care of user support tickets and documentation training. I'm also part of the ECP project. So we're in the software deployment group where we're responsible for like building SPAC, E4S stack. And as part of this talk today, I'll be giving overview of Build Test. And it's kind of funny, it started out as a project when I previously used to use Easy Build. Didn't have support for like, software testing. So we'll talk about Build Test and also some of the current testing updates regarding E4S test suite. I'll talk about that. And just a summary of what updates last year have we done. So let's get started. So talk about what is Build Test. So it's a HPC testing framework for like, the facilities to basically write acceptance tests and for their target system. And Build Test is an open source. You can find the documentation on the right. It's also on GitHub. And Build Test is target audiences like HPC staff. The goal is to help the team to be able to gain confidence that their system is working as operational. Acceptance test requires like periodic testing. You know, typically as a facility, you would do this as part of a maintenance outage and make sure that you try to test as much parts of the components as you can. One of the key features is that we wanted to make a writing test easier. So pick YAML as the front end for writing tests and validate them using JSON schema. Okay. And there's also the need to have automation in order to create these tests and run them efficiently. And I wanted to mention that, Build Test is targeting system and software testing. So it's not like a software build framework with like easy builders pack which are good for installing software. So it's not that. And it's not like kind of like a build tool like makes you make. And yeah. So we picked YAML and JSON schema. And YAML is pretty easy for users to get started and it's actually been widely used in many open source projects. You know, just to name a few like Travis, Ansible, GitLab, CI, even GitHub workflows, Kubernetes, they all use it. And, you know, JSON schema is a vocabulary that helps, you know, annotate like JSON documents. JSON and YAML are like kind of interchangeable and Python they're both like represented as a dictionary. So it makes it easy for us to like convert data. So we use JSON schema to help validate the test but we write the test in YAML. And on the right, you can see kind of one of the example this is of a Docker composed recipe. It basically uses like this version based schema if you're familiar to write this. And build test kind of has like the similar concept of using like a version based schema to allow like, you know, writing tests but having further development of schemas. So it'd be like you pin to some specific version and then you'll write tests. And yeah. One of the major updates that happened at least last year was the introduction of JSON schemas. And this was a core change in how the framework previously was done, it used to validate test through its own validation process. And now using JSON schema we are able to do a lot more kind of rich syntax in YAML. One of the things that we have is we publish the JSON schema in GitHub pages. So on the bottom left is the schema documentation. We published the example tests for each schema including the markdown pages that document each schema and also the schema files themselves. And it's really nice on the right is kind of one of the example schema. This is the compiler schema. These are auto-generated markdown pages basically translating the JSON schema using this workflow called JSON schema to markdown. And then we just publish them into GitHub pages. And the way it looks when you're writing tests you can see all the type properties and their description. And it helps you kind of easily see what each property does. Now we'll talk about the command line usage for like typically building a test. In build test it's typically, it's called build test build. And you can specify a file using like the dash B option and it will like build from a single file. You can also build like all your tests using like if you specify a directory. If it's a directory then build test will just traverse the old directory structure find all the .yaml files and build them. You can also append the dash B option multiple times so it could be a file or directory. You can also build by tag. So this concept of having like a keyword string to search for tests. Like you classify tests by let's say network tests. You can use at least tag names to search for tests and build them. You can also like build like for instance a whole directory and then like exclude files or directly using the dash X option. And that can also be appended multiple times. So it's useful like if you wanna build an entire directory but like exclude one sub directory or something like that. Here's just an example of output. This is an input test. We're using the dash B option. We have a single file. It's basically a Python hello world example. The first thing that it does is build test will discover the build specs. That's what these tests are called. And it will once it discovers it will try to parse every single one through schema. That includes two schemas actually global schema and a sub schema. In this example, the schema is using script version 1.0 to validate the test. If it passes the parse stage and it will actually build the test. We can see the test name is Python hello. Build test will generate a unique ID and it will generate the tests like the shell script and then actually run the test. Typically the output would give you a return code and also like a status message. That could be either pass or fail. Now you can also build by tags. So this is just an example of building by a tag name called pass. And it will discover the build spec. In this example, it found that there was one test. In that one test it had four tests within it and it just basically runs all of them together. So we'll talk about the pipeline. And for every discovered build spec in build tests we go through like this five stage kind of pipeline. The first is we parse the build specs with the JSON schema. And this includes basically just a validation of the file itself before we actually build the test. And during the build phase we actually take the script and the YAML file and actually generate a valid kind of shell script. And then we finally run the test via what's called executors. And think of executors as like basically could be a local executor so it runs locally or a batch executor so it could be mapping to a queue in your schedule. Once we run the test we get the output error file and the return code. And finally we update the report which includes like getting all the metadata and test result and just storing it internally in basically a JSON record. So we'll talk about in this diagram is the build spec validation process that's done during the parser stage. And imagine you have this input build spec. First thing to note is that build test has like a schema library or what's called like the supported schemas. Currently we support what's called a global schema. And we have two primary schemas for the test that's the script and compiler. And those schemas are actually version schemas. The setting schema is for the build test configuration. So that's just another schema. So in this diagram what happens is you have this input build spec. The global schema is used to validate for every build spec. And it validates the kind of the outer level structure meaning in this example it will validate the version and the build specs. These are required properties by the schema. Maintainers is just an optional field but it will validate that. The version is required because it uses it to pick up the appropriate what's called a sub schema. And the type field is used to, it's a look up to find the appropriate schema. So currently we support script and compiler. So with the version we're gonna use the script version 1.0 schema to validate the section. The test section which is in this green box is the hello world kind of test. That's validating each test instance. So now let's dive into the actual structure of the test. So what each of these properties do. So the first is the version. This is a version that defines the schema version to use. And currently that's set to version 1.0. So as in the future when the schemas have newer functionality and versions then you can change this property. Those specs is like the declaration of like one or more tests. In this you define just some unique test name. So the test name is called system default target. And that's the name of the test. The executor and type are required fields. The executor tells which executor you can use to run this. The type is the name of the schema to use. You can attach a short description of the test. The tags is just a list of strings to classify how the test is. If you wanna use the tag feature to build the test. And then the run is just a script section. It could be a multi-line string or a single string. And that's where you define the content of your test. One other feature that we have is we can actually control how the build stage is using the stage option. So when you're writing a test sometimes it would be useful to actually just parse the build spec or just build it but not actually run it. So you can use this stage option and it accepts either parse or build as the options. So in this example on the right, we just use that, just stop after the parse stage and it will just tell you if the build spec is valid or not. And that's pretty useful when you're developing your test. We support return code matching. So by default, a return code zero is a pass in build test. However, sometimes you want different return codes because you may be testing like an actual failure test and you may have different return codes. So in this example on the left, we have four tests, right? So in the build specs, if you know, you can actually define multiple tests. So we have four tests, exit one fail, exit one pass, return code list mismatch and return code int match. The first one is an exit one fail. So it's gonna get exit one, so it will be a fail. So you can see that on the red. The second test, it's gonna get an exit one, but we're gonna say in the return code property, a list of each return code. So we say we're gonna expect a one, so it will pass. In the third test, we're gonna get an exit two, but we're expecting a list of return codes, one or three. So it kind of build test will look if the return code is in the list and if it's not, then it will fail. And the last test, we got a 128 and we're gonna expect 128. So you can do multiple return code. I mean, you can validate your test through different return codes to control like how the test is gonna pass. You can support like different shell properties to customize how the test is run. So the shell property by default is bin bash, you know, the system bin bash, but you can say, I wanna use different shell. So you could use bin sh, bash sh, you can use csh, tcsh, or even Python. You can also pass in Python options to it to the shell property. On the left is just kind of the description of the property itself. So from the schema docs, so it tells you exactly what the property, what it does, what the type is, and even the pattern matching. So that's pretty useful. So following to that, you can use Python shell, the using the shell property to write Python code. So all you need to do is say, change the shell to Python. And your run section where you actually define your script can be used to write Python code. And it's pretty easy to do like simple Python code. Like if you have like your own Python scripts, you can certainly just invoke the Python script using like a bash or sh shell and just change the shell name. In terms of scheduler, we support LSF, Slurm, and Colvault. And one feature that we added was having schedule agnostic configuration through this property called batch where if you don't wanna know like the inner details of the scheduler, like the options itself, the batch property provides a set of fields that kind of map to different schedulers. And the goal was to help kind of provide an interface between different schedulers in terms of the option. And we only implement obviously features that are common between schedulers. Currently, like for instance, like time limit is kind of provided by all the schedulers. So the field name is called time limit but in Slurm is called time in LSF is W and Colvault stash time. And this is just an example of a test that is using Slurm. So we can specify the node count CPU count time limit and it would just convert all of them into S batch directives. We also support Cray burst buffer and data warp. So if you have a Cray system, you may be familiar with BB and DW directives. And in this example, like what we have is we have these properties called BB and DW for defining the directives. We have this, what's called, in this example, we're creating a persistent data buffer of 10 gigabytes and it's striped access. And in the run section, what we have is we're just gonna create a random TXT file of one gigabyte block size, five times. And if you see in the output, the random TXT file is actually in the burst buffer directory. And yeah, so, and since this is a persistent burst buffer like you would see the output of the burst buffer after job termination. So you can do like S control show burst and you will see the burst buffer. So everything that we talked about before was the script schema. Now we'll talk about compiler schema. That's the, using the type compiler property. This is an example of basically compiler selection and the defaults that you can set for compilers. We have this, what's called an example of a vector addition using GNU compiler. So note that when we change this schema, we'll have different properties because we're gonna validate with a different schema. So for instance, source is a required property. It tells you the source file to use to compile your code. The compilers is the start of the compiler block. And in this block, you have to specify the compilers to use. The compilers are actually defining your configuration file in build test configuration. So you can actually find it using like build test config compilers you see on the right. So we have three compilers and you can specify the regular expression in the name property. So I wanna say I wanna find all compilers starting with GCC. The default is a section that's organized by compiler groups. So you can, and you have a section for GCC where you can define compiler flags like C flags and LD flags. And basically this test will actually run for both GCC 10 and nine. So let's dive into some more examples. This one, you can override compiler defaults. So we have this example of, so in our regular expression, we're gonna build in GCC and GCC as the compiler. So there will be three tests. This is basically a hello world C example. The source file is hello.c. And in the default, we say that for GCC, we're gonna use dash O one for the compiler flags. However, we want to override maybe the compiler flag. So we can use the config section and specify a configuration based on each compiler name. And you define the compiler name that you want. So GCC nine, three O will use dash O two and GCC 10 will use dash O three. And what Biltest will do is, if the compiler is searched during the, if it finds it in the regular expression, then it will just apply the configuration. If it doesn't find it, then it will just be ignored. So in addition to this, we can actually do like, imagine you have a single test where you wanna compile multiple compilers. So this is an example of an open MP reduction example using GCC and talent, right? So all you need to do is first specify the regular expression. So depending on how your compilers are set up and your configuration. So we say anything that starts with GCC Intel or program create, this is how the compilers are defined. We introduce a new property called all that tells how it's compiler configuration as inherited by all the compiler groups. And environment is this property that just tells to define environment variables. So we just say, we want to use four open MP threads. And in each of the compiler blocks, like GCC Intel and create, we can define how to actually compile the program. As you can see in this test, this reduction test was compiled with all the GCC Intel and program create. So that's pretty useful. So now we'll talk about some other features. Like for instance, the first thing you can do is you can actually load all your build specs into a cache using build test build spec find. And then you can actually filter and format these cache if you want to like search for tasks. That's the whole goal. And that's the primary use of this command. So in this example, we can actually filter the tests using different properties. So like, I want to query by the tags property in the test and say, find me all the tests with the fail tag. And it will show you the tests. So we see there's two tests. In this second example in the middle, you can actually format the columns. So by default, we give you like five columns, I think like name type, executor tags and description, but you can customize how the columns are for printing. So, for instance, I want to only format by name and tags and it will just show it to you. The format option expects the format fields to be delimited by a comma. And it will just print them in the order that it was specified. And then you can also filter by different or multiple key filters. So in this example on the bottom, you can actually filter, let's say, by tests that are like all of all tags that are tutorial and executor that use local SH and using the script schema. And it will kind of do a logical and filter. And if you want to know the different filter and format options, you can use help filter and help format. So as you build tests, they get reported into a JSON file and you can use build test report to actually display the test results. And similar to the previous slide, we have filter and format option that behave very similar, but they have different field attributes. They both expect key value pair and separated by comma. So the filter in this example, let's look at the one in the bottom left, like you can filter by a name, the test name. So I'll say like, I want to filter by exit one pass or show you all the tests and you can also format by different columns. So you can also filter by return code. So in this example in the bottom right, you can say, find me all tests with a return code too. And it can be any return code value. It has to be an integer. And then you could also filter by multiple keys. So the state property tells you if the test actually pass or fail. So that is a really useful one. So you can say, I want to find all tests that fail, but only on an executor, only they use the executor local usage, okay? Now we'll talk about some of the testing capability that we have in Cori test suite. So you can look at the link below for all the tests. We just basically try to test multiple, like various components of Cori system, including system network, file system, we try to test for, if all the login nodes are available, using PING or SSH, show that since we just learned, we want to test basic slurm commands, like SNFO, SQ, make sure the controllers are up, partitions are available, submit jobs to all the queues, make sure like burst buffers, we can access a burst buffer. On the application side, we like, you know, test different like apps like OpenACC, OpenMP, you know, like stream, we use shifter for like the container runtime. So make sure we can pull images to shifter and the E4S test suite, which is like the SPAC stack. This is just an example of one of the tests. So it's just testing SSH PING and uptime. On Cori, we have 12 login nodes, Cori 01 to 12. So we just make sure that we can PING them. We have six data transfer nodes, DTN 01 to 06. We just make sure that we can run up time against all the nodes and, you know, just run SSH against all the nodes and just run some command. So it's pretty, you know, pretty simple way to actually write tests. Next, we'll talk about E4S, I guess test suite on Cori. First talk about what is E4S? E4S is extreme scale scientific software stack. It's basically a collection of SPAC packages. And these are, you know, built and tested on several platforms. And, you know, E4S is deployed as a SPAC manifest. At least that's what we use, but it's also deployed as a container and build cache. And the one that we're interested is the E4S test suite. This is the collection of the tests that's used to validate the E4S stack. Basically to increase confidence in the deployed stack. If you clone this repository, what happens is there's a script called testall, that SSH that's used to like run the test. So all you need to do is clone test suite and then source some SPAC environment, source their setup script, and then you can like test a particular application. The test run validation test directory. So this is just an example of like Qthreads test that's run. So what we've done, I guess, I believe in January, like this month, we deployed E4S on Cori. It had up to 135 specs. And you can see the link for the documentation over here. The idea is we want to leverage basically E4S test suite to validate parts of the E4S stack because it provides majority of the tests. But we have integrated that like into the Cori test suite so that it can use build tests to actually run all the tests and have all the other tests that we have outside of E4S. We make use of GitLab to run the test to automated like schedule pipeline. And for the next release, we can like utilize like SPAC test to also run tests. So basically leverage other tools to run tests, but we also want to be able to develop site-specific tests that go through our custom modules and queuing system. To run the test, you just use the tags E4S. That's how we just classify the test. And what we have done in order to customize this, we have a custom executor. You said in our configuration, what it says is that this E4S property is the executor name and the options is the slurm kind of options that used to submit the job. So we're going to use a regular QKNL partition where the dash T is at 10 minutes. And then before script just tells the script to be executed for before running any tests associated to this executor. And the idea is just basically to source the SPAC environment and the test suite setup script. And on the right is just an example of a test. This is an audio test that uses the E4S test suite to run the test. I'm not going to go over this in too much detail, but this is an E4S pipeline that we had. It's available in SPAC config repo was recently added. But what I wanted to just mention is that we built the SPAC stack using the SPAC CI generate to actually get the GitLab to actually build the SPAC jobs in parallel. That's in this, you know, this generate SPAC pipeline and the build E4S job. And then finally at the end, we actually do like post installation tests. You know, in this example, we just use build tests to actually clone the repository and just run build tests on the E4S tags to actually run all the tests. And finally at the end, like we just report the test results. So this was all done in GitLab. Just some updates, you know, at least for last year, you know, we introduced JSON schema for validating build specs and the configuration. The JSON schemas were published to our schema docs. The, we removed the build test module features into the L module project, which we'll talk about next. And the regression test, we moved them from Travis to GitHub workflow because it just works much faster. And we added support for Slalom, LSEF and Colbot. And yeah, we moved the organization, at least from each busy build test to build testers. And we also recently added a GitLab CI check to run regression tests on Cori. And yeah, and then, you know, just to wrap up, it's my last slide. You know, build test is, you know, a testing framework. You know, it's there to help the facilities to actually write tests. The users just need to know basic understanding of YAML. And build test is not like a test suite. You know, facilities will still need to develop their tests, you know, appropriate for their system. And you know, if you want to get started, like we have like, I think a hundred plus tests in the Cori test suite. And you know, if you need help, you know, you can just please join like the Slack is a self-invite or if you want to post issue. And then the references are there for your help. Thank you. Thank you, Shazab. We have time I think for one question here and there will be plenty of time after Shazab's next talk on L-Module, which is right after this one. So if somebody has a question right now, we can take it now. So raise your hand and zoom or drop it into the EasyBuild Slack channel. Maybe I have a quick question, Shazab. Do you have any examples of collections of tests? Other than the ones you've showed? So are there any sites actively using build tests and sharing their tests with the community? So right now the Cori test suite is available on the same build test organization. So it's like available there for public. I had also set up an organization on Stampede like over there to actually run tests. However, it's hard to like, I mean, I think it's like easy for people to like look at other sites and start seeing how they do the tests. There are generic tests and build tests like when you clone it. I guess anyone can just write the tests there but yeah, I think it's mostly like you just have to look where the tests are and go from there. There are some tests that I have. Yeah, I mean, I think in all CF and yeah, but I mean, they're not right now public. Okay, I don't see any other questions right now on build tests. So we'll wrap things up here and get ready for the Elm module talk.