 So do you all have access to the virtual cluster that Victor set up? It should be accessible here. I mean, you must have like a reframer, XX username and then reframed.cscs.ch. Yes, do have access, but. It should be empty basically if you are in, let me see, there is a chat here. But in the available modules, even the frame is not there. There's no need. We're gonna do it everything from scratch. Okay, all right. So it's gonna be really, I would say, Victor, let's move, it's better if we use the chat of the Slack instead of the, and generally, I mean, the Slack channel from the frame inside the tutorial, EUM 21. So if you guys can ask questions there. Yeah, because it's also better for marketing. And so let me just copy, paste the link if you have not been, if you have not joined yet. I'm copying the link here and then the channel you need to join is one, I mean, we're already five minutes past, we can possibly start and then if more people join. So first thing is, yeah, I mean, I hope you all have access. So if you have any problems, just type in the Slack channel and there's also Victor in there who set up the cluster. So today we're gonna, generally in this tutorial series, we're gonna follow the tutorials, they are online. So they're just updated. So you should be here in 3.4, which is the version we're gonna use, basically the subsequent version, that version Bob, so you might see 3.5.0, but it's essentially the same. So we're gonna go through the tutorial and I'm gonna adapt those tutorial exams to make their own live on this reframe cluster. And along the way, I'm gonna show you how it is and then you also practice along with me. So let's start. So the first thing is I'm gonna start really from scratch. I don't have anything here. It's empty as in everybody's, so, oh, sorry. So installing reframe and make it work is really pretty straightforward. So all you have to do is just clone it, get into the directory, and then there is a little script called bootstrap that will basically install reframes dependencies inside the installation directory of reframe and the Python path, et cetera, they're not at all touched. So they're, yeah, sorry. So just a reframe knows where to find its own dependencies. I'm gonna show you in a moment. So you don't, it doesn't mess at all with the installation just a different way, instead of having its own dependencies in the repo, it, yeah, just have them inside it. So essentially it fetches everything inside this extended directory using P, et cetera. And there are also some options to the bootstrap script in case you want to build the documentation or if you want to use the gray log bindings which we're not gonna look into this tutorial. So we were good to go with just this sequence of commands. So essentially here we have reframe. Everything is installed like this. That's it. So then there are alternative ways either through EasyBuild or through Beep or through Spark. Any of those will work. The easiest is compare between EasyBuild and Spark. I think it's EasyBuild because Spark will really install the Python for you. Whereas in EasyBuild, we're just using the system Python. So let's start. That's the very, very first example. So here is a HelloWord program and here is the test. I'm gonna, it's the one that they showed in the presentation. So it's really pretty simple. So all the tests in reframe, as I said, they are classes that they have to eventually derive from this class, a regression test class. There are also some subclasses. You can also subclass it yourself but they have to eventually derive from that because it has all the logic of the test plus some metaclass magic to do some fancy stuff. Then all the tests must define valid systems and valid program virons. If you don't, then reframe will skip the test and it will issue a warning. If I'm not mistaken, we added this feature recently. And then in this particular case, you say, okay, here is your file that you need to compile and after you compile, you run the executable you produced and here you have an expression that says, okay, fine, try to match this part in the standard output. Let's run it. That's the very simple example. So also here, by star, we mean that this test is valid for any programming environment, any valid system. So it will essentially run immediately. And by default, reframe comes with a configuration that will run on any system. We will see the configuration in a moment. So yeah, I forgot that before running. Let's see the structure, how this is structured. So you can have any type of directory structure you like but at the lowest level, for example, in this one, you have your Python test with your test files and actually gonna have multiple reframe tests inside the Python file. It's not that here, just for illustration purposes, we're using one reframe file per test. And now inside that directory, inside that code, that the directory structure, you have an SRC directory where is your source code of the test resides. And here you can see the hello.c. So reframe, if you do this thing, it will, it knows when it loads the test, it knows where the test resides. So it knows its prefix and then it can look up inside the SRC for a hello.c. And then it will figure out also from the extension which compiler to use and since it will produce the executable by itself, it knows the executable. So then it can knows how to run it. And here is the basic of how you run. So essentially when you start a reframe, we load the configuration file from a set of locations you were gonna see in a moment and then you can pass with minus the option at the path either to a directory where test files exist or the specific test file. And then you always need an action. So it's either a minus L, let's try the minus L first. What is it? Ah, yeah, sorry. That's from my previous testing. Sorry for that. So everything also is clean. So it's like you. So you can see here that you get the listing of the test of the test that it found and where it was found. Some information here also which was the settings file. I'm gonna show you which one is the built-in. And it doesn't show the built-in because you should not generally change that. So the built-in is here. The built-in settings file. So let's run it. Okay, so that's it, let's run. And you can see that I run on a system named generic on a partition named default and using a built-in environment. Yes, the answer to Kenneth's question. So let's see what it produces. So generally, after it finishes, while it runs, it uses a state directory. We're gonna see that in a moment. But when it finishes, it saves some files in an output directory, let's say or archive, we just use output because that's what we have been using all the time. And it creates a directory structure there, like output, then the name of the system and the partition, then the environment, you run and then the test. So here it just saves some files. So like the build script, it generated. Let's inspect them. So you see, so it really uses a generic compiler. That's the built-in environment. And what's the job script? Just run the what you produced and then you can have output. And actually, this is what inside the test is self.std outpoints to and similarly. And then in your tests, you can also specify additional files to store. Now, one other thing that is relatively new. So for those that you've done, yeah, you have been using it for a while, I think since the previous release, we generate a JSON report with details. Now, by default, this goes to dot to the frame reports. And it's a single file that gets over it and every time. But you can change the pattern so that every time, every time you run a reframe, it generates a new one. If you open it, it's a JSON file with really all the details of the run. So the command line, when it started, it laps time for the whole test, how many number of cases. And then you have for each of reframe runs because the frame can retry, you can use the frame to retry fail checks, et cetera. And you can see the actual test cases, it has run job ID, all the details. So that's useful for debugging or for archiving or for keeping those if you want to process them later. And you can essentially, there is this report file option which you can use. And then you can also have like session ID, I think it's called JSON. So every time you run the reframe, we generate a new file, a new report. So let's see. So you see the my report, et cetera, for every time. Okay, now let's move on. I mean, if you think, yeah, you can interrupt me with questions if you want. So now let's move to another one. Let's extend our test to compile also for C++. And let's say we have, here I've shown you we had another one, hello CPP, the same hello word. I mean, it might look very basic but believe me the hello word programs, especially if you go up in PMPI and then in combination of the two reveal regressions. Okay, no, here we introduce a new concept. So before you remember, we had the simple test which essentially we wrap a test and that's it. Now here the test is exactly identical except that we wrap it with parameterized test. Now parameterized test decorator takes a set of arguments which is its argument of that decorator is a list of arguments to be passed to different instantiations of the test class. So in here, you say the first time you tell to reframe instantiated with C then with CPP. And actually the frame will instantiate multi-lang test one time with lang equals C, the other equals CPP. And then you can use that as we do here and parameterize the test. So we don't have to define two tests or having inheritance or things like that. It's much easier and it is very, very powerful because here you can have a function that generates those and you can create any combinations you want. We're gonna see some examples later on. So let's see how, now let's list the tests. So you see now that the reframe also generates automatically a name for the test which is currently a bit naively. So essentially it tries to convert to string whatever is the argument. And in doing so, it also converts any non-unforumatic character to underscores. So in more complex cases, you might see some big names with lots of underscores but it's something we're looking in ways of improving and allowing the users to define mappings, et cetera. But you can also set the name for yourself but you should keep in mind that the names reframe then they must be unique the names of the test or the names of the test that reframes loads. So you can have tests with the same name but if you don't load them at the same time, it's fine. If you do load them at the same time then the reframe will issue a warning and I think it will either stop or just ignore one of those, I don't remember. So let's run. Okay, so we have a failure. Now, here is a failure, the normal type. I mean, it's a failure. So whenever the frame failures it gives you for each of the failed tests it gives you more information. So here you see the stage directory. This is the working directory for each test. So basically the frame executes everything from that. So it copies any resources to it. And the idea is that this is like self-contained so that you can even start it and give it like to a vendor if it's a self-contained test case. And then you have also how you can reproduce it. But here if we go in the stage directory basically we're not gonna see anything special except for the files which were copied from the resource directory of the test, the SRC. The reason is that the error is a bit more fundamental because the frame says, complains that it doesn't know how to compile a C++ program. I think it's time that we look into the settings. Okay, so let's start. By default I told you that the frame uses the default settings which are here which I'm gonna copy in a setting in a config that's called My Settings and let's inspect it. Now here is the configuration. We're not gonna focus on the logging for the moment. So as I told you also in the talk it's a big JSON file that is a JSON object that's stored inside the site configuration variable. You can also have it as a JSON file if you wish. But the Python format is more convenient if you want to dynamically generate stuff in the configuration file. So here is the, let's see the system. So here is the generic system and it's pretty dumb. So it say, okay, match any system that you run on. That's why it recognizes every system. So what the refrain does when it launches it issues hostname and it tries to match the hostname in one of those hostnames that it finds in the different configuration entries and it will pick the first match. Then each system defines partitions. And here we name one default that has the scheduler which is local that is, yeah, there's no scheduler. Launcher which is local that is just run the executable. And here are the environments that you should be testing on this partition. And here, these are just pure names. So here we have one called the built-in and then there is another section in that big object that's called environments where you defined the different environment. So there we defined the built-in and we have just CC. Now I'm not gonna change this. I'm gonna augment this configuration file with our own. So I'm gonna copy here, that's from my laptop but I'm gonna adapt it to our keys here. Okay, let's call it reframe, E-U-M, the name of the cluster, the description is optional, but cluster. Now the hostnames here I'm gonna do, I think it's reframe, okay? And then we can even be a bit too strict. And partitions, let's name just one partition for the moment, the login is where we're running scheduler local, launchers local. Now I'm gonna still keep the built-in environment but not exactly as it is this one. I'm gonna tell you why, why this is a good practice. And then let's define another one called GNU. Sorry, we didn't have licenses and stuff so to install like Intel, et cetera. Now let's define them. So we have one building, so reframe but we need to redefine it. And actually I'm gonna do it. I forgot something, sorry for that. One thing we need to do is we're gonna say that in this system, there is a module system and it's L-MOD. So then reframe when you start putting in configuration or in test modules, it knows how to load them. If you don't put anything, it will simply ignore them. So here, wait a second because I'm not the good VI user to automatically. Okay, so let's redefine building. Okay, now we redefine building and we're gonna use here the built-in GCC that they are on the system. If you see they are the H3 or something. And here we're gonna say that this definition is valid for the reframe EUM system only. So you can essentially redefine the environments as you wish. And then we say we have also the GCC environment which I'm gonna copy this thing here. And here we call it GNU and here this stays the same. And actually one new thing is that for this one we're gonna load, I think it's GCC 903.0. We can or we, yeah, I'm gonna remove this for the moment just for plurality and we're all set. So we have... Basil, there's a question about the built-in. I think we have built-in multiple times. So if you want a different name for that environment. No, you don't need. So because here we restrict its scope. We say that when you're running on the reframe EUM system the built-in is defined like that. And as a good practice, I would say that it's better if you define your environments with a standard name. So then it really makes standard way also of how you define your test. And then in the configuration you specialize them for the different systems. For example, for another system the GNU environment might be different or, yeah, so, or the FOSS environment later on in the examples that I'm gonna show. And another reason is that I keep, we prefer keeping a built-in environment that doesn't load any module is for when for tests that you don't want to compile anything. And in those cases you don't need to do to load any software stack or any compilers. So if you just want to do some host name checks or some system utilities or things like that. Sorry, so only name plus target systems and current host affects which environment definition ends up being used. Yes, yes. So the process is the following. So reframe as I said, look at the host name then tries to match it in a system with these lists here. The first match it finds it picks up this, the current system in this case reframe EUM as the current system. And here it will start trying the different partitions we define. Now for each partition we say which environment we want to test. Now here I know that I am on reframe EUM login partition and I'm gonna search for a definition for that. So first it will start for a built-in definition for the reframe EUM login partition. It doesn't find one. Then it goes for a definition you reframe EUM. So these are scoped and in which case it found one. If not, it will fall back to the default built-in definition. So that's how it works. I think I'm done with the configuration for I think it is saved. Okay, now let me see. Yes, everything is saved. Okay, now let's try to put it down. Now one thing is with a minus C uppercase option you can point to your setting files but for convenience we're gonna export that to an environment variable. So here you see some differences. So now it says it picked up the reframe EUM system and now started to run on each of these systems partitions which one have one which we named login and then using the different environments. So it took the one test and run it twice here and then the other test and then run it twice. So remember we have even a single class essentially and we run already four times without changing anything. That's the nice thing about it. And now let's also inspect just to show you in the output again. Now you see we have a different system here appearing if the directory login. Again, I'm gonna pick this thing and test RFM build SH and now you see. So the frame did the module load use GCC. So it generates everything. Okay, now that's a convenience for the tutorial but it's also good if you're loading the frame from a module file then you can use environment variables to set it to change its behavior but you should be careful with environment variables. You've seen that I messed up when I started because I had this environment variable set. Okay, so let's move on. So we now have a multi-threaded version of Hello World. I'm not gonna go into the C++ code but now let's have a look at this test. Okay, so here again, same things. Now we want to set some environment variables sort of some compilation flags. So the way to do that, the frame has what is called build systems. So you can specify what is the build system that the frame is gonna use to build and in this case, we define it's single source that means that it has a simple file. It's just a single file that it must compile and then there are several properties depending they might change depending on the build system. So you define it here, then there is some magic that essentially converts that to an object. So then you just say CXX flags and we pass those flags. Now, we can also set the executable options with this thing here. The frame still knows which is the executable because it compiled it. So it knows which one it is and then the part is the same. Okay, now I'm gonna try to run this. I took this example basically directly from Howard. Okay, we have a failure. So we have a failure, both of them failed and the reason is build error, standard error can be found here. Because compilation errors can be quite lengthy, we don't paste them here, but now that I'm thinking twice, perhaps we can just truncate it and point you to the file. So anyway, let's see what the problem is. So let's go to the stage directory. Okay, so when it is undefined reference, so yeah, the problem is that on my Mac, this works, but generally we have to pass the minus pthread. Now, I can do just that and yeah, I will do for the moment, just this thing to keep things simple. We could differentiate based on the programming environment, but that's later in the tutorial. And I think, okay, and now I'm gonna run. Let me know if I'm going too fast or too slow. Okay, it seems to be happy. Now let's examine a bit the output because, okay, we have a multi-threaded test, but yeah, no, I'm gonna reframe if you are looking. Let's pick GNU and that's the threaded test. Why? Why is it behaving so well? Okay, I like this more. I don't know why GCC9 prints that nicely, but here the output is mungled. Obviously, I mean, our sanity check is too naive. So we are now gonna write a bit more, a slightly more robust sanity check. Just for demonstration purposes, I have everything in the yellow two MP, which this is what I'm going to use. So what we're doing here, now here we're demonstrating a bit this thing about expressions here, the sanity patterns. So before you remember, we had this SN assert found and then a regular expression and the output, the output file. So here we're a bit, what we're doing is we're actually counting the matches and then we make sure that these are 16. Now, there is a whole library inside these packets. It's all documented. If you look into the documentation, sanity functions reference, that provides lots of utility functions. And here we say, okay, find all the matches of this in standard output. This produces back a list, which then we can take its length and then we assign it here. And then we just make sure that this equals 16. Now, the nice thing about it and why you need to use those sanity functions is that none of those is evaluated here. None of these things. These are just deferred for a later stage in the test execution. Here it's just you're specifying where it's going to be, but you specify it in a very natural way. And let's run it. It fails now. Okay. So we get an error message. It's like, yeah, 14.0 equals 16. It's usually seen other testing frameworks like PyTest, but you can also adapt the message. But I'm gonna skip that. You can see the documentation on how you adapt the message. So you can have like number of found messages 14, whereas expected 16, it's a bit really. You can have whatever you like. And then you see also the failing phase. It's a sanity so that you know it's a sanity problem. Okay. And now let's fix that. Again, just for demonstration purposes, it's a different file. So here, the trick is to add a, I mean, that program accept the sync messages flag, which is a pre-portal sort of flag. You can either pass it the CXX flags or in CPP flags. I would just use the same convention as like the other build systems. Oh, sorry. And yeah, this as well. And there we are. Should be, everything should be fine. Okay. Now, let's run a performance test. So here I have the classic stream benchmark. Now let's go and see the new things introduced with this test. Now, we're fetching this test from its source. Now you can do that. There is this attribute called pre-built CMDs, which is a list of commands that you execute before running. And then same stuff, single source, string to C, stream array size, different stuff here. We're just looking for this and the pattern and then, ah, I forgot this. So there is also these variables which is environment variables, obviously, that you can set. Those will be set both in the build phase and in the run phase, but you can, depending on where you define those, you can control in which phase they're only defined, but we're gonna see how this can be achieved later. And here we have the so-called performance patterns. So here, the syntax is exactly the same. They use the same mechanisms with sandy patterns. And you, so you define here environment performance variables like copy, scale, add, try it. You can have as many as you want or just one. And you tell the frame how to extract the value for that from the output. And here you say, that's your regular expression. And the trick here is that you use a capturing group. You can also name it. And then here you tell the frame which capturing group to extract and you pass it also conversion function. So the time that this comes out from the frame from this extract single function, it will be a float. And then you can use that float and do whatever you like with it. In Python code. So the same for scale, add and try it. Now, let's run. Okay, this new option performance report will, if there is a performance test, it will bring the summary at the end. So now you see that they haven't specified the configuration file and they took it from the environment variable. So here you see the values we're getting. And yeah, and none is for the unit because it couldn't find any unit. Now, let's add some reference values. And actually, yeah, I'm just using four cores, but I don't remember. Yeah, let's stick to the, let's don't open too many fronts. Now, we're gonna add reference values. Okay, to add reference values, this is the attribute that you have. So the self dot reference, which is essentially contains different dictionaries. And the keys are the different systems. It can be also partitions. So, and you can have as many as you want. So I'm gonna keep that. And here we're gonna have for the login nodes. Now I think it's a bit naive, but okay. And here are the thresholds. So the thresholds that you allow and say, if it is beyond those thresholds, minus five to plus 5% issue an error. And actually here we can do, we can basically replace those with none. And that means if it's higher, don't issue an error. Only if it's lower than 5%. And we are, I think we're set. And now we can run it. We can, we can overrun the time, right? Kenneth, today. Yeah, you're already overrunning 15 minutes, but yeah, it's not an issue. You can definitely go for another 15, 20 minutes. Okay, okay. Yeah, everything's good. Now you see we have units because the frame figures them out from the, we specified those in the references. And by default with the standard configuration, the frame comes with which we copied for our config here. There is every time the performance test is run, creates this directory per flogs. And then it has the frame, I mean, the name of the system, the name of the partition and then creates a file where it depends always the different runs. So here you might see more, it's better if I do like that because yeah, I was running also yesterday, but every time you run, it's going to be appending to this file. I'm not going to cover other aspects of sending, for example, to syslog or to graylog, but you can see the documentation. There is one question in the chat. Okay, second and third values in reference and percentage is always, yes. You can't use actual values in that one, in the references. But if you want to use the references for, there is a, if you want to use it for doing a validation check, like you have Gromax producing an energy and then you want to make sure that this is within some bounds, you should not do that with the performance patterns. You can do it with normal sanity patterns by combining them. Ah, yeah, I forgot that. Okay, could you please share your My Settings file? Yeah, I forgot to tell you that basically if you pull the repo, the final, final config is EUM21. So that's contains everything that we're gonna start, we're gonna build up during the tutorials. So it's this one, I'm calling base here. Okay, now, time to port this to an HPC cluster, which is supposed to be this one, which in the original example, I'm going from my laptop to be stained, but here we're gonna simply expand our config. Okay, so since we are in the same system, I'm not gonna define a new system. I'm just gonna define the partition that actually goes to the compute nodes. And I'm gonna copy one partition from here and just adopt it. Okay, so let's name it this partition CN, these are, let's say, compute nodes. It has LERM, it uses SRAN. And now here, the access is how you get access to that partition. And in this case, it's a real partition, but it can't be constrained, it can't be anything. So in that case, you use the minus p dot. And here are the environments we're gonna test. Now, again, we're gonna use built-in GNU, but we need an MPI compiler as well. So we're gonna define an environment that we are gonna call FOSS. And then the max jobs is actually how many concurrent pending, not how many concurrent jobs reframed around. And by default, it's eight and we're gonna make it 10. Now, let's define also the FOSS here. This is MPI CC, MPI CXX and MPI F90. And here it is FOSS 2028, that's it. Okay, now, let's run this stream. Here, it's much more because there's Intel, PGI, PGI and everything, but in our case, it's much simpler. Okay, let's rerun some tests. Now, the minus n option selects tests by name and you can have patterns like here. So say any of those, we can even see them like that. Okay, now I'm running them. So you see they run on Login and then it tries the CN partition using now three environments. So the same test, we haven't even touched the test. It runs the six instances of the test they run now. Same for CPP, same for stream. And imagine how this could scale if you have multiple environments without even touching the test, except for the CPP, CPP, CPP, CPP, CPP. Without even touching the test, except for example, adding a supported environment in the test. Okay, and then we get also report here. Now, let's also see what was produced for the CN partition, right? Okay, I'm gonna pick the GNU, can RFM is here, stream thread. Okay, so you see now you have a build script. Sorry, the job script with whatever is needed, the module loads, exports, anything. Now for the same exact test, if I go here, I just get this one. So now moving on, trying to adapt the bit, the test to different programming environments and things like that. Yeah, give me a sign when we're, I'm close to the end of the tutorial of the first one. Keep going. Okay, so let's now get this one, get here. Okay, so here we have the GNU already defined for our system, but I want also FOSS. So actually here you can have any number of programming environments. It's which one of those will run depends on the system. Here you just have a big list. You can have a big list. Again, same things. Now, in our case, it's not so different, but for example, Dain, for example, it's programming environment. You must have different compilation flags. So you need a difference is a test behavior based on the programming environment it runs on. Now to achieve this, first of all, this is not reframe, this flags. This is something that I defined. You can define it as UBs. The different flags and let us also define for FOSS. And also we can even specify for the different partitions, how many cores we want. And I'm gonna use that for the reframe. The course is also a test variable. It's not something from reframe. You name it as UBs. It's just a way of programming it. You can program it differently. Now, reframe UAM again. I think here we are with 16 nodes. There is a feature request that those, the number of cores, et cetera, be defined in the configuration, which I supported. And we, it's gonna be in sometimes. So you won't have to do this thing. No, it's a feature request, not the pull request. And then on the compute nodes, yeah, we just have to. Now, let's see the juicer. Type of reframe ROM, not UAM. Reframe, yeah, ROM, yeah, sorry. Okay, so now you need, in order to differentiate a reframe while the test runs, define some variables, which is there are three of them. So it's the current system, current partition and current environment. Because you've seen that when we were running the test, it was taking the same test. It was essentially running into a different partition, different programming environment. So you can differentiate by just doing an if on those. There is a little tweak here because those variables are not, except for the current system, I'm just, except for the self-current system, all of those, the rest, current partition and current environment are none when the test initialized. They're only set when the test has started to run. So that means entering the different phases of the pipeline and especially after the setup phase. Now, for this reason, we have what we call the pipeline hook. So you can have, you can attach in every step of the pipeline. So imagine it just, you can imagine as the lifetime of the test. It has a setup, then the build phase, and the run phase, the sanity phase, and the performance. And you can attach any number of functions before or after. So in this case, we want to set accordingly the compilation flags. So we name a function, we call it set flags here. And we set it to run before the compilation phase. Now we just pick the current environment name from the current environment dot name. And then we, with that, we index, we look up into this self dot flags array, which is our array. And that's it. Now, same for the setting the number of threads, the different environment variables. So again, we use the current partition here. We get the full name, which is the frame, E, U, M, colon, C, N, or colon lagging. And then we use that as the new CPUs per threads and then you set the variables. Now, those variables will only be set when running the test, not when building. And that's the trick, for example, when you want to define some environment variables only when running. Now, I find this mechanism is quite powerful because it's also very, very composable because you can have hooks and then you can, in base classes that they are inherited, for example, from derived classes. Now, let's run now. Let's run and let's see also, we get. So it's the multi-sys test, okay. So it runs on login, MCN, using GNU. So we should be seeing FOSS as well. Okay, GNU, FOSS, I think, yeah, these are all of them. Yeah, we could even have built-in, but nevermind. So now we can even go here and see output, reframing, EUN, CN. Let's pick the FOSS, stream multi-sys test. Let's see the, so here you see the build script. We module load W, you get MPICC, no other environment variable set. And here you have, because of the hooks, you just generate those here. And basically that concludes the tutorial for today. I thought, I hope you got an initial understanding of how the frame feels like. So I don't know if we have time for questions. I'm gonna be around in the Slack channel if you want to ask anything. Yeah, we do have time for a couple of questions if anyone has a question they wanna raise in the Zoom session. Or if not, as mentioned, maybe the Slack channel afterwards and I guess throughout the day. Yeah, yeah, I will be available, I mean, yeah. But I've shown you here how exactly we got from the bear. I mean, as we cloned the frame and we ended up, I don't know, in an hour even going slowly because we had to demonstrate things and have it running some simple tests on all, I mean, on the cluster. And basically I can get those tests and on pitstained and run as shown here. We have a question up here. Minimum changes, yes, please. Hey, thank you for a very nice presentation. Thanks. So my question is, are we linearly sort of planning to go over the whole sort of read the docs as a tutorial or are there sort of elements outside of it? Yeah, there are new, I don't know, if you have read the old ones, there's new stuff that is just published. But my plan was that, yeah, to go over the tutorials which have now enhanced. I don't know if you were expecting something different or... No, this is pretty good. I'm just asking the question. Yeah, yeah, yeah. So the plan is, the only thing is tomorrow we won't have so much, we will not be able to overrun. So I'll try to be faster. There's no session tomorrow, it's Thursday. Yeah, Thursday, sorry. Yeah. Yeah, so the plan is to go over the, not the fourth one, I don't think we'll have time, but two and three to go through. And actually there's some new stuff here that was added just in 3.4, like merge today. Thanks. You're welcome. Okay, if there were no more questions, I suggest we can wrap up the session here. Okay, thank you guys for attending. The next session in this part for reframe is Thursday at quarter past 10 in the morning UTC. Yeah. So using the same Zoom link as you used to be. I would have another one, it is in hand or no, it was... No, I think, yeah. Okay, okay. I think we're done. Okay. And then the stream and the recording. Okay, yeah. Just one note, the cluster will be alive until the end of the week. So we will kill the cluster by the end of the week, okay?