 So, welcome everybody to our session, which is called Enjoy Creating, Executing and Enabling Tests Using TMT, which is a tool we've been working on for about almost a year. Here we are, Miro, Mies, Petra and Pavel Volna, and we will guide you through this presentation. Let's see if it works. So, the agent for today, we would like to give some short introduction, really very short brief introduction, and then spend most of the time with examples, give you some real-life examples, some more examples, then a bit of theory, so that you understand a little bit the details of what is under the hood and then even more examples, then some overview of CR infrastructure status, some information about how you can get involved and give some space for questions. And we encourage you anytime you have a question, anytime there is anything you don't understand, you want to get explained or whatever, please just jump in and ask us. It can be on the chat or even audio. And I just want to remind that tomorrow we will have also workshop if you would like to get some hands-on experience with TMT. So, we started the introduction. So, my name is Petra Spiegel. I'm from the operating system CI, I work at Red Hat. I love to improve tooling processes so that they are efficient, elegant and good and so that one can enjoy them to use. Give her a tool, Miro for a while. Yeah, hello everyone, I'm Miro Vazkerty. I work for the testing farm team and also from Red Hat. And I have been involved in CI, or have been involved in creating CI for REL for the next couple of years. Yeah, and I give word to Pavel. Hello, my name is Pavel Valada. I work in Red Hat in the Ruby maintenance team. And we mostly try to push our work upstream into Fedora and that's our daily part of it. But there's GitHub as a forefront, but there's no Fedora there so this is a crossover for me. Okay, so thanks Pavel. And now we continue, we start with why. So, where we started, why we started to work on this project. At the beginning there were a couple of stories and here I picked some of the most important ones. As a developer or tester, I want to, I want an easy way how to contribute tests. Because finding out how tests should be configured, how the test has to be created, how the syntax is and all that can be very complex and is sometimes complex. Second one, as a developer or tester, I want to easily run tests in my preferred environment. So usually people have different preferences, someone prefers to run tests very quickly on local hosts, someone in a container, someone in a virtual machine. We wanted to provide all of this flexibility. I want to easily reproduce issue revealed by the testing tool. It happens that in the CI there is a problem. You want to debug, you want to see where the problem was, but it's difficult to reproduce the same environment in the same way. So TMT is looking also into this to make this consistent. And the last two, have more flexible test execution method that's stored at a single place. We were fighting with this little bit inside Red Hat because we have scattered metadata on several places. Something is different in the test case management system and we wanted to have everything on one place, ideally in Git. And that it should be human-friendly, unified and concise. So I jumped next and give a very high-level overview about the plan. So we had this presentation last year on Flock. And because we received positive feedback for the proposed, for the idea how to use this, how to improve this configuration, how to make it more simple. So here is where we are. So first, as I mentioned, we want to store all test execution metadata at one place in a plain text, human readable and versioned. And the best place for it seems to be Git. We would like to prevent any external dependencies, for example, test case management system. We want to use an efficient format to store this configuration in order to prevent duplication. Because if you have like hundreds of tests, there are some common data you don't want to duplicate in the plain text files again and again. So we used FMF, which is a flexible metadata format, basically YAML plus a hierarchy and inheritance elasticity. I will be telling something about the features later. So that is done and the tool is there. It's a tool in Fedora and it works. We wanted to have a clear specification which would describe how the things should work. And we dedicated quite some time to design this well. So here on the link you can click and see the metadata specification which shows some details about it. We are constantly working on that. And then the next step is the TMT tool. So to have something for users so that they are able to very easily create new tests, execute them on your laptop, debug them, enable them in the CI and generally maintain them. And so here we are. We have the TMT tool, some proof of concept, and we want to share it with you and get a feedback. And the next steps, we want to enable this consistent configuration on several places across products for testing on GitHub. We already have something there for testing REL. We have something internally for Fedora. We are working on that to make the pipeline available for CentOS and maybe more. And I think there was enough theory and start. So let's see the first steps. So if you start working, if you want to start working with TMT, you just install it. There is a package in Fedora so just the Unif install TMT. There are some options. The TMT is a minimal package which has some basic, the core functionality. But because the provisioners have some large set of dependencies, there are sub-packages. So if you want to run tests in container, you would install TMT provision container. If you would like to test under a virtual machine, you would install TMT provision virtual. And the TMT all is a special sub-package which depends on everything. So you get everything which we currently have. Okay, so let's see. So far the sound is okay. Can you hear me? No problem. Okay, fine. So the first command, the shortest one, as I promised, I will be giving a lot of examples. So TMT. If you enter a repository where you have metadata already stored, you can run TMT and it would look around and tell you what's available in the repository. So here we see. I'm testing on the TMT because we have a lot of metadata here. There are almost 30 tests, around 20 plans, and 140 stories. So that's the first thing you would do when you come to a repository to check what's there. Now to get a very quick start video component, there is this init command which does the basic initialization and it has templates. So if you're starting with your company, you would like to enable some basic small test. The only thing you need to do is to enter your kit repo, make this init initialization, and you have started. So I will show how this could work for you. So I create something like, let's say, this is a new repository. And there's nothing here, and I do TMT init minus TMT. So what would happen? Here you see there was a directory created, the template called mini was used, and this file was created. So you see there's directory plans and there is example plan. This is intentionally very short because we wanted the simple use case very short. So that's what I'm showing here. That was one of the motivations. If you start with testing, you don't want to learn much stuff. You just want to make something working. So this is the simple config how to enable testing of this basic small test. And the summary part is optional. So these two lines are the minimal config. So if you create a file like this, execute colon and script, TMT minus help works, should just work. So here, if I look around TMT, it says there's one plan. Now, if you want to run, again, very common use case, we wanted this way. If you come to a repository, everything is configured and prepared. You would like to run the test. It should be super simple to run it. So TMT run is all what you need to do. So let's see what will happen. I see like it might be a little bit slow because this is by default running tests in a virtual machine to make it safe and make sure that your machine is not affected or broken by some destructive tests. So here you see a virtual machine is booting. We are using the test cloud, Python test cloud from the Federer QA guys, which takes care of fetching the images when they are needed and then firing up the VM. This is done through Libert. So it seems like the video is taking some part of my machine and this is getting quite slow. I will be continuing. If I may, when we are waiting a shameless plug, when the vagrant plugin is finished converting to dynamic API, you will be able to use like everything that vagrant supports for your provision as well. There are multiple plugins you can use for virtualization. Super, super. Thanks for the note, Pavel. So here we see everything went well. You see, you had the impression of how the steps might be worked. You see that there's some discovery provision, prepare, execute, report and finish. And we see there's an error and we are like, what happened? I would like to see what happened. And I will see, oh, I could run the job once more with more verbosity, but that would take another minute or two. But what we can do is give me the results from the last round. So team T run minus, minus last. Give me report. If you see report, it would be like this, very concise. But if you're interested in some more details, you can add some more verbosity to it. So you'll see this is the log. Very good. Have a look at the log. And here is on the third level, sorry, on the third level, there would be the complete output. Where you see like the team T, there is no such file directory. So, of course, this does not work because it's a cleaner virtual machine and the command is not installed there. So let's change this for, I don't know, by version. And this would, something like that would work. But because it's slow, I would skip it for now. So let's jump forward. Minus, minus help is your friend. So if you don't know any, if you want to know some more details about any of the subcommands that TMT provides, just use minus, minus help. So here I'm giving a TMT run. So for running tests, TMT run minus, minus help would give you the detailed description about the options. And here are also the list of available commands. So what are these commands for? They'd describe the steps as you've seen already. There are several steps which would you go and run tests. The first one is discover. It is used for gathering information about test cases to be executed. And then there's the third one, it's prepare. Prepare, prepares the environment for testing, is starting some packages, starting services and such stuff. Then you execute tests, then you report results, and you finish. And we want it for each of these steps to have multiple implementations. There are some additional commands like logging, which provides you a little bit interactive shop on the guest, so that you can debug. And plans and tests which can be used for selecting plans to be run and tests which should be run. And if you are interested in more details about some of the methods, some of the plugins, so you do of the steps, sorry, TMT run, provision, minus, minus, help. So the same thing, provision, minus, minus, help would give you information about the steps. And here we see what's available. We can connect to a provision guest. We can use local host. We can use one minute backend. We can use a container or virtual machine. And again, if you want to know some more, you can say, I would be interested how virtual options, what virtual options are available. So here you are. And also at the top, you can see some examples for inspiration, how to write a configuration in the plan. But we will get to that. So that's the example with the minus, minus, help virtual. So if you specify the particular method, you would get some more information. OK, so that was the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, OK, so that was the, the simple, the simple way, the most simple example. And now let's look on TMT run, what this sub command can do and what options we have. So as I mentioned, the first step is called discover. Discover step is used for gathering tests, which will be run and it covers one of the user scenarios. Please just tell me what would be run. I don't want to run the test, but just, just tell me what will happen. So if I go, let's, let's, let's have something more interesting. TMT run discover and the TMP report. And you would see, you would see what, what are the plans there and without the tests discovered and, and that happens for all the other plans, for all the plans. Now you can select a particular plan. So TMT run discover plan name, plans basic would give you for that one. And you can also enable verbosity to see some, some more details. So here I could say plan name, plan name, plans basic, let's say, and I would do some more verbosity. So you would see, you would see for that the list of tests which, which go, which go like this. So that would be for the discover step. The next one is provision. In the previous example, we selected a step, which means that selected step would be, would be run. But here we see apply the option minus minus all, which means run all steps, but modify some. So here, what do we have here as I would like to run all steps, but I would like to modify the default provision, which is virtual and instead use, use a container for that. So let's, let's try, I have a smoke test here. And I would, so let's, let's try to run planning smoke. And we will be running all or minus a and the provision would be container. So minus, minus how or minus age container. So this is, this is how the smoke test will go. You see the provision is done in container. It's much faster. The test works there because I have prepared a container with, with TMTO already so it, so it works like that. So that's about choosing container. You can specify the image by default. The latest federal is used now virtual. As I mentioned TMT run by default runs in VM to make it, to make it safe. But the planner, the configuration can change that you can, but from the command line, you can override the configuration. So if you have like, like me, because I want to, I want to test quickly. I have provision local set. So one, once you need to, you can, you can override this default from, from the command line. And again, specify the image. Using a running tests local machine, a very, very, basically very, very the same thing. Just, just be aware that if the test is not safe, you can, you can break your stuff. So TMT smoke, I would say run on local machine. So it would do, it would run, it would run the smoke completely on local machine. Fine. So that's about local. We have a connect plugin, which allows you to connect to an already running guest, either with a private key or a login and password. And that would be it. And I mentioned also one minute tip, but that's available only for, for those who have the one minute tip available. Now let's see some, sorry, some examples for the prepare step. Any questions so far? Everything is working. I'm also reading everything on the chat. So everything. Okay. So for the prepare step, it's quite frequent or quite common that you, you need to install packages on the system under test. So we have a plugin plugin install, which is for now able to prepare the machine by installing packages from the, from the default repositories. You would do just TMT run minus minus or prepare how install and package, for example, HTTPD, you can enable a copper repository and install the package from there. You can also, you can also build a fresh, fresh RPM switch like local. So let's see, I can do make RPM that will create, that will create RPMs locally. And I could do installing these RPMs inside the container so that, so that there is everything, so that there is everything fresh. So the previous, the previous stuff which, which I did was running locally and then the container. And I could do prepare minus how is install. And now I see that the, for example, the TMT packages here. So I can provide the, I can provide them. I can provide the full path. And if everything goes well, it will be provisioned. Then there's a preparer prepare. It runs the install. Now something happens. If you are interested in more, you can enable the verbose output or debug output. Verbose is basically for users. If you want to see some more details, which are usually useful for, for those who run the tests and the debug output is usually useful for those who want to debug the problems. So here you see it went well. One package was installed and reinstalled because just to make sure that the latest version is there. And you could do also to install all RPMs from, from a directory, from a directory. So just, it will take some time but I can do it like this. You can do install from the directory and let's be a little bit more verbose now so that we see what happens, what's happening under the hood. So again, it will take some time. It discovers what's available. It provisions and does the preparation. Here you, here you see that the, the RPMs are copied and then they will be, they will be, all of them will be installed. Then reinstalled because there was no, it's not possible in the NF2D reinstalling like one step. So that's about install. We also have Ansible. So as you would expect, you can provide an Ansible playbook. You can apply it on the, on the guest so that it's prepared for testing. And we have also a shell method which allows you to like run arbitrary, arbitrary shell commands there. So I guess that would be about, about prepare. And now I would jump to the report step. So here as I already, you get, you've got already some glimpse of how this could work. So the report step is used for giving an overview of, of the results. So for example, let's, let's, let's see this. So I run TMT run, plan name, plans, helps. So here we see everything went, everything went well and the packages have been installed. Now let's see TMT run, plan name, plan helps. And let's, so if I run it, you see that you would, you would be given just an overview. So like it's executed. It, it says like Fortress executed, but I would like to see, see more. So TMT, TMT run last report would give you just the, just the overview. But you can, you can force to provide some, some more variables output. So here with minus V and you need to do force. At least for now, we can improve that in the future. But with the minus V, you would, you would see the results of individual tests with the, with two V's you would see for each test a link to the log so that you can inspect it and, and see if there is a fail so that you can see what happened there or you can provide three, like level three variables and you see the whole output immediately. And so that would be about the exam plus, the first bunch of exam plus and now a little bit of theory. So what is FMF? We chose or we invented FMF to store metadata and plain text files so that everything is in it. It's close to test code, test the source code. It's basically YAML, but we've added a couple of features. One is hierarchy in addition to the possibility to have files in the file system hierarchy. There is a virtual hierarchy which can be in the, in the files. You can define a value, a key up in the, in the hierarchy and it will be inherited and elasticity. I will show the example that would be, there will be preferably better. So here's the link for the project. So we started the simple use cases should be simple. And so, so this is how it works. It looks like basic YAML. You have a key value, key value, key value, key value and that's, that's it. But the hierarchy as I, as I mentioned you can provide a virtual hierarchy directly in a single file. So if you have a couple of tests just a couple of tests, you can have a single file and everything defined there. But if it grows you can separate them. And here this inheritance example shows how this how the, the sharing values sharing key, key values can work. So at the top we find the contact and text and test, which is common for the tests. And now we have two tests. One is for downloading. It takes three minutes to complete. But the record which has a different summary it has to it has to have some, some extended extended time. So that's the inheritance part and now just a couple of words for the elasticity. This is the feature because you can define this virtual hierarchy. It means that you can have everything in a single file. But once the project grows and you see, oh, it's too long. There's not enough, it's hard to maintain. You just separate the one file into separate multiple files or even multiple directories and the the hierarchy still works and you don't have to change anything. The metadata are the same but you can move the files freely as you need. So you can read maybe some more if you're interested in this concept the FMA documentation. And now to finish the theory part and the metadata levels so the metadata specification defines a couple of levels because we found it's important to give each use case a separate level to prevent duplication and a little test sharing and such stuff. So I will mention the level one are the core attributes which are common, like summary description order this is like common ground. Level one metadata we call those which are related to individual tests. So for example test script, it's path, it's duration, it's summary, tags and that stuff. Level two we call those metadata which are related to plans. This is like the set of steps which is describing how to prepare the environment, how to provision it, how to prepare the packages and how to execute tests for given artifact. And recently also added level three which is used for stories and this is like advanced stuff. If you find yourself sometimes lost, like what are all the features you would like to implement and what is already covered by tests and implementation documentation. So stories can be used for that. A brief example of level one. So this is example test. Here you see key values, summary, description, path to the test and the test script tier one used for like picking and filtering tests, tagging some tags should be single, duration maximum time so that you don't waste resources when the test is stuck and the required packages. Similarly, here is an example plan. So this is level two, the configuration of the steps. You can provide summary as for all other objects. You can specify the provision. So here we have virtual. We have specified the amount of memory because it's needed. And then the prepare step. Here you see there an example that you can have multiple multiple conflicts in a single step. So here we do we install a package packages HDPD and curl and then we do shell preparation and we call system control start HDPD. And then we do some very simple smoke tests just storing the file and checking it with the curl. So that's about the theory and let's get back to the to the examples before I give a word to get all. So the TMT plans sub command is used for maintaining a plan. So that means level two metadata. So if I if I'm in the so let's clear it. If I'm in the sorry. TMT plans. If I'm in the TMT repo you would see so there are 22 plans. TMT plans list. You can list them and you can filter by regular expression. For example, I don't know smoked smoke unit. You see we have planted planted plans here because we try all the combinations in different environments. So the tests are run in a different environment thanks to the FMA hierarchy features and sim links. But this is a little bit too advanced for quick introduction and TMT plans show TMT plan show gives you some more info. So let's say for the plans basic what what is there. What is there in it. So this is testing basic command line features. There are some description. There is some filter. So it filters available tests and picks only tier one and tier two tests. And it's executed because these are weekly tests. So it's executed as a big clip. A TMT plans show I've just shown and to get some more examples or inspiration, you can have a look at the or you can have a look at the plans here in the TMT repo. So that's the basic one. The core run it chooses it chooses the tier zero tier zero tests for some quickly quick checks. Here is an extra example show that it's also possible to list the tests directly in the plan. So here you see there is one test. There is a second test. It's included in the discover phase. So if you're not interested into like creating many files, everything can be in one file. This is running PO three deaths which need virtualization. So they are slow. Here you see the smoke, which is the super super easy thing. TMT minus minus help and nothing else and description summary. And we also run unit tests and unit tests they need to install some additional packages. So here are some prepare which shows how that can be done. And you spotted the fact that it's this main which is something like index. It's it's like the parent the parent metadata from which all other plans inherit. So here are it's possible I change the provision at one place and all the plans would pick it. They would inherit it or I can say let's just install all the freshly built packages from the RPMs directory and all the plans would pick it. That's a very nice feature to have this inheritance here. Okay so now I will be quickly finishing going through the last slides because the time is flying fast. So create if we've already seen the TMT in it can be used to quickly get you started. But if you want to start with some more complex examples you can use TMT plan and create. So for example I can I can show this full so I would go to the mini example which we had and do TMT plan and create a minus template full and it could be plans full or whatever. So this will create an example with a full config and here you see it's not just those two lines but you have an example for discover step which is fetching tests from a different repository and it uses Ansible to prepare them and prepare the box. So in this way you can quickly you don't have to learn the syntax you can create use the template and quickly quickly start. If you are not sure about the syntax check the help message. So for example TMT run prepare minus minus how install minus minus help and you would you would see you would see what what the config which config can be can be used maybe just add that the how is there important so TMT run prepare help can give you the preparation steps and if you put how there it gives you additional context to help. Okay. Yeah. Thanks. Thanks. The TMT test is a brother of TMT plan. It can give you an overview of available tests in the same way you can list the test you can filter them. For example I want to see only for tier one test. So for example TMT test list would give you like all the all the tests which are available. You can you can filter only tier one or tier zero and you would you would get only those those tests in the same way you can you can of course do the TMT test show so we could you could see like the information about the tier zero test TMT test land so far very very very stupid but the intention is that you would we would like to check the metadata against the specification and make sure that the test how they are defined there are okay so you can imagine this is something it's an RPM link creating tests in the same way as plan. TMT test create we have two templates so far a simple shell test or B clip test so let's see I go back TMT test create let's see there's some small test and I would select a B clip for that so let's see what was there there's a directory which contains a simple metadata level one metadata and a simple skeleton for the B clip test so now you can start and this is already working so so you can you can just run it and adjust as you as you need because one of the motivations for the TMT was to get us free from from the test case management system systems and I have everything indeed we have also TMT test import and export sub commands which are used for importing metadata from the old format from bigger make file and purpose and also from the from the nitrate test case management system so just to give you the idea of this would work TMT test import yeah so I changed the directory where there is a make file and purpose so let's see how it looked before so make file and purpose before that that was the old way how the metadata was stored and the purpose extra file and then this test case would be stored in a test case management system I would show you in a while and now the result is here everything in one file nicely put in a in a formal format test case in the CMS how it could look like so there will be about import and export and finally very very briefly TMT story as I mentioned the level 3 metadata can be used for storing the features or requirements for your project and you can so it happened to us very often that we had area and the TMT could do this and that and that there was not enough time to implement it so we just put together the story and store it so if you want to investigate you can do TMT story which gives you like the overview of the stories which are available TMT story list or show let's say some command line interface stories related to testing so you can see that we have these stories which are relevant to the test command on the command line but what's interesting here you can see the coverage so here for example TMT story but instead of show I will do coverage CLI test and you would see so these are the stories which are covered by code these are the stories which are covered by test and these are the stories which are covered by documentation already so in this way you can easily have an overview of the status of your project and that's it from me for now and giving word to Miro Miro? Can you see my screen? Yeah I guess you can stop sharing So I'm going to speak a little bit about the testing infrastructure so last year we have enabled packet integration that means that if you are using packet packet is a tool that is trying to automate the integration from upstream open source projects into federal operating system so if you are using packet you are already also running partially some testing so packet is able to run these tests that you can define in this TMT format in directing your diskit and when packet builds basically the copper builds from that PR that you submit to it it will the testing farm one of the services that we have will run automatically these FMF tests on it one thing that we added from the last year is that each copper build that you build via packet is also automatically tested at least for installation so even if you don't onboard any tests in the way that Peter described then the testing will be still executed right and then those results are available next to the PR checks we have all everything documented here on this on the packet documentation page where you have basically step by step step by step instructions how to onboard the testing and test your PRs I didn't say that this testing is only available currently for github itself because the packet currently integrates on github then the test results display as usual for github setting in the end of this slide here this is how it looks like in the github interface where you have the test results available from the TMT execution and there is some simple console available so that is for the packet integration I have to say after a year of being available this testing there are quite some components already using it packet lately released this dashboard which has this type of test runs where basically we can see that there is quite a lot of testing going on here mostly our packages but there are a lot of others on board it so if you have some github project feel free to look at this integration so one thing that we didn't accomplish to deliver for this year's flock or nest is the federal integration unfortunately where we wanted to have in federal CI the possibility that you place you place the test directly in this git and you run the tests from this git repository so that is ongoing and we have slipped a month so at the end of this month there should be an early access to that integration it means that federal CI basically when you build on new body update it will be firstly enabled for row height then if in this git there will be some tests available the federal CI will call this service that we will have and that will execute the tests first install the packages from that build execute the test and report it report it as automated checks directly in the body interface so this way of running tests will be used also for the generic test which is already there for some small portion of the packages it looks like this so this is where this test will appear in the body interface currently these are only static analysis checks or the generic checks that are being run there but here once this is ready and you have tests they will appear here as an additional test so that is for the integration part the system that we will be using to integrate is it we call testing farm it's basically a testing system as a service that we are preparing inside Red Hat and it will be shared across LCI, federal CI and packet possibly others in the future we have a public API where basically everything testing will go through that's just in short we will be supporting generic package level tests in the empty format so that is something that federal CI will be able to easily add in this test format then will be functional tests that will run against a container or a VM that will be hosted on federal AVS infrastructure and we will support also running the functional test in the standard interface what is currently the current format of the federal CI functional tests that I said so we welcome any feedback and contributions here are some links that you can follow there is a very good documentation I encourage everybody to look at that it helps a lot I don't know what I'm sharing this sorry and there is a federal quick start guide which we will be using on the tomorrow's workshop so the quick start guide will guide you basically through the main process of getting the tool and then the packet integration does that I mentioned later on the federal quick start guide will also include the information how to onboard the tests to federal CI when that is ready if you have any feedback we have an issue tracker on github and we have also at the empty channel that I'm giving you back the words okay so there will be there will be it from the content and now a space for questions is there anything you would like to ask comment any feedback I'm not sure how it works with the audio also not sure I think somebody has to ask for access so we can enable him so I know if you have any questions feel free to type it we are here still few minutes yeah I have to say it's a very interesting experience with the virtual element actually my first this year yeah so how does everybody like it yeah so seems everybody is happy on the IRC, I tried the IRC on the chat I tried to answer all questions cool okay so if there are no questions we can call it done okay so thanks all for attending the session