 I welcome to our talk, during which we would like to show you how easy it can be to enable tests in third row GitHub and possibly many other places in the near future as well. So for the introduction at the start, I'm Beto Spiegel, I'm from the operating system continuous integration team, and my focus is on improving tools and processes and best practices, and this TMT is one of my focus items. The next one is Miro. Hello everybody, I'm Miro Valkerti, I work testing farm team and we are creating a testing system as a service, which is now used for federal CI internally for row and also for packet and our main format run and how we how the test are defined. Pavel. My name is Pavel Valena. I'm a maintainer in federal working for RedHeath and I'm looking into the adoption of TMT on GitHub and Fedora CI from my developer, Frantisek. My name is Frantisek Mecels and I'm part of the core services team and I focus mainly on automation currently on TMT. Okay, so thanks guys for the introduction. Let's see what we have before us, so on the agenda, after a short introduction of the topic, we would like to give some real life examples of the TMT inaction, then some more examples and then some even more examples. Basically, the session should be filled with examples how to how TMT works, how you can use it, what you can do with it and how you can use it to enable tests in an easy way. At the end, we would like to spend some little time about the overview of the infrastructure status so where it's already working and then give some space for questions and feedback. Also, as I mentioned, please share your feedback in the chat anytime and at the end, if we have some extra time, we can dive into some a bit of tech. Please note that in the schedule or in the chat, I include links to the presentation. So if you don't follow or because I will be switching between the terminal and the slide so you can open the slides and shoot them there. So okay, so that you have a nice experience, I turn off my camera and we start the session. So let's start with why. So at the beginning, there were a couple of user stories and based on them, we started the work. These were, I would highlight, I would like to, as a developer tester, I would like to have an easy way how to contribute tests. So it was complex to various test case management tools and doing some needers have to create accounts and stuff like that. Then the second one, I want to easily run tests in my preferred environment. So someone prefers to run tests in low-couple, someone in containers, someone in virtual machine, we would like to support all of them. If there's a problem in the CI, I would like to be able to reproduce the issue very easily. That's another one. Next one is I want to have the flexible metadata for tests stored at a single place. So not scattered across multiple places. And I want to be so that they are human-readable and easy to use. So here's a high-level overview of the solution we propose. And this is, so, store all the metadata related to test execution in one place. Everything in plain text, human-readable version, under Git. No external test case management systems. We chose or we developed FMF, the flexible metadata format, which allows us to prevent duplication. For example, you have 100 tests and you don't want to store the same information about the required packages 100 times. Then we designed a specification, which describes the syntax, and the next step was to implement a tool which would provide an easy way how to create new tests, how to run tests from the laptop, how to enable them, and basically maintain. So that's the test management tool, the TMT. And the next one is to enable it in the pipeline, so it actually works. And many of that has already been done. So let's see. So if you want to start experimenting, we encourage you to experiment during the session. These are the first steps. So you should install the tool. The tool, it's available in Fedora, so you can do just the end of install TMT. You get the basics and the features. There are extra packages which have additional dependencies, like for provisioning in container or provisioning in virtual machine. If you don't care about this space and want to have everything, just install TMT all. I would like to point out that there is a copper repository. So in order to get the length is greatest, you can enable pss.tmt and install tmt from there. So we would have the fresh bits there. Also if you want just to experiment safely, you can use our fresh container images from Quay IO testing firm tmt or tmt all again. The first container is quite small. The second one is a bit large because it has a virtual provisioning dependencies included. So that's for the installation. And we start with simple stuff. So if you come to a repository where somebody has already prepared tests or has configured tests, the only thing you do is tmt. So let's see. I'm in WGIT repository. I say tmt. Oh, okay. So here's nothing. I changed to our tmt repository where we have our sources for tmt. And I say tmt and here we see. So there are a bunch of tests. We have nine plans. We are extensively using plans for testing things. And we have also 157 stories which describe the features we want to implement and are in different statuses. So in this way, if you're not sure if everything is set up, if everything is working, you just write tmt and it says what's available there. And let's start with actually enabling the test. So we wanted to make it very easy to enable a test. So you don't need to learn the syntax, the right location for the files and all the stuff like that. So the only thing to enable tests should be something like a simple command. tmt init does exactly that. tmt init supports templates. So if you have your repository and there's nothing, you can do tmt init minus template minimal. Okay. So now you're tmt and it says, okay, here's one plan. So let's have a look how the plan looks like. So here we have a minimal smoke test. It can be like this, really two lines, nothing more. And this is just enough. Okay. So this is for wget and we do a very simple smoke test, wget minus minus help. And that would be it. Okay. So what now? That's the example I just shown and the minimal example. Okay. So it's time to enable the test. So we just get at, you check out, check out some branch, let's, okay, smoke test. You commit, let's say enable basic smoke test. You push it to your fork and create the actual pull request. So there should be, there should be URL shown here. Okay. So let's see, find the link. Here we have it prepared. It will be against raw height, create the pull request. Okay. And now wait a bit, now wait a bit of course. But if you're impatient, you don't want to wait. You want to try things. So let's see. We thought that the scenario when you come, you want to develop a component and you want to make sure that the change you did didn't break anything. And then you want to verify it should be very easy. So TMT run should be the way how to run all available tests. So okay. So if we are here, I can do TMT run. Let's see what would happen. I will use dry. Okay. So here you see, so there will be some discover phase, provisioning virtual, some preparations, no preparations, no tests because the driver does not actually do the checking. So let's try actually what will happen here. So as you see, the first step is discover. Discover checks for available tests. Here we did a simple shell discover because we have a single line with a bit of smoke test there. So it revealed one test. Then the second step is to provision the machine. By default we have virtual so that you have full support for full virtualization where you can basically test anything. We could make containers to make it faster, but we thought the virtual would be the best default because it doesn't affect your laptop. So it's safe, but you have the full virtualization. So that's the default. And you see, okay, provision is done, preparations. We didn't have any preparations. And for the execute step, TMT one test was executed, but there was some error. Oh, okay. But what was wrong? So you don't have to run again and like in verbose mode and debug mode or something like that. You just run TMT run last. TMT run last report would show you what happened and it says it's done, but I want to rerun it again. So I place there minus V for more detail verbose mode. And here you see, oh, of course, it was not installed. So what do we do? CI pipeline will take care of that because there will be a fresh package. So it will make, take care of that. But we say that it would be good to prepare. So I say prepare, how is to install, and the package would be double yet make there, okay, TMT run. And let's, let's see what will, what will happen. But until the command is finished, help is your friend. If you want to know about how to, how to work with, with the tests or which commands are available. So for example, here, TMT run minus minus help describes the individual steps. So what discover, provision, prepare, mean, execute, how we can report, resolve, and finish. There are some additional subcommands login, which can provide you an interactive show on the guest. You can select plans and select tests, which will be executed. If you dive a little bit more, TMT run provision help, it gives you an overview of the provision methods, which are available. So here you see, we have connect local, one minute tip, and container virtual test plan. So these five provision methods for running your tests, and you can, you can freely choose what's the best. You can dive even, even more deeper, the internal provision, how virtual help. And you would see the options for the virtual plugin, virtual provision plugin. So for example, specify amount of memory or disk. Let's see what happened here. We are installing a little get, slow metadata fetching. So that would be, that would be, I will be continuing here and describing stuff, what we have. So the discover step, as I already said, it will tell you what would be wrong. So imagine you come to a point, you don't want to execute the test because it can take hours. You just want to know what will happen or what would happen. So you run just the TMT run, a TMT run discover, oh, and we see the test already passed. So that's, that's very nice. And we can see that actually there was output from the, from the help command and it nicely worked. Fine. So second time, TMT run discover. So was there one to select it? Oh, okay. So that's, that's the script zero, zero, because it's not very interesting. But let's see what we have, for example, in TMT report, TMT run discover. Let's see. So a balance plan. I test another basic, basic punishment tests, features core, 10 tests. We have some extra plans for minimal installation and installing gears and pip to make sure this, these are broken. And we also test have a couple of plans for probing, testing that the provisioning actually works. So someone, if it's a set of tests, it's run against connect, container and everything. Okay. Good. So this is discovered. So now you can, you know how to discover this TMT run discover, you can shorten it if you are lazy. Plan name, plan name core. Okay. So you can select. Tell me what's on the, oh, okay, discover verbose. Let's see. So we have 10 tests, 10 tests for the core, features core plan. Okay. That was covered. And now for the provisioning part. So you can provision, you can, if you, you saw I selected the discovered step and it ran all in the discover step, but you can run all steps, but adjust some. So for example, here, TMT run all provision, minus, minus how container means run all steps, but instead of the default virtual you run in a container. So let's make it much faster. Our previous scenario, TMT, TMT run all or a, let's see provision, provision container. And I will make it faster because I have fresh for the language. So let's see. And we are interested to, to see what actually happens under the rules. So let's, let's leave this way. Okay. So it's the, the provisioning container is ready. We do the preparation, install, get, do the make, oh, that was much faster. Okay. So that's, that's, that's, that's the way how to go if you don't need forward virtualization. If you need TMT run is the default, but you can explicitly override with what's specified in the plan. So you can specify provision virtual while, for example, I'm used to, because I want to test the local changes very quickly in the, in the top hierarchy. I specify the provision is local for, for all TMT plants. Fine. So that's virtual. Of course it's possible to run the local. That's what I have just, just shown to you. And if you have a test box, you can just provide the IP or a host name with the password or private key. You can connect to a machine. And that's about it for the provisioning, provisioning part. Now let's jump to install for the installing who you can, as you, as you already, as you already saw, you might want to install some additional packages. So you can, we have this prepare, prepare step, and you can, you can use the plugin install, which can install a package, package HDPD each, it can enable copper install fresh packages from there. It also can install packages from local directory. So let's see. For example, this is interesting because you make RPMs, make RPM, so you get a bunch of RPMs. Okay, some test is running written. So let's see, TMT, the RPMs, no R, so here, here we have the RPMs. And I want to test these, these RPMs. So I would run, for example, TMT run when core, this name smoke. And I would run all steps, but I wouldn't run it in a, in a container. So provision would be container, and it would be image would be Fedora fresh to make it fast. And we do the preparation, prepare, install, and directory is this. So it would take, it would take the packages from the directory. Here you see, it checks the directory, and we, okay, we see not much details here because I didn't turn on the publicity, but in this way, you get, you get the fresh RPMs installed on the, on the box, in the container. And actually it has the latest bits, which you have just modified. Okay. So that would be for the install part. We also support, of course, and simple playbooks. So you can provide a playbook to prepare, prepare the guest by the playbook or some shell script. No problem with that. For the reporting part, you've seen that already. So TMT run plan, one name core, I can, I can run some plan, and it gives just some brief overview, but we can use TMT run last report with different verbosity levels and see some more, some more details there. Fine. Okay. Now what's, but what's nice. So let's see. So what we have currently here, we are just still installing the packages. Okay. What a shame. But I can, there are, finish, can use the container where it would be stopped. I have some example, example plans. And here comes the new feature at TMT status implemented by Fanta, which was just recently. So here I have this feature, some recent run. And I want to make a report on it, TMT run. The idea is this, and I do the reporting. And I want to do it like an HTML and I want to open it in a, in a web page. So here we see, here you see, there are like three plans and all are green. Very nice. You can jump to the logs directly from the web page, which is quite fine. Okay. So, and for the status, there are different verbosity levels. So you can, you can see what, which plan already has finished, or for example, which steps were finished in the plan, which is quite nice. So that would be for the report test results. Okay. TMT plans. So far so good. Everybody. Can you hear me? Does it work still? No problems. It's good. Just as you are typing, it's quite choppy. I would say that like the, the screen sharing is quite slow. So as you are typing fast, it's, it's, it's sometimes hard, hard, hard to watch. Just, just maybe that one observation. Hopefully the slides, slides will, will help with the examples because we have most of that covered in the slides as well. So here we go with the TMT plan. So the TMT plan is command for managing plans. The plan looks like this. There's some summary. You define the provision. You define the prepare step. You define what should be executed. That's, that's a plan. What you can do with plans is you can check what's available, TMT plans. You can do TMT plan, TMT plan list, or you can do TMT plan a show to, for example, the core, the core plan and you would get some, some, some, some more details about that. So that's four plans. Listing, showing, and we could dive into some examples, but I think let's go, let's go on. Of course, if you want to create a new plan, there's the command TMT plan create and it would create a new plan. There are some templates so you can choose what would be the best way. Some extra features for, for mass mass pre preparing plans. And now for the test. This is, this is probably the bottom of the most interesting one. So if you, as a tester developer, you want to create tests, of course. So instead of old, ancient, ugly, make files, this is the way how we want to go forward. So a little nice YAML file with support for FMF with the hierarchy and that stuff. So you place the summary description, path, test, you might take some tags or tier. You want the duration record packages and that's it. Nothing more, nothing more. Again, tests, TMT test, TMT test also, TMT test, test list. You can filter, you can filter, for example, only tier one tests and do such stuff. So that's for tests, tests show, of course. And TMT test lint can check whether the metadata are well set. It has some basic functionality for now, but we don't really extend it. For creating tests, TMT test create, TMT test create, test smoke. So if you, if you want to create a new test, this is, this is the way, this is the way to go. We can try this in the, in the data repository. So let's see, TMT test, create, it could be become a, it would be tests, basic, I don't know, something like that. Create template, sorry. Okay, so here we see, we have one test and one plan. And, and if you look into the, in the, into the repository for the basic test, you will see that there's some skeleton for the, for the metadata and the B-clip skeleton actually. So we can, you can quickly, quickly start with this, with this example. Libraries, I wanted to mention libraries because quite often you don't want to write the same thing again and again. So we have B-clip libraries. Enabling the B-clip library is as easy as this. You use a require library, HTTP, HTTP. And in the test, you can then use libraries features. So for example, setting up quite complex things around like Apache with all the certificates, certificates and putting, putting things at the right place. So you just do HTTP secure start, HTTP install, certificate. And it's done. Okay. So much, much better. And that's basically it. Yeah. We have, I have an example here. I created blue request yesterday, which, which is, which is enabling, enabling this simple test. Here we see, this is the simple double get. So there's some setup. As I mentioned, setting up starting certificates, creating some test file. And here we do the actual check and it, everything works, including the certificates and all that complex stuff. And even there are some green results. So scratch build and this big test shows that everything passed. And the test actually had a full log. So that would be about libraries. We want to make it easy to users to come from, from the old solutions like nitrate and have migrated that data to get. So we have the tnt test import tnt test export. And, and so, so, so let's do some example here tnt examples convert. You will do tnt test import. And it would check out, check the make file. Okay. I didn't do it. So let's see. So let's see. Okay, that would be for the nitrate. Okay. So everything imported. Here we see some new file. It has all the stuff imported from from this. There might be some things you want to, you want to remove because this is, this is just a test case. But here you see, like, there's a bound test on test, which, which was imported from the nitrate and you can, you can right away use it. Quite fresh support we have for context the just for those who know test case relevancy. This is future, which can be used to filter relevant tests. You have a set of tests which, which can be used across distros for some of them, you know, that the feature was added later, you can adjust the metadata based on the context. So here you say when the destroy is less than federal 33, we turn it off. I have an example here and the test discover memories, because I, I needed to disable the test, because it runs just too long and it could not run in the same pipeline because full virtualization even stuff. So I did something like that. So the test is disabled and less context how, how is set to set to full, which, which means that you will you provide a context how is full and then then it would work. So the empty test. So, let's say filter enabled enabled enabled is true. This was let's see so it's your view when you see the library it's, it's here, but we provide context, how is full, we run a full. There's no no test disabled everything is enabled. So that's, that's how it works. That's what I just shown. Now for debugging tests. This is quite useful. So if you debug test, you don't want to, for example, do the complex preparation many times. So instead of repeating the provisioning and prepared part, you run TMT, TMT run on to a report. And then you see the result of the test and then you just to execute the test again. So this is this is how you can quite fast debug the test. We started to work on some aliases, because these commands are just a little bit too long for now, and we want to integrate them. Here are some examples of some common aliases, which, which, which you can use to to make use of this, of this functionality. I want to also mention to the possibility to log in. So you can log in actually into into the machine. So, for example, I can say TMT run provision, provision container login and finish. So it would run the container. I can do something in the interactive shell. I leave it and finish will take it, it will close the, you know, close and here it does for all the plans. So, so I have to, I have to stop it now. You can log in in during a selected step. So at the end of the preparation, you can do some many tasks, or you can say please give me a login if there is a failure of error, so that's possible. And for the stories, we close, close this overview of the objects. TMT stories is used for tracking stories. So the features requirements which need to be implemented. TMT stories lists. So we have, we have a few of them. We cover the specification as a stories. So if I was mentioned mentioned, for example, core adjust, adjusting the context. So here's the story and you can show it. TMT stories show core, core adjust. And here you see the things which are relevant for it. And we see it's also it's implemented here and it's tested here. So that's about stories. Yeah, maybe one more example, TMT story coverage, coverage would show you how are you doing. So here you see, for example, for the CLI test. Area, the coverage is like this for code and coverage by test and coverage by documentation. So, okay, so that would be the overview of what TMT can and I think the time is to very quickly say that we have this enabled in GitHub. So packages are built in copper by Beckett, triggered by pull requests or comments, and there's a full TMT support quite freshly and we're very glad for that. It's supported in Fedora CI, so packages built by the CI pipeline built in Kochi, triggered by pull requests or builds, and we have full to full TMT support including context adjust quite fresh. Thanks to me for cooperating on this. And the real CI internal, it's also possible to use TMT for testing pull requests and builds. I wanted to include a couple of real life examples, but I want to ask if we want now to jump into some questions or should I continue with the, with the examples what you say. Mero, is there any question yet, so I think we can continue, but if there is any question from the chat, I will try to answer it or I can just interrupt and we can answer it together. So, so here are a couple of couple of examples. So, the smoke test. Actually, we did. I tried yesterday and we tried today. So let's see what happened with our enabled basic smoke test pull request. It's open. And this is the test is running. Okay, so we'll see. It might take some time. So, the result from yesterday. Here you see, example of a minimal smoke test, and the results and if it is the test. It looks like it looks like this script and it, it should, it should show what's the content. We use the version or help commander. I use the version there. So, so here you see the version. First, how it works. It's a reliable example on federal CI. So I use this to request to disable a couple of tests do some other filtering. And it, it seems to be working because, because only those couple of tests were selected. We have nice failure here. But instead of running like the 50 or 60 tests we have it run only, only the basic part. GitHub pull request. Just the fresh github from, from yesterday, I guess, here the results are as usual in the, in the bottom bottom section. Okay, testing Apple and thanks to me, we have nice, nice overview of xunith results. Here you can click on the contracted to make it hidden to collapse the results and you can have a look. If there's fail for some details for the for the test output. So there will be for this FMM example. I just wanted to highlight that it's possible to reference tests from other repositories. So for example, here in the FMF plan, I'm referencing tests from TMT to ensure that a change in FMF does not does not break TMT. So in this way, you just include a new integration plan. It references the repository, it might choose some ref and fix, fix, fix some sort of tests and it runs there. So in this way you can, you can keep components which are related in a good shape still. Here is a live example of LFTP. Yeah, I was just doing fresh pull request here guys are using a test namespace for their LFTP tests. So from the RPMs namespace, they reference the reference test here. And the advantage is that they can store the tests at one place and then they reference them from Fedora, from Github, from internally, and you maintain only one place. Another example of short test is our test for Bech, Bech, Konchel and other POSIX compliant shells. And they have this set of plans in the test namespace, so it's possible to test changes. So if you do pull request for a test change, you would receive results for that change. For example here, we don't have the good naming of the tests, but for all five tests, those for all five shells, tests were run and it gives these results. So here we find an example of open source, a badge of open source test, previous internal are open and we want to go in this direction more. So an example of success here. And we also store full-end transition. So users are still able to schedule new tests migrated to Git using workflow tomorrow. So users are using FMF in the CI pipeline to define how they would be, how they would be around. So RPM inspect for example, there's the discoverer, discoverer one step and other provision description and all this stuff. So that would be enough of the examples. Let's, are there any questions? So if anyone has something, just ask. From Andre Buda. Sorry, you have some question in the Q&A section. Yeah, Andre Buda is asking, can I safely pass credentials to test cases when using DMT upstream in a Ditha project? Credential test cases when using DMT upstream. So far, we haven't resolved any such use case. So probably cannot provide here any other guys, do you know any. So far we haven't used credentials for storing. I currently is not possible and it's in discussion if in the future we should support it and how it will be done because like we talked with Pekit, who is what is basically a GitHub application where you need to have a GitHub application or something where you store the secrets, right? And Pekit doesn't want to take the responsibility. So we are thinking about putting this into the other service which runs the tests, but definitely we didn't yet even plan on the future. So if you have these requirements, I guess we will need an issue to be filled, maybe in DMT, and I can take a look if there is already something, but I think it's some, it won't be implemented very soon, because the number of users. Can you share more details? What are you looking for? Do you want to, what do you want to credentials used for Andre? I will continue with it like the outline of our close future, what we see. So for the next steps, we would like to enable the context adjust feature in our pipelines. We want to do some cleanup in the documentation and to extend the guide, we started work on recently to add some new plugins like Christian executor, improved debugging and usability. Just as I mentioned, that would be integrated directly in the tool, made perhaps a wizard mode for those who are new coming and not want to learn options, but would offer some possibilities. And there's a lot of nice ideas shared in the DMT issues on GitHub, so there's a lot of stuff to do and a lot of bots, of course, to be fixed. So if you are interested to contribute, you can pick a good first issue from the list and start working on that, or if you have an idea or some you run into back, please submit it. You can also join the TMT channel or the telegram room to discuss the design, there are links in the slides, you can use them. And here we have links to some of the documentation. So the TMT and the reader.site, there's a TMT cheat sheet. So to get a brief overview of what's available, you can use the cheat sheet, it lists like it gives some overview of the functions which are there. And the guide, I mentioned the guide, so we started to write a guide which should help users to understand how to work with the tool, and we will be after this first section we will be providing more. I'll link to the Russia documentation and packet testing farm as well.