 Yeah, so however one will be present in the framework for integration test life cycle Which we built for our project and we think that it might be very useful to others so our plan today is to Firstly explain the problem which pushed us to create this framework then give a quick overview of tooling Which we are used to within this framework and provide details on the solution Introduction to the problem. So this is a project which we have it's a OMG team convertor L It is written in Python and it is application for converting centos or Oracle Linux to rail Yeah, I must explain a bit why this picture is here There was a guy from organizations of this conference and who was assigned to us to help To make the presentation looks better for auditory and he asked us to add pictures But I don't know which picture. So I will get them just random pictures. This is one of them So how to approach integration test within our team. So some typical technical needs It is complex manipulation with the testing environment operation system. So for example, we need to install Red Hat kernel or on Oracle Linux They need to reboot the system even multiple times during the testing Then we need to imitate the interaction with the user We need to adjust files like for example, some configs install various various RPM packages We need to access resources on Red Hat intranet and we have to somehow deal with the secrets And from an organizational perspective due to complexity of the previous integration test framework It was complex itself. The test language was different from the upstream. So not not Python project maintenance were unable to cover the features by themselves with integration test and How to organize a contribution process so new features won't be stuck waiting for verification As a result we built a so there was a separate key engineers team who was covered our features with tests and We had a batch-based testing framework, which we need to maintain and the strong separation of the duties. At least it was before irrelevant picture New solution tool in overview. So here we will so there is a three pillars of this framework. It's TMT testing farm and packet So now I'm giving the word to the author of TMT better split how take over better Okay. Hello. So as Adam said, my name is British quick how I'm from the OSCI team with focus on the improving processes and tools and one of The tools I've been working on recently is is TMT test management tool is an open source project and one of the main motivations for creating it was to create a very comfortable and Superuser friendly way how to how to manage tests how to work with them how to create them how to execute them How to debug test code so on the next slide? The tool itself is TMT consists of like of three three parts The first the first is the specification which defines the test metadata. So in another in another way, it's it's the way how The tests and the plans are described The config is quite concise and consistent across across different places The second part is the command line through itself, which which provides this functional before for users to to nicely work with tests and efficiently work with them and The next part is is the Python module which can be used from from Python scripts One of the important points we wanted to address With TMT is to have a unified way how to enable tests across Across multiple places so that users do not have to learn new and new configs and new and new ways how to enable test And how to work with them. So I'm glad that I can say that like this this way of Enabling tests currently works for github projects It's possible to enable tests in a rel fedora and sent to a stream and In the in the future possibly some some other places So now we wanted to give you some for those who are new to TMT just a very brief introduction So that you have some tangible tangible idea of how it works and how it looks like So for example here TMT init minus minus template mini. This is the way how how to get quickly started With the minimal template which creates a very easy plan which enables Some simple smoke test With calling with executing a simple script like like tmt minus minus out so That was the simple one if you have more tests more complex scenarios You would you would initialize your your git repository with with some basic setup and then you can use Commands like tmt test create minus minus template shell template the clip So here you see you can you can get quickly started with some initial skeleton Which is prepared for you and after you see the new files you will very very quickly know what to do On the next slide The core part of course of tmt is to be able to run tests TMt run is basically all you need when it coming to a new project You would like to Contribute there you do some change and then you would like to run tests so tmt run This should be everything you need to save for run all the test coverage, which is available there and of course there are some advanced features like If you can move a bit back our timer, I would like just just to mention You can you can do you can Execute tests in verbals mode to see more details. You can investigate the results Of the previous runs you can run until execute Which means like the provision preparation everything is done until the test is executing and then you can investigate There is some failure you can use last login to Login to the guest and do whatever you need On the next slide. I think one of the quite interesting Ideas or features of of tmt is we have dynamic plugins for individual steps and for the provision step we have multiple implementations currently we We support running tests in a container in a virtual machine connecting to an existing Box or directly on the local machine. So you can you can choose whatever you prefer And in this way safely or very quickly run run tests How you need And on the next slide. There is some Very brief mention About an example how to share test code across projects This is quite important thing So instead of duplicating code in multiple places one project can reference tests from another project and in this way you can for example enable very easily enable Integration testing of projects which which are closely related The next slide shows just a very very Very simple example of a tmt runner. So here you see the individual steps First we discover tests Then we provision a guest for running prepare The box so that it's ready for testing we execute tests when the report is done and the finishing step So there would be it about the brief overview and for all other information You can you can have a look at the tmt reader dot side And that's about it for me Thank you. Sorry for sometimes interrupting or jumping ahead. No problem. Yeah, so testing farms. This is a second pillar and Miro will describe So, hello everybody Miro Watkerti the testing farm team lead So testing farm is a testing system as a service It provides a reasonable http api to run tests against real life operating systems Our users use the service to get rid of the burden to maintain a stable infrastructure for test execution And rather invest the save resources Yeah into their product Testing farm currently supports tests defined via tmt as Introduced by petr in the previous slides and and tests are consumed from a remote git repository Which users define in the test request tmt itself as a tool does not limit what test framework actually you write your tests in so with tmt Shall executor actually you can integrate any of them. I think this is important to mention next slide pretty tartan So the current main users of testing farm include packet We will be mentioning packet on next slides and others are federa cei and rel ci currently Testing farm runs around 30 000 test requests per month as the current number and we are Planning to get a lot larger So for viewing results in a unified where they be Provide basically a simple ui to investigate the test Failures and errors are them will be showing that later in the slides testing farm next slide, please So testing farm is capable to run tests in public or inside redhead network in public We rely on aws and we provide intel and arm architectures Inside redhead network. We provide all rel supported architectures via beaker open stack and aws infrastructure providers The execution of the tests inside the redhead network Which is also used by convert rel in this presentation Makes it easy to qualify upstream code against rel nightlies or internal tests something which was fairly hard to do before That is important to highlight And next slide, please. I will be reporting also for packet because tomash is on pto so Lucky guy. So about packet Next slide, please So changes to open source projects are usually happening in the upstream repositories So when a new upstream release is being integrated into the downstream distribution It's already too late to provide feedback to some of the breaking changes The workflow packet is offering is to build your poor request changes and rpms And test them via testing farm on downstream distributions of your choice such as feather linux or cento stream This way, uh, you would know right away if a dependency is sane or Some breaking change was included. So test it as much as left as possible So packet next slide, please So packet provides a dashboard which contains a pipeline view and the overview of github poor requests Build targets for which rpms had been built and tested then via testing farm the test results are actually on the right side here in this picture Art and back to you Yeah, it's again irrelevant picture So now I will be describing this new solution which was using those technologies which guys are just described So how we wire all this together? So it's of course based on tmt tft testing farm and packet tools Uh, testing environment is declared using unsymbol, which is uh, pretty known and popular within developers So it's usually not a problem that developers could do this by themselves then test implement and using py test, which is industry standard for testing for tests and All the tests stored in the upstream and secrets are handled as environment variables stored in internet. It's Exactly relates to secrets So then uh developed tests locally uh and running them uh using the vms Submit test from local to be executed in testing farm. So it's if you'd like to test To run a lot of plans and you do not want to so For example, you can develop tests Locally and run some of the tests in testing farms So you could have some parallelism and better productivity then set of convenient features a testing feature which helps you to simplify a Development process of to solve complex problem in very efficient way And we need definitely a friendly ci system which executes all the tests and internet for each marriage request And as a result, so this was what we achieved and as a result contributors can write features and integration test because integration does use py test So it's also python very famous language. Few engineers can do the same They could write the features as well and they can do integration test and external contributors can do the same So they could cover their external features with the integration test This was a huge win for our team. We got unblocked So we start merging actually our features The team performance gradually boosted and quality of the application rise gradually as being one of the developer I could say that writing integration test for your feature give you so huge Experience and it improves the feature basically So let's take a look to the code So this is our converterl project upstream and plans are defined here Each plan team team plan Starting from like preparation and Ansible Playbook it runs ansible roles where It installs some testing dependencies install redhead compatible kernel for aquilinux So prepare the testing environment basically for running tests and In tests we have Test itself and I forget to mention that plants are actually they are kind of used to the test So you could with plants you could use special system of tags or Tires to select which tests to run So the test itself Stored here and for example, we could take a look. So this is a basic python language This convention convenient feature converterl, which is our utility there. We were providing command line arguments and dealing with some secrets And then we are expecting that once running in the test environment converterl with this Argument this text in logs will be here and exit code will be here So it's simple and then if we are doing something different then We expect in some different text and we could imitate the user behavior by sending counter c in this case It's just a stop execution of the utility faster. So do not wait until the very end and exit status Of course not zero so Yeah, basically, let's go to some interesting places. So this is how we are dealing with Environment files. So we could refer the environment from the intranet URL as you see In this case, we are able to inject some secrets from intranet then Here just example how we are dealing with Configuration, so we have again convenient feature where we are replacing something and it is being restored Then we expect again something and Yeah, this is more general feature which is changing another config So we are changing the release number and testing again something so Here is interesting in terms of we have conversion tests. So we are testing converted system after reboot So in the preparation step, we still could run the shell script that it is still pie test And in this case, it's it has advantage Because we have access to those convenient feature and this test is just basically run the conversion And then the other test will be running on converted system Okay, so We passed through these links going next. This is how the CI looks with the packet And this is how testing farm output looks like. So this is how plans are passed This is how it looks like if you're running the TMT plans locally So you could add more verbose information, of course This is how to list all the plans to For the given repo And this is basically it. So Slides are available on my github github github chukavgreen talks and you will see the defcon framework for integration tests This is it. Thank you chukavgreen current data engineer and apsa I love python scala and big data Nero, yeah be my guest guys present yourself Yeah, I think we already did Principal engineer when we were doing the our part, but yeah, I'm the testing farm team that is important and Petra already did his introduction Uh, there was a question actually in the chat about if we have some Examples for how TMT is used for federal changes builds Do we have some examples of some components which already onboarded TMT Petra? I link there the fedora ci quick start guide Which we have there And this is the main point if you want to use TMT for fedora ci For packet, there is another page. I will link it, but it's all linked in the presentation Under under my slides Yeah, I think more questions in the q&a section Uh, I'll read them out to you for the sake of the recording Um How is from chris evich how is downstream testing say from malicious upstream or public changes example in a pr Yeah, I can describe and answer this question if you don't mind guys So hi chris, uh, the point is that this integration test they're not triggered in the ci Just by the commit but only by comment from the upstream project maintainer In this case, we are interested, but yeah, there is still uh, you're right that like During the review we missed something then we will trigger and there is some risks So that is one part of the second part is that basically the uh, tests, uh, which we enable To run on downstream are Are not just not anybody can enable it or any red hat project. So we it's it's uh, it goes under our review So that is one thing also the test artifacts are not shared in uh upstream in that case They are only available to upstream downstream and also the code that you are able to reference is just red hat come Own domains, so the git repositories cannot be basically just some public Whatever github that you can just run something And we are working on some additional security measures because of course, uh, the this is a security concern and we are aware of it But currently this is how it looks like. So it's uh, there needs to be a special access that we enable It's not just anybody gets it. We are very cautious here Thanks morris 11 autumn Uh, sorry again for my pronunciation. I don't know the correct ones. Okay. Once from on dredge Hi, any reusable examples how tmt is currently used for testing fedora changes or builds I think we we answered that that was the first question. I was reading out. So I think that's done. Uh, and david. Yeah, could replace beaker in I ever read this. Sorry If I can yeah, please please care Could the empty replace beaker in dci for using cloud environments? I mean could could cloud environments like ec2 where beaker can't be used became Become there is the ecp and no pigs boot provisioning um, not tmt maybe by itself, but definitely like Like there is some overlap here and not sure if anybody knows like distributed ci so David, I think it's a wider topic to speak about but Definitely, there is an overlap we could speak about I'm not saying replace because beaker can provide a lot of different architectures Um, but yeah, I think it's a neat discussion. It's not just easy to answer if it could just replace it like that We didn't have that that conversation with anybody so Yes, and and for for connection connection with beaker. We also did some um intensive efforts too for those who are Trying to migrate their tests from the original formats at the old make file format to the new one And the integration to make it make it working the integration with beaker. So, uh, it's possible Using two ways like the workflow tomorrow, which is used for scheduling scheduling tests directly in beaker is possible to be used directly with tcms and for those teams, which have migrated tests to tmt But they don't want to use tcms anymore. It's it's possible also to export FMF IDs of the tests and then Feed them to the workflow tomorrow and run tests in beaker. So yeah, there are multiple ways how to how to connect connect the pieces together Thanks, Marius. I hope that answers your question, David Any other questions? Please feel free to drop them in the q&a section or in the chat itself We are right on time great We are two minutes early because we started late. So very quick very quick Good work guys Yes So then I think we should Yeah, uh, please better carry on. Sorry for interrupting. Yeah, I just want to thanks. Thanks all to uh, who who attended today's session And see you somewhere around Thank you for example. Bye. See you guys. Thank you. Thank you