 Hello, good afternoon. My name is Lukas and I would like to present a short talk on how we test graphical user interface with OpenQA in Fedora, but this talk is not going to be an informative talk only. It should be an evangelizing talk and maybe it could inspire you to start using OpenQA if sometimes you might look for something that can help you testing stuff in GUI. OpenQA is an automated test tool. It's mainly developed by SUSE but we also use it in Fedora and my colleague Adam Williamson helps developing it too and OpenQA is fully integrated into Fedora. It's packaged for Fedora, so if you run Fedora you can just install it in 15 minutes using the RPM packages. However, this talk is not going to be about installing OpenQA or setting it up. I talked about it last year on Fedora Hatch and I'll probably talk about it again someday in the future doing a workshop, but this is just aimed on graphical user interface testing. OpenQA is good because it allows you to test various operating systems, so when I talk about Fedora you could replace Fedora by anything. It could be rail, it could be open SUSE, it could be Windows, it's up to you what you put inside the virtual machine because OpenQA runs a virtual machine and performs the tests on the content of that virtual machine. Good thing also is that you don't have to install anything into the tested operating system, so you can just take the basic operating system, you don't need to add anything to it, you run it in a virtual machine and you can work testing it. So it basically creates and runs a virtual machine. This can be based on an ISO file or a QCOW2 file and you can perform various actions inside that virtual machine and evaluate the outcome. The architecture is that there is an OpenQA application with a database that's sort of a scheduler. You communicate with it using the browser or a REST API and then it has a worker that performs all the jobs that it downloads from OS auto-inst. It has the tests, it runs VNC and it uses Squemu to operate the virtual machine. The controller does the visualization, job handling, live viewing results and it stores the results in the database and the worker tests, runs the tests and checks the needles and all the other things that are needed to work with the virtual machines. You can have the worker and controller can be on one computer, so you can run it on a laptop for example, but it also can be split and you can run a scheduler or the controller on one computer, on one machine and you can run workers on other machines, so you can have a bunch of workers and do a multiple tests at the same time. Of course, you can do it on one machine too if you have enough RAM. The test itself is a Pearl script, although it should be possible to write those tests in Python too. I have never tried because the test language, the test Pearl is pretty easy, so it's just the basic Pearl commands and some other commands from the OpenQA and the test defines what you do inside of that virtual machine. It mainly defines mouse and keyboard actions and it checks and evaluates the needles. We are going to talk about the needles a little bit later, so you will know what that is and it evaluates the expected, what you say you expect in the test script, so then you can compare it to what you get in the virtual machine. Of course, the test or the job can end with various statuses such as past, failed, soft failed, canceled and so on and so on. The tests are Pearl modules and they are placed in the test directory. That's probably not that important, but if you see the OpenQA instance, so there is a test directory with the test scripts and there is also a lib directory where the libraries with commands that you can use and the routines we are going to talk about now are documented in the test API documentation which is Open.QA API, test API, so you can check it there because I am not going to talk about it on a deep level. I will just show you what you can do and what routines you can use to test the graphical user interface actually. Each test has a header where you define what libraries or what files, what packages it will use. You probably know if you know Pearl, so you know the use strict and maybe use warnings. These are useful things to do because if you omitted, then Pearl will not tell you about your own mistakes, so troubleshooting a test script is rather complicated then. We say that we use the install test and we use the test API library and the utils library in this header. In Fedora, these are mostly, the header is mostly standardised and this is, I would say, a typical header for the majority of our graphical user interface tests. Then it should have a subroutine called run where all commands must go. You can also use other subroutines in the test scripts, but then these are only visible in the scope of that module and not outside of it, so you can prepare some routines of your own. You can use it then in the run subroutine, but outside of the module only the subroutine run will be visible. And then it ends with the test flags. The test flags specify what to do when your test finishes and by default all are off, so you can switch it on like this. For example, always rollback means that after the test finishes it will return back to a saved status, because you can save the virtual machine on some point, then you can perform various activities and after these activities are finished you come back to the original state, which is a great thing to do when you want to start from the same point all the time. For example, if you have a graphical application and you start using your mouse and clicking on widgets, so you are moving the state of that application and then for example, you could start the next test, you could start at a certain point. For example, you open the help window in one test because you want to test that the help window can be opened, so you open it and then you do another test that uses another widget, but the help window is still opened, so you can design it like yeah, you can do it like by design, I want to continue when the help window is opened, but then sometimes the test fails and the help window will not be opened and it breaks the consecutive tests also, so therefore it's good to start from a certain point and you can save it and then you can always roll back to it. Other possible flags are fatal, which means that if the test fails then everything fails or milestone, which means that now after the test it's the point where you want to save the status of the virtual machine and no rollback if you don't want rollback or ignore failure if failures should be ignored and then each Perl module must return a true return value, so right at the end you have to place one. This was when I started with OpenQA I didn't know that and I often would omit the one and then I was getting errors all the time and I couldn't run it and I didn't know why and then I realized there must be the one. Okay, so now you want to test a graphical user interface which basically is you want to see something, you want to see some widgets and you want to interact with those widgets using your mouse or using your keyboard. So OpenQA lets you do the following actions for the mouse control. You can set the mouse to a position, mouse set. This is good if you know to which position you want to set the mouse. Normally you would define the resolution of the screen in OpenQA, the resolution of the virtual machine and you maybe know that the position of the widget is on 500, 600, so you could move the mouse to that position directly. Most of the time this is not the case and I will show you how to fix it. You can hide the mouse cursor, move it out of the screen because sometimes you don't want that cursor to affect the process of the test. You can click on the mouse, you can double click on the mouse, you can triple click on the mouse. Each of those commands you can select the button to click if it's the left, the middle or the right button. How long it should be clicked, but I don't want to go that deep right now. And you could also do a mouse drag, which means you start on some point you hold the mouse button and you drag to another position. Mouse scrolls are currently not supported. I visited the OpenSUSE booth this morning and there were some nice guys doing OpenQA also and I had a nice chat with them and I asked about a problem that I have with a certain needle type and then Defolos told me. And yes, but this is not such a big problem, but if somebody wanted to create a patch it would be mouse scroll. So they also want mouse scrolls. We want mouse scrolls, but nobody has implemented it yet. And there is a workaround we are using in Fedora tests and it's a keyboard workaround. So key events that you can use is you can send a key which basically means press a key or press a combination of keys. Press and hold a key is a good point or good stuff when you want to interact with the mouse. So you can press and hold the key and press the mouse button and the release both of them. That might be good for some sophisticated applications. Release a key of course is when you're pressing and holding it so releasing a key releases the key. Send key until needle match basically is press the key repetitively until you find what you expect which I am using when I need to scroll the mouse. So I am sending like a down arrow or a tab or something and then the GUI scrolls. You can type a string in Fedora we have wrappers type safely and type very safely because you can specify in type string you can specify how fast it should type and so on and because we don't want to repeat it all the time. So type very safely types really slowly because when you just type a string sometimes we were getting errors like it typed so quickly that it missed a letter for example or it pressed the letter so quickly that it produced three consecutive letters like instead of password it produced PAAA as W-O-O-R-D-D-D-D-D so this is not what you want. So therefore we are typing very slowly with the type very safely command. Our type password is similar but it doesn't get logged in the log files so that nobody should know about it and enter a command is basically again type a string but this time it adds a enter press at the end of the command so you can have this one to run commands. And now we talk about needles that are a crucial part of the GUI testing because how the test machine should know about what you want to see and it's because you can compare a pre-created image with the content of the virtual machine. So you have a screenshot with an area defined that you expect to be found and then OpenQA compares it with the content of the virtual machine and if you get a match it's good if you don't it's not that good unless you want it. So you have various like assert screen means that you want to check that something is there maybe a widget is present or an icon is visible on the screen. Check screen is similar but it doesn't fail if if nothing is found and instead it gives you it gives you an undefined value so you can use some kind of diversions in your code like if you find this do that if you don't do something else. Assert screen and click is that if it's found it clicks on it. Assert screen and dclick is if it's found then just make a double click. Then for example what we also use is click on last match for example if you have a check screen maybe so then it finds a match and you don't have to write another assert so you just use click on last match and it will pick up that last match and click on it. So you can say if check screen something click on last match and if not do something else and sometimes also it's very important to wait until a screen changes or to wait until the screen stays still because sometimes open QA is pretty fast it's much faster than a user is so we had problems with KDE for example because KDE used sort of how do you say that? Yeah the words slipped my mind. Okay sort of animations yes that's the word and those animations took some time but open QA was so fast that it clicked somewhere even before the animation was ended and the test failed so you say wait until the screen stays still and then it can do animations as it will and the test waits for it and then finds the needle it should. There are some more you can record a soft failure you can record informations and you can save a screenshot however it didn't work for me to save a screenshot I don't know why I must figure it out but it should be there. You could also provide variables either in the test setting or when you start those tests you can say you can define a variable and then you can use the variable by get variable or set variable or check variable so for example you can say that in the test settings that you can say that the tag for example is gnome and then you can have one test script that could operate on gnome and KDE and it knows that the variable is set to gnome and it will only do the gnome part on that one particular run and if you replace the variable when you start the script then it will behave the opposite way and it will also only run the KDE parts so what is the needle how it looks like so basically this is a screen with a graphical application and this is the calculator the gnome calculator and I took the screenshot from a real test real life test so this is what we expect a user might see but because it's so complicated when we compare the entire screen so it's there are so many problems maybe like these little tiny dots you know we don't want to compare the entire screenshot but we want to just compare if there is the button with the number one so from this screenshot we define an area that the open qa should try to find which is the button one and now it only looks for this particular part theoretically this particular part could be found on any position of the screenshot so it doesn't have to be just here and if for example the graphical application is moved to the right part of the screen it will still match when we think about the needle so it consists of two files there is the PNG screenshot and adjacent description file a needle must have at least one area if it should be a clickable needle then it should just have one area and the click point lies in the middle of that area the needle also needs a tag because it looks for them according to the tag and you can have more needles with the same tag the bigger the area the more risk of mismatch and you can also set the needle fuzziness that can be adjusted from zero to 100 where with zero it would be probably everything is a match and with 100 percent is like a totally exact match is a match so by default it's set to 96 and we sometimes go to 90 this is the jason file it looks like this that's the tag to find the the needle and you know that there is the expose ipos at it's the coordinates where to start then width and height are the width and height of the area and the type is match that means it can find it must find a graphical match and the match fuzziness is 90 okay need for needles yeah this is basically that sometimes you need to make more needles to cover for one tag because fonts differ colors might differ and stuff might differ so for example for if you want to check the buttons on that calculator you would need 24 needles at least the example the test example how you would click on those buttons and you would solve the three times bracket three plus two bracket closed equals sequence you could you could solve with these commands so it's basically a certain click a certain click a certain click a certain click a certain click and then compare the result and that's it and then delete it with escape for example and you see that this would require eight needles to do just this so we have some more time so i would like to show you the real test this is a real calculator test uh that does various clicking and tries to do various various you know examples or equations it also shows the help it also shows the about window if you take a look on the code it's slightly more complicated but because we were trying to use a sub routine that does the calculation so that we don't have to repeat all the needles all the time and the last thing i would like to show you is the video i hope i can can how can i make it slow speed yes so half the speed and so now it logs into the workstation it starts the graphical application called calculator and it uh shows the about now it clicks and solves the equations so this is so designed that to solve those equations it must click on all of the buttons so we know that all of the buttons work and we also know that the help worked so this is a very short graphical test of GNOME calculator and this is everything i wanted to show you today it's so easy you can start right away i measured it the installation and setting up the open qa takes 15 minutes 30 minutes when you don't know okay do you have questions uh the the good thing is uh can i repeat the question how uh do we test GNOME with Wayland or KDE with Wayland uh the good point about open qa is that it doesn't care about Wayland at all uh so it only compares the the screens that the virtual machine is giving so if it runs Wayland or Zork it doesn't really uh it doesn't really matter yeah so uh you can test Wayland just fine yeah pretty you can test pretty much everything what can be run in a virtual machine uh pardon uh yeah uh you cannot do it i don't because it's not implemented so there is no way how you could uh how you could tell the the open qa now scroll the button yeah when i uh when i talked with the open suze guys they told me that probably there is a routine which communicates to vnc and sell and sends scrolls but uh yeah so you can't use it right now but maybe we find some time and way how to do it and uh yeah yes yeah uh if you want to uh you can contact me anytime and uh i could help you and show you how to do it uh yeah of course uh we have a repo uh where we store the the tests this os auto in this fedora it's on pegger and uh basically you see that uh there are the tests here so uh in this we have applications and each graphical application has a a dedicated directory where you put all the tests uh you want to run uh i think that it's useful to split the entire application into various tiny steps and start from the beginning again always rollback always rollback because then you know these tests passed this test has failed so let's just uh investigate more this one if this is just one entire script and something fails in the beginning then the rest doesn't get tested never yes of course no problem with that yes but because we don't want to okay uh do we have coordinates only to find the place to click right uh yes and no basically uh the engine operates on coordinates but uh i want yeah i would like to show this once more uh so i could expect that the digit number one would be on maybe 800 200 or something which maybe or maybe not the case so therefore there is the needle system that you say that you want to look for this particular area so a little bit gray with one on it and open qa compares it with what it has in the virtual machine and it will calculate the coordinates automatically and it will click in the middle of that area so it does uh it calculates the coordinates for you but if you want to specifically say it needs to be coordinate x and coordinate y then you use the mouse set and it moves the mouse to the exact coordinates okay enough uh yeah time's up so thank you very much for your attention and have a great defcon