 So, hello all. My name is Peter Schindler. I am from Czech Republic, from the RedHead's desktop QE team. And I'd like to show you how we test desktop applications, how we test GNOME, and how we use Doctel and Behave for that purpose. So, desktop QE. It's quite a small internet team. We do some automation, also some manual testing. It's not long time ago when we decided to start with upstream R test, and it's like my work. And I'd like to continue with what I just begun. You can look in the Peugeot. There is a group desktop QE, but right now there is just one test for GNOME calculator. And I hope that other will show up soon. And it would be great if there would be some community and there will be tests, or if there will be someone who would like to write some tests, that would be the best. We focus on testing of applications on different desktop environments like GNOME and KDE, but mainly GNOME. We also test Office applications like LibreOffice and few others. Then we test some other components which are desktop related, like some console apps, for example, BU and libraries. For example, it's like GNOME Curing. And also we test hardware which probably won't be upstreamed, because I don't think that Red Hat will give community a lot of hardware just like that. But community can help us with testing on their own hardware, so that's where the strength of the community, a lot of hardware. So, does someone know behave or met him before? If not, you can find some documentation on this link. I'm not sure if you can read it properly, it's quite dark. But if you will look for behave and Python on the Google, it will show you probably this page. And there you can find a lot of information about what is it, how it could be used, what's the philosophy behind it. As it is written there, its purpose is behavior-driven development, which means that you define what you want your program to work like. Like you click some button and you expect to show hello world text somewhere. And then you will program tests for it, which will test if it really happens. And then you will write a code and something like that. We don't use it like that. We just want to use behave for writing out tests for those applications which already exist. And developers probably doesn't care about behavior-driven development. So, we just use it for creating test cases or test plans and write them in behave. The good thing about it and the advantage of behave is that you can read the code and understand what's happening there. So, I will show you some tests and it looks like this. So, you can see that there is some feature. This is test case for GNOME calculator you can find on the Pagina. You can probably read it, right? I'm not sure if this can like, I will open it and get it, wait a moment. So, you can see that there is something like feature advanced mode, which is GNOME calculators mode. So, we have some feature and we have some scenarios we want to test. That could be a particular test case and that's the test plan, maybe. In every scenario, which is one test case and one run, you can define some steps. For example, this is not a good example because it's a lot of test cases in one. For example, this one or this one, it has name which is useful when you run a lot of tests. You can see which one is this one in the output, which I will show later. And then you have some steps which you want to do and you want to check some output. So, you can see that calculate ton P divided by 2, it's like you can, you know what it's wanted here. And then there is a step called then, which assume that something happened and you just checked if what happens is right and you just are looking for some error. And that's it. It's like you would say to someone how he should test this feature. It has some advantages like when you have to put it through deeper testing and use more values. You can write something like scenario outline and use variables and write the table. And it will run for each line in the table, it will run all those tests. So, it will click different button, calculate different value and expects the different result. So, you don't have to write this like 10 times, just once. So, that's how it look like. Also, you can have something like background, which means that when you run scenario you want to do some preparation, do some values. So, right here you want to set number format to be fixed. And so, you don't have to write it to every test case. You just write it here that you want to set the state of the application to this one and then run the test case or the scenario as it is called in the behave. So, another advantage of that is that you have reusable steps. So, if you use the same steps, you will program it just once and you can use it in more test cases or scenarios. And also, another advantage is it has nice outputs, which I have open right here. I'm not sure if it's big enough to, so you could see. Can you read it? At least somehow. Yeah, that's when I run the whole feature. I told behave to create the HTML output and it created something like this. It says which scenario was run, its first line in that table and those particular steps. And if it's green, then it's okay. If it's not, then it's broken, like, and it's in red. So, you can check what happened. So, you can see, okay, this should be P. That's the only problem with this HTML output that some unicode signs are shown wrongly like this. But if you know the test case or the scenario, you can guess what it should be. And you can see that it tried to somehow compute the SINUs and the result was wrong. But you can also look what was the error message. Just, it says that the result was different than expected. You can also look on the screenshot. It really was different than expected. Yeah, it's, this is the error in the test case itself. So, there should be just one P. It was, I think, typo because it happened more times. So, but that's something you have to fix. If you find this, if you find something isn't working properly, then it could be bug in the feature or in the application you are testing, or it could, and probably is, bug in your test, which this one was. And you can also see the video. In calculator, it doesn't make sense, but there are a lot. But there are some features, which it's very, it's really helpful to see what happened. Like, I'm not sure it's not here, but for example, it was trying to click on currency button and it didn't do nothing, so you can see here that it really didn't do nothing. So, and if you, so you can see what happened here. You can do this with behaves, which is quite nice and helps to solve the bugs. Excuse me, screenshot, when the moment of the failure. Yeah, it takes screenshots whenever, I think, whenever the step begins and ends or something like that. And if something happens, then it will propagate it to the HTML, I think. No, passwords won't be, if you will click on button to show password, then yes. But if you are writing passwords in those tests, no, you can program it to show screenshots and videos also in past. It's just written like this because when you do a lot of testing, there would be a lot of videos and HTML would be really big. So it's saving space to, yeah. But you can program it in, yeah, I will show you where it is. I'm not sure if I will find directly the code of doing the screenshots, but I will show you where you can look for it. And it can be programmed to create a different output. So you can, for example, make a screenshot when, in every step or when it's passing or something like that. So that was to behave. I did that. So, and Doctail. Doctail is Python library and utilities, which are used for automation. And it's paid on Etspy, which is toolkit, which gives you the ability to click or to call, like, to do user input, like clicking, writing text. And also to gather information, like position of some parts of the screen or of some parts of the application, like you want to know where the button you want to click on is. And that's done with Doctail. So when you want to write the test, you will write the behave test case, and you then have to program the particular steps. And it's done with Doctail. It gives you some ability to execute those things, to search the UI tree of the application for those buttons and text inputs areas and so on. Sadly, right now it doesn't work with Wayland, and I'm not sure and nobody is if it will ever work with Wayland, because you can find UI tree and search it. You can find where the button is in the application, but not on the screen hole. So that's one problem, and second problem is you can do the user actions, like typing and clicking, which is quite bad for this type of testing. So right now we have to use Xorg still, while it is still present in Fedora. So yeah, that's the problem. You have to set custom.conf of gdm to don't use Wayland in the session, and then it works. So you can still test the functionality of the application, but you won't be probably able to catch all the crashes due to wrong, due to matter problems or Wayland or whatever. So I thought that the Wayland developers are still working on, does it work in Wayland? They can work on it because it depends on the accessibility, and accessibility doesn't work properly on Wayland. And doctail won't be able to work properly on Wayland until the accessibility will work, and it will probably never happen. Because it's quite difficult to change the way Wayland works, because the security reasons it doesn't work, for it doesn't work in case it's directly and deeply written in Wayland. So it will be really hard to change it somehow, and it would take a lot of work, probably, and it would probably create some security errors or something. So maybe it's not a good idea to solve it. But we'll see, we are trying to make developers to change it and to make it somehow run, at least for example in some testing mode of matter or something. So how to define the steps? Behave with decorators, which will match the steps. I will show you the code, and I will have to open it here. So, for example, the calculate may be step. So you have the decorator step, which will say that the step looks like a calculate and some variable, which is defined here. And you define the function, which gets context variable, which behave provides the context variable, where there are various information. For example, what feature is run, what step is run, what scenario. You can run with context, you can run another step from the code. So you can just, if you have step, which do something you need to do in this step, you can reuse that step and run it from this. And it also can be filled with information you need for your testing. So, for example, we use it for saving the instance of application, like what we find in the UI tree. We can save it to the context and it's visible in all steps in this context. And also, you have to have variable, which has the same name as in the decorator. And you will get what behave has in its test case there. So, you then do some typing and the magic you want to do with on the session, like click buttons. And everything you need. And you use doctail for that. And it's written in Python, but I forgot to mention that it's all in Python. So, for example here, you can see this instance is instance of the application, which is tested right here. So, it's going to calculator. And you want to find in UI tree, the children of the whole application, which have some properties. Like this child has name result and is showing right now. So, it will find the UI, for example, button, which is named result. And it will returns it. And then you can click on it. For playing with this, the idle thing is to play with I Python. So, I will, if we have time, we have a lot of time. So, I will show you how to play with those tests or with doctail later. And also, if you define the then statements or then steps, you can, for example, use assert to check if those results are what you expect. For example, here, it was the one I showed you in the example of behave. It expects some error message. So, you will check if that message is what you expect. You will get the actual message. You will find the right child, which is sometimes quite, it's not straightforward, because sometimes those items aren't directly named or something. They are just somewhere in the tree. So, we will have to find some item which is near in the tree. And from the parent, for example, go to another branch of the tree. But, yeah, I will show you how to find out where you can find it when this is needed. Sometimes, sometimes, because developers mostly don't write their UI to be really tested like this. But, mostly, you want some items which have names like buttons or have some text in it. So, it can be found quite easily. Also, there is one file which is quite important, which is named environment.py, where are defined special steps, I would say, or special functions. This is before all, before scenario, after a step, and so on. And it says that, for example, before scenario is run, whenever is before the scenario itself is run. So, for example, you want to run some scenario for testing if it calculates sinus of p correctly. So, you run the scenario, but just before that, you want to set up the context to have the right application. Here, as I said, you can save some useful information to context. It could be done, for example, here. We just create the application instance, which will show to the tree, which will give the tree of the GNOME calculator, the UI tree of the GNOME calculator. You can also put there more parameters like record video, and a lot of others. For example, if the desktop file is named unusually, then you can put it here. So, the steps which need to have the desktop file and use them have the right desktop file or right name. So, you can do a lot of stuff here. We have some common steps, which are commonly used between all the projects we have. Right now, there is only one project which is upstream, but we have quite a lot of tests for a lot of applications. So, for example, steps like run this application is used in every test, almost every test. So, it does have sense to have it on some one place, so every project doesn't have to have this in its own project, so it makes shorter code. So, let's look on the UI tree. I'm not sure if on this screen it will be visible, but hopefully yes. I will run it on my session. So, there is application which comes with XP. It's called SNF, and it shows the UI trees of running applications. For example, if I run calculator, I will hopefully find the calculator here. And there is UI tree. This is what you have in the context.app. You have this tree, so it's the same as you can see in the Python doc. And it has quite nice feature which is highlight items. I'm not sure if it's visible. It's highlight items. And it do the thing you would guess. If you click on some item, it will highlight it. If it's visible, then it will highlight directly where it is. If it's not visible, it just blinks somewhere else. So, if I click on this item, I know that it's the main part of the calculator. So, I can open it and investigate further. I want some buttons. So, let me see. Okay, buttons here. You can see that those trees aren't quite simple. But still, the main thing you need is, for example, to find a particular button. So, yeah, I found a button for factorial, and it's called factorial. Great. So, now, when I want to click on the but factorial, I know that I'm looking for children which have the name property set to factorial. If I want to specify further, because there is written factorial somewhere else or there are two items with name factorial or something, I can specify that roll name is push button. And I can do this stuff with a lot of things, with everything which is here. So, I can investigate like this what I'm looking for, and then I can use it in Python. Ideally, I would now not sure it won't probably work right now. That's another session. So, you can, can you see it? So, there is one main thing you want to import from Doctail. It's from Doctail 3, it's root, which is really the root of this tree. And then, if you want some particular application, you just write root application and the name of the application. Okay, and then we have a root of the application GNOME calculator, so we can play with it. You can find, for example, the, hopefully, we'll find it properly. So, I found just find the button I wanted to, I was looking for before. And if I, if I just try to click on it, you'll see what it will do, it will do, and it clicked on it, so magic. And I didn't know that I have XORG, I thought that I have well, and so, yeah. That's a good way to realize that you have XORG session and not well, because I totally forgot to change it. So, yeah, you can play with iPython and Doctail like this. So, this is how I mostly write tests, or mostly fix tests, because from time to time, developers decide that it's time to change the tree, and then your tests are quite screwed, and you have to rewrite it, which doesn't happen a lot, a lot, and that's the main advantage of this way to test UI applications. So, yeah, but for example, it happens to calculator quite in some time, so every button was renamed, but fortunately it was renamed to what it's written on those buttons, because for example, button lock, I don't know, button syn wasn't, was named synos, like the whole name of the function, and now its name is just SIN, like what's written on the button, so you don't have to guess it, you just write, for example, capital N, and you know that it will find the proper button, but you have to change it, if it happens. Sometimes also, when developers want to try new stuff, like UI GUI fancy stuff, like buttons, like something like this, or you have to then change everything, or this is a really great way to store stuff in variables. So from time to time, you have to rewrite the tests also, so you have to go to the SNF or play with iPython and find the proper way to do what you want to do, and to write the step. So, you can also, I highlight in iPython, it's function blink, but I won't do it in my session because the rectangle will stay on the screen, indefinitely probably, or I'm not sure how to remove it, so I won't do it because I don't like random rectangles on my screen. And that's really useful, because it shows you that you have the right item you are looking for. So, if you want to play on the virtual machine, which mostly you won't, because, for example, you don't have the right version of Fedora you want to write tests for, like you want to write tests for Fedora 28, you already have raw height as it should be, then you can install just to the virtual machine, use SSH to connect to it, and from the root session, you can use this command to run a user session and to be connected to it, and so you can use the Python or iPython stuff even in SSH session, which is really useful, because, for example, you can play with it, you don't have to care about rectangles on your screen and everything, and when you end the session, it will just end the session. So, yeah, it's good for playing with it, and you can do everything there and don't have to care about your own session. So, two main functions I use in iPython or two things I import is root for getting the root of application, and some raw input like Preskey, which is function which just Preskey down or up or enter, so it's useful, and also there is a function click, and so you can look in the doctail. It's quite nice to look on the code. So, as I said, you can run it locally as I did right now, you can run it on VM and also in Permetal and use SSH to connect to it. You can play with doctail with iPython or you can just run test with behave. In the GNOME calculator, you can find there, for example, if you will download the code from the peg here or how is it pronounced, then you will find there the run.sh, which looks like this, which will, when you run it, it will download the submodel, which is for downloading the command steps from another Git repo. So, it will include all those command steps into your project. It will remove all the videos, because if you record the sessions, this will get full of those. And then it run the behave again in test session. It's done by this function. This doctail run headless next create session. When you run it with sudo and some user, it will run it as this user, so it will be user session. So it will look like your normal session you are using as user. And you can then put there some function you want to run. When I was showing you this before, there was a bash, which is the way what you want to run when you use SSH, because you want to play with it. But if you run directly behave, it will run tests, those behave tests. It's done like this, behave minus dash t. It says that you want to run a test, which is with, and there is parameter. If I look in the test, all those scenarios have defined tags. So if you want to just run this scenario, you will put there dash t advanced calculation, and it will run this particular scenario. You can also run whole feature, because the feature was tagged with that. And you can also put the same tag to more scenarios. So you can use tags to run different, for example, if you would have tests for different tiers, then you could run it like this. You could tag it with tier one tag and run all those tier one tags, tier one tests. Okay, what's the time? Okay, I should end in five minutes, I guess. Okay, that's right, because I'm on the end. Right now, I'm trying to write some OSCI. I probably won't get it to it right now. It's a good way to run those tests if you don't want to set anything, because normally you have to install doctail, behave, and everything set up. But with that, it could be used on the cloud base image, which you can download. You have to enlarge the partition, because it's too small. By default, four gigs, so you have to make it at least eight to run those tests. And then you will just use standard test interface to run the machine for you with that image. And I have written the OSCI like wrapper for it and YAML. I would have to find it in here. Upstream first. Okay, that's the next stuff we want to do when we upstream the tests to run it in OSCI. But it's under go, because there are some problems with desktop application being run there. But it's like running this YAML with Uncivil Playbook, with test subjects set on the QCOW image you want to run it on. And it will do everything if you have the right OSCI image. YAML set. And this one just set the image to have everything installed, like workstation to have test user for user session. It will install, recreate packages, download git, and run tests. And that's everything you need to do. Normally I run it like I run virtual machine, then SSH to that virtual machine, download from git or rsync from my local computer or my host. I will copy the tests and run it with run.sh and see results. So that's how I run it. And you can try it too. So that was if I had some slides if I would like run through it in 10 minutes and it didn't happen. So it's Ferrara QA. So if you are interested in testing, you can also look on other stuff like Open QA and just ask guys in Ferrara QA team, how can you help with testing Ferrara? And if you have some questions and don't want to talk to me directly, because I don't know, I smell for example or something like that, in that weather it's not... It's not... Yeah, it happens. So you can reach me on the mail, or if you are in Brno, you can come to my office or just find me on Fedora IRC, like Pashindle or... Yeah, that's it probably. Just find me and ask me to whatever you want. If you have questions now, I can probably give you some time in order of seconds to answer. But if you're not, I thank you for your attention and have a nice day and evening playing games or what's... I'm not really sure. I'm really sure. I haven't done stuff like that because I'm quite shortly in the team, but I know that the graphic cards are tested on hardware. Not sure how. There are probably tests running behave, which test if everything looks okay. Wi-Fi is tested probably with Network Manager, which was one of nice stuff, because the developer of Network Manager tests just upstreamed his tests and used OSCI already, so you can look on Network Manager tests there. It's on GitHub Network Manager, Network Manager CI. I think that they just have a machine stuffed with a lot of Wi-Fi and Network cards and run tests on them. I think that this is what's run there. I'm not sure with graphic cards, but maybe we do manual testing also. I think that it's part of manual testing also. Probably there are some automated tests, but I'm not sure. But if you will ask me when I will be in office, I can direct you to the right person who will answer your questions happily. I will make him happily answer questions. Another question? You can make sure that things are rendered correctly, like if the button is in some other row or something, or it's wrong. That's what I'm not sure about. You can do it with OpenQA, but we don't use OpenQA. But the cocktail doesn't do that. You can write a step to do it. One member of our team written OpenCV tests. So you can give it a picture of how it should look like and it would make a screenshot or something. But still, it's a step for behavior. Written not with a cocktail, this one, just in Python. That's what OpenQA is. It just compares pictures. So you can make it work like this with behave. You just write the correct step, which says check if the button is on the right place or it looks like it should. But I'm not sure if it will discover some hardware issues. I'm really not sure how this is done. Because my work is in testing of the application, not the hardware. So I so far didn't care about the hardware tests. Doctel doesn't work on real-end. So are you planning to switch to some other framework or waiting for the developer to work? I can say that the developer is not that much active. Excuse me, I'm probably deaf. Doctel doesn't work on real-end. Are you planning to switch to some other framework or are you waiting for the developer to fix the issue? That's to be decided, probably. And it's on our manager, not on us. Because we also plan to use the Doctel. Yeah, it's a real problem. It's a real problem. It's a blocker for us. And we are working on making someone to solve it. But if it won't happen, it will be a real problem to run it on XOR because, for example, XOR will disappear. Then, yeah, it will be a bit... What kind of Doctel does it work on Python 3? I think that there is already written Python 3 Doctel. And behave not sure. I haven't tried, yes. And it should be rewritten. But I think that Doctel is written for Python 3. I think that I use it already. But the behave probably isn't right now. And that's something I'm not sure. And because of changing the default of Python or in 29, yeah, that would be the issue probably too. But Python behave is someone else's package, probably. So, yeah, that's a good question for us to ask the developer. So, the weather is getting better. Yeah, some storm is coming. Okay, so, any other questions? So, thank you for everything.