 Hello everyone, I'm Katarzyna, I work for Red Hat and I work in QE department. I'm not a software developer, I'm a QE person and I will talk how we test our system and it's not about every Red Hat program, it's only about my team. So it's only about Red Hat Enterprise Virtualization Manager, so this is downstream version of Ovid. So this is tool which helps you to manage your virtualization. So on which level we are, you mentioned that you are a developer, you develop your application, it passed all your unit tests, it passed all your functional tests, it was already built, so you have RPM or in our case it's RPM, maybe you have another packages, you want to distribute to user, but before you do that, you want to test if everything is okay, so you want to test how it will work on real system, in real cases and so on. So you give it to us and what we do, we develop your application on real environment, we try to do it on as production system as possible. So for example, this is system for virtualization, we will run what we can on bare metal machines, not on virtual machines. We will use official APIs, we will not take a look on your code at all, almost at all. We will simulate user, more or less we will do what your user will do and we will test both negative and positive cases, but we will do much more positive, it's like okay, there is a problem if something is broken, but it's much bigger problem if you cannot do some case. So what are problems in such situation? First of all, we touch environment, actually we break this environment and we do it on purpose. We want to test how the system will behave if we break network connection, if we break a disk, can it recover? So the problem is you have to maintain this environment and it's hard. What else? It takes time, you have to set up this environment, it takes time, it takes, it's not a fraction of seconds, it's minutes, sometimes it's half an hour. Then you have to run your test, it's quite short usually, and then you have to clean this environment and because this test may fail at any moment, you have to have this clean up quite well, you don't want to reinstall everything starting from the operating system every time, okay, and of course we want to automate it, we may do it manually, but imagine that you will have to do it every week, the same, install everything, click on it, click on everything, so why we want to use Python? First of all, it's easy. Quite a lot of people in QE departments are not software engineering people, they have some background in IT, quite often they are from admin departments, the only language they know is Bash, and you have to somehow force them to write code, you don't want them to, you don't want to teach them Java or C++ or anything like that, Python is easy, they can understand it. On the other hand, it's normal, match your language, you have libraries for almost everything, and it's popular, so if you need people who really know this language, for example, you want people who will write tests and you want people who will write libraries and so on, and for these people, you want someone with coding experience, and finding someone who knows Python is not that hard. So what one such test does, it must get resources, you don't want to run your test on the same machine on which your system will be developed, because quite often you will want to, for example, reboot this machine and test what will happen, so you have to get a machine on which you will run the test code, you have to get this definition, you usually keep it in a repository like Git, and you need resources, resources like Bermeter hosts or virtual machines or some kind of storages, then you have to install product packages, you have to take them from somewhere, so you have to specify from which repository you want to take, then you have to configure this product, and your test may want to change this configuration, then there's what's specific for the test, this is a setup, because, for example, when you simulate a user and test if you can add virtual machine on your, to your installation, you have to do everything which is required for this step, then you have to run this test, then you need to perform it down, and this part quite often is more complicated than the test itself, because these two parts, setup and running test, they may fail at any moment, and it may leave your environment in a really crazy state, and you want to clean it up, then of course you have to collect logs, you cannot, okay, you don't run this test manually, you run them from Jenkins overnight, and when you come to work in the morning, you have results of this execution, but you cannot assume that you will have access to these machines where you run this test, you have to copy every log which may be important, and you have to copy it, you have to run clean versus sources, because you will run next test, and you have to report results, you keep your tests in some kind of test management system, so you have to take execution of every test and see that it passed on this release, failed on this release, failed because of something, okay, so how we do it? We use tools which we found whenever we can, so for example we use Jenkins, we use Farman, we use Git, the problem is that we didn't find anything which will help us for managing the resources and for leasing them to this test, then checking in which state they are, and leasing to another one, and we have our library, we call it art, and this is library for running this test, so okay, so what this library can do? First of all we use official APIs, we use all of them, so for example in our case you have Python SDK, Java SDK, CLI, GUI, and the API, and you don't want to run the same test for every API, you want to run it once, and do your library, let your library to run it for every API, what else? We don't run a test collecting functionality by ourselves, we use nodes for it, so everything which you have in nodes like tagging, you can use here, and when you take a look on the scope, one test is just a unit test, and usually it's not that long, because everything what is in this test is only, when we go back here, it's only this one test, this setup, one test teardown, or sometimes even without setup and teardown when it's common for a lot of tests in one suit, okay, we have a lot of helpers, we have helpers for pure open Pyramid card, sometimes it's too hard for people when you want to run something quickly, it's a little too complicated, we have tools for managing remote files, directories, and things like that, you don't want your user to run I.M. minus I.F. or anything, some slight mistake, variable breaks, I think, really, really bad, and of course helpers which we use in our tests, you don't want to run I.P. tables every time you want to simulate network connection failure, so we have just the upper split, same for disk simulation, same for load simulation, and multi-fading, multi-fading is hard, multi-fading in tests usually means that you want to do something, and in other fact you just want to wait for some event to arise, and that's it. Fat pull executor is almost easy enough, sometimes it's not easy enough, okay, so, yeah, this was the library which we used for running the test, so when we go back, yeah, so we have this part from setup environment to teardown, but still you have to install product packages, install configure the product, collect logs, release resources, and so on, so getting machine, getting test definition, we use Jenkins, Jenkins can do it for you, we use also Jenkins for collecting logs, we use our SBI case for getting resources, and then for cleaning them, I will go to this point later, but still we have this point, install product packages, install configure, so we have something which we call job runner, and this is actually what is run by Jenkins, and we have something called jobs for every stage of the test, so this is installation, this is running this test, this is for example for upgrade, because sometimes you want to test what will happen with your system during upgrade, and each of such jobs defines its cleanup, so for example, if you have job for installation, it defines what cleanup is, remove all these packages, okay, and it has plugins for getting resources, because sometimes you need resources on this stage, just for installation, sometimes you need it within the test, sometimes you need it also here, so how we, what we do for getting resources, we have, we call it resource brokers, and we actually have two types of resources, some of them you can just create on demand to some level, like VMs or NFS shares, some of them you just have a pool, nothing else you can do, for example, bare metal hosts, and of course when you have bare metal hosts, all of them are not the same, so your test has to specify that it needs to host one of them with such setup, another one with another, okay, and of course when you have resources which you can create on demand, when they return to the pool you just disturb them, you don't try to clean it, but when you have something you cannot create on demand, like bare metal hosts, you want to check in which state they returned, you try to clean it, if you cannot do it, you just reinstall everything, versus why we use Formant, you just mark this host built in Formant and reboot it, so if we cannot fix it, we will not spend time on it, okay, okay, so we have integration with external tools like backtrackers, if you know that there is a bug in, which is found by this test, sometimes you don't want to run this test, because for example, because of this bug, environment may be in such state that you cannot do anything, it may be so broken that it doesn't make sense to run another test, okay, version control system and so on, so my main thing is do you know such libraries, such tools, because as you have seen, we use a lot of tools written by ourselves, and at my previous job, we were in the same state, we also use what was publicly available, but when we still didn't have anything for getting resources, and we also use the same tools, and it's like I don't believe there are no tools for it, because if it happens twice into totally independent companies, it must be a common problem, so if you can, if you know anything which can help, I will be very thankful for any suggestions, so thank you. Thank you, Katrina, so we've got plenty of time for questions, has anyone got a question they'd like to ask? Hands up, okay, I'll bring the microphone. Hey, do you know of any solutions if you have a bare metal server and you maybe need to push a button or, well, anything that wouldn't require a robot like taking out a disk or something like that? No, we don't do things like that. Okay, thanks. Thank you, any more questions? One at the back. I think we can hear you, yeah. So you ask the question, then Katrina, you repeat the question for the microphone. For that, for those times, for managing those kind of tests, just like in the situation, we've been jacking for a month and you have to customize to reproduce the workflow that you have described. Okay, so the question is if there's a package which enables to reproduce this flow and gives you integration with Jenkins and Forman and so on, I don't know about anything like that. If I knew, I would use it. And not, yeah, it's again something. If you know something, please share it. Okay, I hope one day we will open source it. Because it's not true that everything which is in Red Hat is open source. Some of our tools is not open source. I hope it will be open source. But one thing is politics. Another thing is code state. Now I would have not show this code publicly, at least parts of it. Any more questions? Are you doing all this stuff remotely? So you're using Paramico to execute stuff on the remote VMs or servers? Or where do you, like Jenkins runs the unit tests and uses your framework and the SSH to the server or how does it work? In Jenkins, you get a host on which you run this code. And then when you want to run something on remote machines, because one test usually uses at least two machines, then you do it via SSH. And this is painful. This is, for example, Paramico is not very good. It's not also very easy to use. Is there something better? No. Unfortunately, we use Paramico. We just have our wrappers for it. So it's like, it's just, you write, you specify what you want to run on this host with such credentials, this comment, and you just get output. And it uncopped, of course. Anything else, any other questions from the audience? One thing I was going to say was that I know that where I'm working, we've set up, we've got code to set up, to run IP tables and then tear them down again after the test is finished. And if you, I was going to ask you, if you'd open sourced any of those helpers that you've listed on your slide, but the answer is no. The other thing is that we've recently opened sourced something to allow you to set up network namespaces in Linux. We're running IP tables in namespaces, which might be useful to you, and that's called nomenclature. OK, thank you. OK, any other questions? OK, well, thank you again, Katrina. I'm sorry for mispronouncing your name.