 No, try it. Check, check. Check, check. And actually, what's the... Again? Wish one up? Wish one up, but... So, wish one up, but... Yeah, but, wish one up, but... Wish one up, but... Sorry for that, because... It's very... Wish one up, but... Wish one up, but... And testing automation using this stuff. Okay, great. Okay, we can start. Well, hello everyone in last presentation today in this room. Um... Here's a... Free poster. Do you want that? Do you want to add somebody? And here's some sticker. Sticker? No? Okay. Don't forget, the presentation will be... Uh... Other presentation in the... 105. And you can win something there. And now, please welcome Wish one up, but... And his test, automation using this stuff. Hi. Hi, I know this is the last session, so... I'm not going to take long. I'm going to do it in 20 minutes. Um... Yeah, my name is Wish one up, but I work in Red Hat as a quality engineer in Glacier storage team. And my responsibility is mainly on automation side. So... I'll be talking about this stuff. This is my agenda. I'll be going through Overview and then what actually led us to come up with the the solution and writing tests and then how we did CI with it and what's in the pipeline next. Okay, so this stuff stands for distributed systems test automation framework. Uh... But actually, this stuff with the double f in English also means some tool that's used for spinning clouds. It actually makes your work very easier, so... But that's not what we came up with, but then that's also one of the things we named this stuff. So it's a lightweight test framework written in Python. It's very modular, so... We actually rewrote a couple of parts without actually changing the test cases or anything, so it's very modular. And we use all the existing frameworks like Unit Test and RPC and it's not... We didn't write anything new from the scratch. We made use of all the existing Python modules to write this. So... And we were using Beaker B framework for our test automation in clusters, so we... And one of the reason we had to write this was our frustration with that framework, Beaker B. So... And as I said, we were using Beaker B framework so it was really very hard for people to... Beaker is a Fedora project. It's an open source Fedora project, so that's what we were using. So... The programming paradigm there is... It was very hard, so we had... I mean, especially when using distributed systems. I mean, it's pretty straightforward if you have a single system project. But Galashraf as being a developer, we had very difficulty in getting people on board. I mean, most QE folks... If they are not really very much into programming, then it's very hard for them to getting on board with Beaker B because it's not very... I mean, you write... In Beaker B, like, things... I mean, things get executed in parallel multiple machines and there is no very straightforward connection. You have to do sync set and a lot of issues with that. So people were taking very long time to learn and then writing and debugging test cases was very, very difficult. It was really painful. And there was a couple of issues such as, you know, there is no very simple way of communicating with other machines. We have to do SSH and then write to some file and then other nodes should read some file. It was very difficult. And that framework was very much tied to anything with RPMs, so we couldn't use it with Debian, Ubuntu, I mean, in upstream... I mean, we had a lot of upstream customers using Ubuntu and Debian, so we wanted a uniform test framework for upstream and downstream as well. So we came up with this staff, so it solves most of the issues that's Beaker used to have. And so in here, we have kept the framework and the infrastructure management as separate. You can use, I mean, to provision machines, you can use anything. You can use OpenStack or Rackspace or your own Wegrant or even Beaker itself. But the test framework, once the test machines are provisioned, so you can use Distuff for that. And Distuff by itself doesn't do anything much, so it just gives you a small test runner and then provides a couple of APIs for you to run the tests or orchestrate how your test should go in the remote machines. I think that will be clear when I go to the architecture diagram. So the dotted line, dotted box here, that's the Distuff. So it can Distuff this whole of that thing runs on a single machine. So we are just trying to we are not solving the problem of distributed system here. So we're just trying to solve the problem of testing distributed system. So it's okay to have a single management node. So this is our management node where Distuff runs and the test machine one to machine and that's our test machine. And they can be physical machines or VMs or even a container. All that needs is SSHD should be running in all those machines and there should be a passwordless SSH setup between the management node and your test machine. And it's logical nodes. Distuff can also management node can also present in one of those test machines as well. So this is how the flow works. So you have a config file where you specify this is my server configuration and this is my servers and these are my clients and the config parser reads it when the when you start Distuff, the config parser reads the config file and connection manager establishes a connection and it basically uses SSH tunneling inside. So we establish a connection to all those remote machines and then manage it. Connection is made only once and all the tests are run, all commands, remote commands are run through that connection and then the connection is taken down at the end of the test run. So once the connection manager establishes, connection is established, all your test cases and all your libraries should talk to remote machines through one of those APIs provided by the connection manager. And we have a way to discover test discovery, so I will come to that a bit later. Okay, so writing test cases is writing test cases is very easy. I mean, what we had in mind was making automation write test case automation very simpler. So eventually what we came up with is like test cases are written in steps, I mean, step one, do this in this machine, do this in this machine. So with Distuff, you can just write whatever is written in English, you can just write them in Python and you need not to write Python, I mean, it's very basic, so the onboarding process is very easy, I mean, all you have to know is very basic Python, that's all. So connection manager once established, I mean after the connection establishment, it gives you a couple of APIs and a couple of variables through which you can actually access all your servers, all your nodes. And that's that's agentless connection in the sense only SSHD should be running, so no Distuff doesn't run any program by itself on the remote machines but you need to have SSHD running and so at least in case of Gluster, most of the commands that we wanted to run on the servers are basically bash commands, bash or shell commands. So Distuff gives you two APIs through which you can actually run all your bash commands synchronously or asynchronously. And then if you want to run anything Python commands on the remote machines, you can request for a connection and all the Python operation that you do on that particular connection runs on that particular remote node through which you requested the connection with. Suppose like if you want to open a file and read a file in the remote node, you have to get a connection to that node and all the open Python open and read write that you do will be on the remote machine. Yeah, so that's how we make the discovery, so I mean test discovery, so all your test cases should have a decorator called atTestCase, the test case name. So when Distuff goes through your test list or test source code, it reads all the decorator and puts them in a list of them one by one. You can actually control what should run, what should not run in giving CLI options. So I have a sample test case code. So if you see from Distuff you import TC, so TC is my global connection object through which you can access all your servers, all your clients and it provides a lot of APIs which you can run commands on the remote node. And so that's my, so when there is a okay, so the test case can be either a class or a function, it doesn't matter. But then if it is a class it should have the methods set up, run and tear down. So and so you write, you specify the classes in that test case decorator, so everything at all Python functions or classes that has particular decorator is considered as a test case. And if you see in the doc string I have two values, I mean key value pair and that's basically written in YAML format. So what we do is like we, I read the doc string and then pass the YAML format in it and we get the dictionary. So we make a couple of decisions based on that particular particular value. Suppose like reuse setup, so if your test case wants to set up a fresh setup then you can make it reuse setup false, which means like setup will not be run and directly you can go to the run part of your class if you have a class. And you can actually have, this is not implemented yet, the tags. So the plan is to like have tags for a particular test case, I mean this is my regression case, this is my longevity case or this is a priority zero case. So while running you can specify around all the cases that has this particular tag. So you have access to your servers and clients with the dot nodes, dot clients, their Python list. So yeah, I will just explain, I mean I have just, this is a dummy test here, so I have written, I have explained, I mean three basic APIs that are provided by this type. One is dot run, dc.run and run async. So both are used to run any bash or shell commands in the remote node. So that dc.run takes two arguments, the server in which you want to run any command and the command itself. And it returns you three Python tuple basically, so that's one is written code, the STD out and STD here. So if you want to do any negative test case, negative case where you want to parse the STD here and then check something that you can use, you can use the third STD here. So I'm just running some ntpd start and then I'm running, I mean using this find command in the client, I'm renaming all the files that's in the mount point and run async gives you a object which is basically inside a Python sub process object, so with which we can do wait, kill, return code or anything that Python sub process can do. And so while that is running in the client, I can do anything on the server, so this is just an example, so I'm getting a connection to the server as a root and then I'm just running a github's name. So this connection, so all your commands, so Python commands, if it is a built-in like open read write, you should specify a connection dot built-in dot open file or read, something like that. If it depends on a module, like here socket, connection dot modules, dot your module. If you want to make use of a non-standard Python module in the remote machines, you should make sure that this particular module is installed in the machine that you get connection from. Yeah. Yes, there's a typo. Yes. Yes. Yes, but then they're just I mean, I'm writing, I'm using this for Gluster, so there's a differentiation, but then there's no Yes. Exactly. It's just a connection manager which once connection is established, you can run it's exactly the same methods that is available for both servers and clients. Yes. I mean, in our case, server sees why Gluster is inside client, so that's all. Yeah, so yeah, so after I mean, after I get the host name, I'm actually waiting for all the re-names to complete with the wait method and I want to actually check the output of that. So I have .value return, that's one of the things that that's written on top. Other than what Python's process gives you. So again, that gives you the same output as the return code, and then you have to close the connection. So as you can see, so writing test cases was pretty simple. I mean, you have cases written in English, so you just have to convert them to Python, that's all. Mostly not always. So you have to specify your servers, so we have servers and clients, so you can actually omit one of them, in which case that will be an empty list. So but either servers or clients should be there, but if you change the config parser, you can have your own keyword. So we specify all the servers and clients and all of them. And then I set up a password list as such to those machines. And if you have a lot of machines, a lot of test machines, then it can be a bit harder, so there is a script that basically makes use of bash, expect package to set that up, but then you have to have all your remote machines password same. And then currently we have options to run all the cases or all the cases inside a particular directory in your test case source code or everything from a file or just a test list of test cases. We want to have a couple of improvements on there. And the results are either in text format or they are in JUnit style format. That again can be specified through command line. So JUnit style can be rendered by Jenkins friendly and can show a pretty output on the Jenkins project. So CI. So this is, we are using this mainly for Gluster so far and couple of the recent changes that went in are very Gluster specific. So in here, in Gluster we are making, actually we have two options, both are work in progress. One of them is we are making use of CentOS for CI. In CentOS, they have a program called Duffy with which you can actually request all CentOS machines with couple of options. So we make use of, the plan is to make use of that and we have like halfway through that. Once you get the machines, it gives you host names of, I mean whatever the machines that you populate them in the YAML file programmatically and then start the cases that you want to run. So in this case, Jenkins live acts as your disk of management node and the test machines are reserved through CentOS infra. So in other setup, also work in progress but it's mostly more or less similar except that we request or we provision all the test machines through Wegrant with LibWord, I mean instead of Duffy. So you can actually make use of RackSpace or OpenStack or internally we use Beaker as well. So what we have in plan, so if we have couple of very basic test cases that doesn't require any stress or anything, we can actually make use of it. So in this case, we want to make use of Wegrant with LXE plugin and as I said we want to have couple of options with tags and then skipping it or running only those cases which have particular tags and currently the results are only in either text or JSON, sorry JUnit, so we want to have couple of more options there that's all. So if you have any questions, yeah. No, I didn't understand to get you a question. Yes. So the connection manager knows that server 0 or client 0 has to proceed but it has to have client 0 that now make a connection to those. So the servers and clients at least this stuff doesn't connect servers and clients by itself. So client doesn't know anything. Right, the client doesn't know anything. You have to get the receive whatever you want to do through client and then tell it to the server through the management node. Yes, but then it's not very straightforward. Even in multi-host they each machine knows what are other machines in the cluster but then it cannot talk to simply it has to use SSH and then all those things. So in here the servers itself and clients themselves are dumb so they don't know anything what other what server doesn't know how many servers are there in our client. Yeah. Yeah. That's I pass it through management node. Management node knows these are my servers so I tell client to mount it through this host name. I can pass it. The management node knows everything so it can pass around all the information that it like but there is no direct communication between the servers and clients. Well we don't have anything in the plans at least but yeah if it makes sense but then as I said like we came up with this because of all the frustration that we had with multi-host beaker environment and at that time we did research I mean we did a couple of others but then that wasn't I can't tell all of them I don't even remember all of them on top of my head but then we had a lot of issues with I mean one or other issues with many of them like in here like we wanted to be connect to windows for Samba testing we do that through running open SSH or free SSH in the windows and then connecting via making use of PowerShell so each of the test framework that we looked into had either one of one or other shortcomings so we had to come up with this but then at least now we don't have any it's been I mean at least it's been working good so far for us so we have no plans of migrating but then if there is a lot of issues with this as well then yeah I have heard of I came to know about after this but then as I said nothing in the plans no so there is a global YAML file that has your environment information like these are my servers clients and then this is my log file log level and all those things and since we have cluster we have a lot of options to specify the volume configuration in cluster as well but each test case can have its own configurations that you can specify in the YAML format in the doc string there so before running a test case we actually at the beginning of running all the tests we read the doc string and then we parse it I mean parse it through YAML and then make decisions based on that so but then again we have very few options I mean like right now we have only three options and those are all cluster specific because the test case is written for cluster but then you actually you can make use of I mean write your own fields in the YAML format and then make decisions on that but then you'll have to write that config part for that whatever it is written you have to have a config I mean you have to parse it and read it and make decisions based on that there is no YAML file separate for a test case that's the one for particular one that particular one I mean it doesn't I mean it makes sense to have the test case and it's related metadata together I mean it's related information in the same file at the same place no I didn't understand okay run test yeah so provisioning and tear down you can use anything you like it's not tied to particular yeah I mean but then you have to have but yeah I mean you can but this stuff doesn't take care of any package installations and everything you have if you want to do that you have to write that module or functions or whatever to do that in in this stuff yourself right now it doesn't do any package installation that's all kept separate so you can use anything you can use Ansible you can use whatever you want to install the packages and softwares yeah just the test case just the test case test cases in this stuff no questions thank you thanks for coming