 Okay. It's recording. Hi. So thanks for coming down and the title of the talk is Testing Containers with Tune. A little bit about myself. My name is Kushal. Kushal Das. I work as Fedora Cloud Engineer in the Fedora Engineering team. So the cloud part is mostly my day job. I am a Fedora contributor otherwise. So I'm doing Fedora for like contribution part for over 10 years now. Mostly in the ambassador, marketing, documentation. I mean, that's the various parts of the Fedora. I'm also involved with the Python community. I'm one of the code developers of the Python programming language. I'm also director at Python Software Foundation. And that's mostly about me. So before going into this talk, I want to say something that this particular project, Tune, we don't use it for testing containers in Fedora. We are going to use our official tooling that is Taskatron to test container. And it will do the CI work for us. So this project, what I'm going to talk about mostly is for the users directly. And my users are software developers who are like me. So the first thing why I wrote this project or why I started doing working on this. So I can admit that I'm a pretty bad system administrator. So I'm not very happy with or I cannot do a very well maintenance of our big thing like Jenkins or anything which require a lot of setup and then maintenance, all those things together. And I was looking for something which can use to test things out, which can I can run mostly on my laptop if I require to that means it should not destroy my current laptop the way it is. And also it's something, as I said, easy to set up. So this particular tool can actually use both INI based configs or JSON based configs, which is basically 45 lines of text. And the idea is happy developers. So you can and it's a command line tool. So it does not require any service to run or up. So what you can do is that you can SSH into any server you want and just fire up there. And you can use it. So it does not require any cloud. What it can do is actually, it can use QCUT images from any cloud provider right now, like mostly federal and centers is what I tested for. And it can take those QCUT images, it can fire up a VM on your local computer or wherever you were running the tool, and it can do the all the testing on top of it. That means you'll be able to do it either a cloud based image or federal atomic, for example, and then do whatever you want to do, and then get out and it will destroy the setup and everything. And your system will be still normal because then you have a chance to do the setup and everything cleanly on a cloud instance. Because that was one big problem because yeah, I managed to screw up many server instances where I did not know what exactly I was doing. So I always wanted to have something where I can destroy and get back to the same state. So while working on this thing, we got another actually a small requirement that instead of cloud QCUT images that some people want to use Vagrant, they want to use Vagrant either Livered or Vagrant virtual box images and then fire up the test on a Vagrant instance rather than the Livered part. So it can handle Vagrant images. So it's actually been used for testing the cloud images we released from Fedora. So but it's only for the cloud part. So we do the cloud based image and that of two week atomic release testing, it's completely automated in that URL, the project name is Auto Cloud. We fire up jobs there and you can visit it later on if you want to. But it has nothing to do with the containers first. I used to need for my personal containers. So I test the containers which I run for my own usage. And I think the most important one is for my wife because her blog is running on a Node.js container on Fedora. So I want to much very much feel that whenever I'm upgrading or doing anything on it, I don't break it. So I have some personal tests which I've written to and system where I can fire up the latest container build to test that container using today. And there are other things like if you go to my blog, which is actually a search engine small called search.qshelders.ing. It's actually running in a container Fedora container. So before anything else, this is the most important part of the talk, I guess that few years back, people were going cloud cloud cloud cloud cloud cloud from there we moved into the Docker Docker Docker Docker because everyone wants to be in container and things. And while discussing with people and when I started learning about containers and stuff, I found two different world altogether. One world is where Dan is coming from and many other people. And then there are a few people who always said that containers are some rocket science. And like, we have to think completely differently while thinking about containers. But that's not my view. So a little bit about my view part. What I think about containers is it's just another process in your computer, but with some kind of restrictions. I'm not saying the word jail, but containers are again a similar process which is running in your computer with some sort of restrictions depending on how you are firing up the container and what are the features or permission it has. So my idea about testing containers was or it's still now is, is you tested the way you test any other process in your system, but along those restrictions. So you may want to like if your container doesn't have network access, I don't know why, but if it doesn't have network access, the process should not get network access. That's the idea. So a coming little bit into, I mean, how many of you are actually using containers somehow somewhere? All of you. So you already know this part. So what developers normal workflow, they do Docker pool, Docker run, if we are running something from Docker hub, correct? I'm going to take few examples. So I'll not try not to bore you with my tooling things. So I'll take few actual containers and see a very basic level testing, which we can do with things. My most important thing I use in my all infrastructure projects is Redis. So if you know, it's a key value store. So you have a set of operations, a few are examples like these, like you can do set, get, you can increment a value, you can have like has maps and other things inside ready server. But end of the day when you start the service, Redis, it's a process which is listening something over this particular port. So it boils down to that, we can access the port and we can fire up the actual commands which Redis expecting and giving giving me the similar output. So I'm going to the first demo, not serious. This is large enough. Is this large enough for everyone? Yeah. Okay, so I can wait from here. So I should do this. So today requires basically two files for any particular test. We accept either a CFG file, which is basically INS syntax key value, or we can have a JSON file, which is the old style of doing things. So going to the Redis example. So this is the setup, which is actually required. You can tell me as developer or users if it is difficult enough, I found it was easy enough for me, like use this cloud image, one CPU and this much of RAM. And the user, by the way, because if you are doing CentOS or other image, you may have another user. So that's what it requires from the setup point of view to run on your laptop. And then it requires a TXT file. So actual the command of the actual test happens from a text file.txt. So each line is supposed to be a command. So you can see here, because I'm using the base image, I'm just installing Docker and Redis also because I wanted to use the client from the host to the container. I'm starting the Docker and then pulling the Redis image, firing up the Redis container, then using the Redis satellite to set a name and getting the value key is named. So for when we are actually doing the cloud image testing, we have some code written in Python, Python unit test cases, which we just fired up Python space something. So if you have any actual test cases written, it doesn't matter which language it is, because it's a dumb tool. So it will just see that execute this command, it will execute. And if it is a non-zero written code, so it will stop the execution and will tell you what exactly went wrong. So that's the whole idea. So let me try to fire up the job. So this is how we fire up the job. Tunis space, if I'm not, yeah, tell me. What does hyphen hyphen multi mean? Yeah, so coming to that. So previously we had like a bad name choice. So I only had hyphen hyphen job and the job name. So it and multi a job, whichever way. So for multi it actually expects that there is a dot CFG file, which I'll deprecate and I'll use hyphen fn job and it will try to find both the CFG and .json because I accept both of them. And it also expects that there is a .txt file with the name and you can give the config directory if it is not in the current directory, some other directory. But yeah, that was just because, yeah, it's only me, the only user till now, the hyphen fn multi part. So it's just there. We just have to decide like one single parameter. And so in the newer system, it actually queried in the SSH keys and stuff and firing up. You can actually see the command it's using the cubic KVM and it's waiting for some time to see the boot images booted properly or not. And then it's going to fire up all those commands. So you can see it's firing the command will be still looking into the screen for the next few seconds. So yeah, because it request to the access type properly with the pseudo part. And then it will execute those commands one by one. If it stops, if it fails any of the commands, it actually stops there and there. It gives you the output and then actually destroys the VM. So there is a command line option hyphen fn debug, which will tell you to not to destroy and it will give you a way to log into the VMs and if you want to manually introspect what's going wrong. So a lot of output pasted fast. So if I go back, you can see it actually install Docker started the Docker Docker pool readies. Start the container. Then using from the host, just accessing the container. So if any of the steps fails, literally, it will stop there. It will tell me this is what I mean. And this is, as I said, it's an example. So this is doing the very minimal level of testing for the Redis container. Just one single command. Yeah. Okay, so. And the last part that was a special feature request from Matt Mathew Miller for our cloud stuff. So you can actually mark up any test case inside Redis as a non getting tests. That means even if that test fails, Redis will continue to need will continue doing the test cases. So those are non getting. If they pass, it's okay. If they fail, still okay. I don't care. I'll still continue till things. So and we wanted to have a report like total how many number of non getting test cases and how many of them passing or failing. So the way we use in federal cloud, this part is any new test case. We enable them as non getting and wait for few days, see it's passing properly. The test case is actual correct or not. And then just remove the first two hasses in front of the file and then it becomes a getting test. What what defines as getting or not getting? Oh, it's your choice. As you know, how do you tell the system? Oh, so the test case, it contains double has in the front of the command. So two hasses front of the command is non getting. And so yeah, this is extra because we have to do something to figure out those things. You can also do I think has a person sign to say that some command will fail. So you are expecting a command to fail. You can also specify that part. For example, which we found later on that if you do pseudo reboot or pseudo power off kind of things, theoretically, it's not failing because it's actually turning off the system. But because the command or system in this case suddenly sees that Oh, there is nothing. I don't have anything to do. It thinks it's a failure. So system B returns or the command actually returns minus one or some other minus values actually. So I had a chat with Leonard about it. And he said, you look into it later on. So that's why we actually first time, I think it's double accurate sign for that. If you know for sure, it's failing. Double what? Adulterate at double that. So if you're doing the summary for non getting test, we're about summary for the remaining test, why not doing that one as well? It's just like it is confusing that it is telling me that because I'm a dumb developer, I don't read titles, I see the numbers and the numbers are telling me there's no tests. I'm good. I didn't fail. Which is why there's no test. And I'll probably start looking why it doesn't pick up my past. And only after an hour or two, I'll find out that there's no getting test at the beginning. So that's mostly because I can say that's mostly because there are not many users of the system. So it's a valid point and we can it's just a printing the output in a proper way. So we can just enable it. But because there is like we use it. And as I'm saying it as for mostly personal use case. So yes, like if it is a I'm thinking it's a valid point, and we can do a better summary in the end. I'll come back to that in one part of the slides. Go back to the slides. But in real life cases, if you are doing it for any of your professional work or a private company developer, or even if you're doing it a little bit seriously, you may not want to download things from ready sub Docker hub. Instead, you want to have your own private repository and then pull the images or the containers from there. So I don't have one because as I mentioned, all my things runs on one single bare metal and in containers. But I have a use case now, which is thanks to Adam's work, we now can build cockpit as a layered image container in Fedora. So I'm just using it as an example. So that's the demo number two. So, okay, I never wrote any actual test for this because I don't know how to test cockpit. So I just wrote a basic thing that you can understand this one thing that I'm running an atomic host in this particular case. Adam gave me the commands what to do to run the containers on IRC. So I just added up and then I'm just running the next step tool to just see the output. Nothing fancy. Could you pipe that to grep? Like with Tuneer, would it be okay with the five characters? Yeah. So Tuneed itself doesn't do anything at all. It just picks it up, fires up on the SSH to the VM. So it doesn't know what it is doing. It knows that this thing is supposed to execute properly. So this is the old style, the JSON file, which we have. So almost similar, like RAM user, the image, which you can see the Fedoratonic, that is last month's 21st. And I'm saying type VM. I'll come back to that point. But if I remove this and then just run, and this is the hyphenfn job part, which is the real running job thing. So if I fire up this job, so it will do the same thing. It will create SSH key if we fire up the VM with that SSH key, then it will wait for a few seconds to see if it is booted up properly or not. And then we'll see the exact same thing. Wait and see. Okay, it's trying to get the latest cockpit from our registry. So for the video users, if you're not seeing this properly, in my slides, I'll link you to this demos. And it was kind of fast after that. So it basically pulled down the whole image and then it fires it up. And then it's running the next command. And that's the cockpit actually running there. If we are moving on, next big thing. After container, container, container, suddenly people moved into Kubernetes, Kubernetes, Kubernetes, and they want to now do containers on top of Kubernetes. I myself don't know how to set it up. I don't like it's something long time back, and only once or twice properly maybe by hand setting up Kubernetes and stuff. But thanks to the upstream, we have a pretty nice Ansible repository in Kubernetes country. That's the project. And we can actually use that to fire up Kubernetes on any host. And for my example, I'm going to run their example, whatever configuration is there. So that's a three node setup, one master, and it's also content that it's a TCD and two nodes for Kubernetes. So I'm going to do that part. So multi host demo. So the same way I'm using the CFG. This is a new thing because for multiple hosts, the only way to do it is over the CFG file. So on the top, it is containing the CPU, the RAM, and one extra thing, the Ansible directory, because I'm going to from our jobs, we're going to fire up some Ansible thing. And this has to be pre set up, like whichever directory you're going to use or where your playbooks and the roles are. And then I need three VMs. And I'm also setting up the host name for all the three VMs. So this is one more thing, like we can actually pre define an inventory file along with the host names, because we still don't know what will be the IP correct. So I'm going to do this. Moving out. So and this example is copied from their current example, and put a little bit smaller node names of host names, keep master example dot com, keep node zero one, keep node zero two, that is, yes, I DNS for the VMs. Oh, that's our standard livered, whatever livered is using the it will use DNS mask, but where are you configuring these host names? So okay, so when today fires up multiple VMs. So the first thing it does is that for I mean, as we mentioned in the configuration file, these are the host names, I try to find like, or the tool tries to find all the VM IPs for each of the host name, and it puts inject six in the VMs for the PTC hosts. So no DNS handling, it's a TTC host entry. Okay. All right. So it's easier. Yeah, just yeah. So this is the extra thing. One extra thing which is we are doing for Ansible and only for Ansible part. So if I do this, again, this is a pretty, pretty old example, I have a blog post about this part. So what I'm saying is that that's a playbook. And you we know the path from our configuration. So run that playbook, please. And in the second step is, what do you call it? Nulliq, correct? Nulliq. Okay, so that's the Nulliq example app written by the Nulliq team. And so I'm going to just fire up that app and it's going to use the whole Kubernetes setup and everything and then fire up a proper Nulliq specified container. So the same old command, have an event multi on the cube, that's the name of the job file and the text file. So now instead of one single VM, what it will try to do is it will try to fire up three different VMs because as I mentioned, if you remember the configuration file here also can see I'm using the federal atomic image. Because that's where I'm supposed to do. And then it's firing up the whole Ansible part and it will go through all the steps. I chose this, this as an example, because as I mentioned, I did not do enough Kubernetes work. So I found it's much easier that if I use an Ansible playbook and use their official example to fire up the Kubernetes things, it will take almost two minutes, I guess, you can see it's firing up the whole panel and other things. Yeah, we're going to wait for this. A lot of output. So the idea is, right now I'm doing only Ansible, but in future, we should be able to fire up any host level commands like instead of Ansible, if you're using something else, like some other system to configuration management. So in the, in the tenure value that all caps playbook? Yeah, because that is the only thing which have like particularly like this is an Ansible playbook and fire up this. But for everything else, I mean, in future, I hope like we'll just write down the whole command. How do you, how do you tell it where you're going? Oh, so that was coming from that configuration file where we fired up the VMs. I had Ansible underscore dir. And that's where like, we assume that you already created most of the inventory file. So now it is running the actual thing. Okay, so it fired up the stuff and it says the application resides in some place. You want to see that foot? Yeah. Okay, so I think I asked Jason Brooks over IRC, like, how do I test this? And he said, this is how he test if this command is successful. That means the all the containers came up properly. And it's actually talking to each other nicely. Okay, so or little talk to each other because it is some time for it to deploy to capabilities. But yeah, I mean, yes, no, it was done. Okay, so no liquid job is done. So that test is successful. That is multi host. So but as I mentioned in the first that it cannot do any cloud or it does not use any cloud, that's true. But at the same time, it will lie because to me has the capability to fire up jobs in AWS. So you have to just provide it, what do you call that indication all the odd details and what which AMI you want and where you want to fire it up, how much things you want. And it can actually fire up that instance over AWS and then start using it. So and we have a patch waiting in our GitHub repository, which is the open stack patch. So after we march that patch, we should be able to fire up the similar jobs in OpenStack. It is still waiting because OpenStack requires apparently a lot of different configuration values over command line tool or over the Python. I mean, we are using live cloud, Python live cloud for that. So it will require this. And as I mentioned, it's, it's a dump tool to me does not do much other than firing up a VM and running some commands and telling you about output, it doesn't know anything else about the whole environment. And we are pretty happy how it behaves till now. So and we are really not looking other than maybe few like, as he mentioned, he wants a little bit better output, which is correct. So other than these kinds of simple things, we're not adding up any new feature. We don't want it to do 100 things. We wanted to do one thing and just one thing properly. And so the one last output is actually in future, we want machine readable JSON output. Because right now what we are doing is only plain text, which may not be nice for people if they want to use to need from another tool or another programming language or anything. So we'll have some output file along with the text file as a JSON output, which will have much more details about how it went and where it went. Actually, that's it. If you have any questions, anything about part, I'll file up a bug for this. No, yeah. So how did you implement the later part? Do you have part of the vagrant commands? Yes. So the vagrant part came up very nicely over a meeting that almost started at 1 a.m. in my time. So we had like federal 23 time, I guess, like the release, like before release with like go no go meeting if the project was correct for automatic testing part. And suddenly figured out like, oh, we have to do vagrant also only. Yeah, somebody came to us like, we need vagrant. And like, well, nobody asked for it. And what we're asking now. So yeah, so yeah. So what it does is it actually fires off the step by step vagrant commands. And that's why like, it can handle both the vagrant leave word or vagrant virtual box, because it's firing up commands. So that's how we are doing vagrant stuff. And how do you do the think the same as so you message to the machine. Yeah. Yeah, since I don't know vagrant has these plugins. Yeah, so I'm not to use all those vagrant plugins and stuff because I don't want to write more code. Okay, as mentioned, there are bugs as well. So what I would be doing actually just after firing up the VM, it finds out, I mean, it creates a string three lines in this case for all the hostname IP thing. And this keeps injecting in all the three VMs in the ATC host so that each one of them can talk to each other without. And it's all like, in this case, it's all getting handled by live word, all the actual VM creation and stuff. So I don't have to do anything. What I'm actually doing is firing up some commands. The whole code size, I think maybe at max 1000 lines of Python code. And that's why I said, I don't want to add features, but want to just whatever is there makes you that's useful. And maybe a little bit clean up things. That's this thing. And the documentation is hosted in reader docs. So to need to read the docs.io. My contact details, my blog and my Twitter handles, if you have any questions, feel free to ask. Probably did all this much carefully about the queue part. But why not spinning up whatever you want against the queue against already running queue? I mean, I have an external Kubernetes Kubernetes, I have a external Kubernetes running somewhere over there just for testing. And what I only care about using tuner is to verify the images that I produced against that particular queue or for our ownership instance. Okay, so this is the benefit, I believe of tuner being the dumb part. So what you're going to do is write a big command correct, which will do the thing for you. And some configuration. Yes, correct. Well, you could you could spin up. No, I'm not talking about even the spinning up part, let's say I want to use your already set up Kubernetes or open shift. So what I to do so I have to fire up a command correct. Yes. So just type that command into tuner.txt. Yeah, what I have here is TL or OC create should be just type that command. Yeah, I mean, it doesn't care where you are doing or what is in that command. Yeah, that's true. That's so it will use open safe it we can if you're using that command line mode so it can use anything you just push it to if it's a Linux sale command, it will fire it up. It will do the work because it doesn't know what that command is doing. No, read the line if there are like only three things to check in the front double aggregate this command will fail double has I don't care if you fail or pass playbook Oh, that's a playbook. Yeah. So keeping it simple was the goal from the beginning and I started writing it at the very beginning was because I was kind of tired of manually firing up a VM going into it and then running the command to see if the image is correct or not. So it was doing things for me personally. And we found it that time was useful. So keeping it simple was from the beginning as a goal. And I think some I'm somewhat there but like, as Adam pointed out the hyphen admin multi thing which was a stupid maybe decision from my point, this could have just use half and job as it is there. But with the multi part one thing came up was hyphen admin div up thing. So like when you I'm writing a new setup test, I may want to get into the VM and then see what's going on or what is the kind state. And then I'm also creating a destroy a cell script for you and the host so you can just run the cell script clean it up completely and then fire up the next job. As I mentioned earlier, my slides, you will find the links to the demo and you will be able to type it. Any other questions? Thank you.