 Thank you all guys for coming. I wasn't expecting this many people, but that's a good sign, I guess. First, let me start a little bit by introducing myself. My name is Gonzalo Raffuls, and I do quality engineering for Red Hat in Czech Republic for its virtualization solution, which is over on the AppStream. So during this talk, I will try to go fast and make it light, it's been a long week, so to begin with, I'm going to be talking a little bit about I'm going to actually show you a simple test scenario for running PyTest on Jenkins, and we're going to have a little bit of a higher review on PyTest custom plugins and how PyTest loads custom plugins, and then we will look a little bit at the challenges faced at the time of trying to replicate these Jenkins job on your local environment. With Docker, and we will look also at Docker as a solution for doing this, and then if we have enough time, I'll make a little small live demo if it doesn't fail miserably, but yeah, okay. Now I'm not going to be talking, I'm not going to be, I don't want to intend to sweet-talk you and to start using containers. You should be aware of the security implications that comes with containerizing your applications, and we're not going to talk about containerizing Jenkins that's already been done, and it's ideal if you already have some some high overview on Jenkins, PyTest, and Docker, and also most people would think that this is about the Docker PyLibrary, which is for managing your containers through Python code, but that's not the case in in this talk. Okay. Now, so why did I decided to talk about this and how did we came up with this solution for for containerizing Jenkins? Now when I first came to Red Hat, I was given the task of replicating my Jenkins job, our testing framework on my local environment, and for doing so I was given a six-page document that it actually took me approximately two weeks to complete, and then when I realized how big of a fool's errand it was to to try and do so, plus like I also realized that most of the people on my team was was having issues when trying to replicate their environments as well, but at the same time those environments that were successfully working at some point with the constant evolution of our testing framework those setups were easily lost, so there has to be a way to to keep up with this and how it's centralized and make sure that no one had to waste two weeks of their time to set it up. Now at the time there were, we were not using virtual environments, therefore you had to install the all the requirements from the testing framework on your local machine and that could have potentially caused some dependencies, some dependency issues on the libraries that you already had installed on your machine. Now also like the biggest benefit of these was to be able to put Docker image on an internal registry that you can run on your organization so that whenever a newcomer started working just by pulling the image would be able to start debugging straight away. Now, how many of you have been using PyTest on Jenkins already? Okay, that's a good number. So there's a couple of things that you need to take into consideration at the time of trying to replicate this Jenkins environment or this job run. So first thing if the job that you're trying to replicate is PyTest, you should look at the PyTest options on your job, on the configuration of your job and here you can have all sorts of things and these parameters can be used by Jenkins in many other places during the job execution. In the case that I will show you in this sample scenario, this small sample scenario, we only have a parameter that will be used on one of the build steps that I will show you further on. Now, you might also have distributed tests and you could also have parameters for determining to which hosts you want or on which hosts you want the job to run. So that's one thing to take into consideration. Then if the job itself has any git integration and if you're actually this is the most common practice, so you probably do so. So it's important to check that all those repositories are going to be used for our local environment. So this is really important and we will actually link all those repositories to volumes on our containers. Now, finally, I would look at the build steps on the configuration of the Jenkins job and it will depend on the complexity of your testing framework, but you might have, in this case, I just have one build step where it's actually a dynamic bar script that takes one of the parameters that I defined before for choosing which tests to run and you might have additional build steps that could, for example, in our case, generate configuration files that are used by custom plugins and you need to make sure that all those steps are covered before you decide to run PyTest on your container. Now, with PyTest virtually, oh, I see, I got it. Okay. With PyTest, virtually any Python module can be registered as a plugin. So how PyTest does the plugin discovery, how it checks all the plugins that are on your testing framework, it first starts by loading all the built-in plugins that were installed with PyTest. Secondly, it looks for, it actually loads all the plugins that are registered through setup tools entry point. I will show you a small sample on the next slide where I am actually passing one plugin that is outside our testing framework through the setup tools. Third thing that it does, it pre-scans the command line for the minus p name option and it loads the specified plugin before doing the actual command line parsing of PyTest. In this case, you can also choose to deactivate already registered plugins by prefixing the minus p option. Actually, I didn't expect this to show like that, but it's minus p, no semicolon and the name of the plugin. So that's how you avoid loading plugins that perhaps are causing issues on your testing framework. Next thing that PyTest loads, it looks for all the comftest.py files as inferred by the command line indication and if you don't have any test paths specified, it uses the current directory as a path. Now, you need to note that PyTest does not find comftest.py files in deeper nested sub-directories at tool start-up. So it is usually a good idea to keep your comftest.py files in the top level of your root project. Next, it does also load recursively all the plugins specified by the PyTest underscore plugins variable that you can specify inside your comftest. So that is actually the last place where PyTest looks for custom plugins. So now if you want to make your plugin externally available, you may define what is a so-called entry point for your distribution so that PyTest finds your plugin module. Entry points are a feature that it's provided by setup tools and PyTest looks specifically for the PyTest 11. It looks for this entry point and it actually looks for this entry point and it loads all the plugins that you have defined under this entry point. Right. So now when I first started to trying to containerize these environments, I started by creating a base image with Fedora and a couple of instructions that I already knew that had to be there. For example, one of the first things that I had to do was at the time of linking my local repositories to volumes inside my container, I realized that I had to include all these repositories into my Python path. So I managed to do that by including the Mv instruction on the Docker file for creating the image and later on I started trying to install all the requirements that were defined on our testing framework requirements.txt file inside the root directory of the testing framework and it was then when I realized that some of the requirements that I was trying to install with PEEP had had header files from RPMs that were missing or were not included on any place on my testing framework because all these dependencies were either already included in the operating system that was being used for running these tests or being provisioned by Foreman or other services that were invisible to my testing framework. Also, how I managed to see which plugins were loaded by PyTest at every point while I was trying to containerize these was by using the trace config option from PyTest which actually shows you all the plugins that are being recognized by PyTest at the time. Now, as you can see there, I have this run statement that I included on my Docker file. Now, it is recommended that you minimize the number of run commands because in that way it makes for less overhead at the time of actually running your container. So, now, when you're leaking a volume on your container to local repo directory on your machine, if you have the SE Linux policy enforced, then the container won't be able to write to this directory. And in the case that it might be possible that your tests are generating log files that are saved on that same repository. And if you have SE Linux enforced, the container will fail with a permission deny or an IBC message in the host sys log. So, to avoid that, you need to change the context of the directory, in this case the repository directory, to that tag in particular, SV sandbox file T. Now, this has become much more easier since Docker 1.7 because as you can see on the second piece of code here, I am preparing the set at the end and since Docker 1.7, this statement automatically changes the context of the directory that you are trying to link to your container. Therefore, the container will be able to write to that repository. Now, further on, at the very end of my Docker file, I define these two instructions, the entry point and the CMD. Now, the entry point, it virtually makes your container executable. How so? Because when you run your container image, the first command that the container will run will be the entry point and the CMD, it's any additional options that you would like to run with your entry point. But why is it separated like this? It's because the CMD is overwriteable. You can overwrite that by adding options to your run statement for when you want to run your container and therefore it makes it more flexible. Now, at the time of debugging your PyTests, you might want to, I'm not sure, I use IPDB for example, but unless you use PDB or PyTests on PDB, you won't be able to debug because PyTests is capturing the standard output, so you need to run PyTests with the minus S option so that you are able to use any other debuggers apart from PDB or PyTests PDB. So, I went through that real fast. Now, I will show you a little bit of a live demo and a small scenario that I made with Jenkins. You can find the testing framework that we're going to be using on this live demo on this repository and the second repository contains the Docker files and the Docker compose file that I am using for instantiating these Jenkins environment. So, I'm not sure if I'm able to see here, but yeah, cool. So, this is a Jenkins instance that I have running on a container, of course, in my local machine. And this job in particular is a parameterized job that takes only one parameter and based on the input that we put there, it will run one test or the other as we saw on the bash script before. So, I'll run a bad test and a good test. One should fail, the other one shouldn't. Most likely both will fail, but... Okay, so, we got a failing test here and the other one, it's success. So, let's say I want to debug this failing test that is running on my Jenkins instance. Now, I already created a container image that will replicate my Jenkins instance and a container that I can use for debugging my byte test locally. Now, I have some scripts here. So, if I run this script for running this bad test, I can show you what does this file contain. And it's basically a run instruction for Docker to run with the FC24 container that I created with this Docker file. And basically I am telling straight away to byte test which test to run. So, now in theory, if I... This is the testing framework. So, in theory, if I edit my bad test to include this breakpoint, then I should be able now to run my container and I should be able to hit that breakpoint. And, yeah, there you see. So, and you can... Because that actually doesn't... It actually... It actually hits the breakpoint when an exception is raised. So, it actually stops the execution whenever a trace bug is raised. Now, you might have no failing tests that you will still want to debug. So, okay. So, let me show you... I got another script here as well. And let me show you... I have defined on my comf test, I have defined a small plugin, a small custom plugin that is in the form of a mark. And I am using that on my good file. So, basically this mark is for including an option name on the byte test run that will look for any tests that are tagged with whatever I introduce on my byte test run. So, if I go here... Now, so if I run the good file now, this file in fact has... As you can see, I am passing the minus E option and the good tab that was defined on my marker for that test in particular. So, if I try to run this container, and I change the label to whatever, it will actually skip the test because it's not the label for good that we had defined on that test. Now, additionally, we can see with the trace config, this is also just running byte test with the trace config. On the trace config, you will see all the planes that have been registered by byte test, as well as the comf tests that have been loaded by... So, if I do this, you can see... In this case, it's showing me that it has loaded the comf test from the root directory, which is on the container itself because I am routing my volumes... I am linking my volumes to the main root folder on my container. So, that's pretty much it. So, yeah, that's pretty much it. Now, if you have any questions, I'm afraid we don't have any more time. Do we? Okay, yeah, sure. Sorry, man. For this container in particular? Uh-huh, okay. Yeah, sure. So, this is the one for the Jenkins instance that it's actually picking up the base image from Docker Hub, this is the Jenkins latest, and I include some other instructions for what I needed at the time. Then you get the Jenkins slave Docker file, which is just the Java container, base container, and there I am downloading the Java agent from Jenkins to Jenkins for executing the file then. So, you can find this on GitHub. There, you can find the Docker files there. You had entry points and stuff. Where are they? Ah, well, for this talk in particular, I didn't use any entry points or CMD, but it's like that, it's just like you can only use CMD once, and the entry point should be your default command to be executed with Docker. Hello, thank you for the presentation, it was very nice setup. Sorry, I cannot hear you. Thank you for the presentation, that was a nice setup. I have two questions. First of all, this looks like one of the benefits of this might be able to run multiple tests from Jenkins at the same time. Is this something that you've tried and it works properly with the setup? Yeah, you can run any test that you have on your testing framework if you already have it set up like that on your Jenkins instance. So this Jenkins instance was just queuing them because that's how Jenkins was set up? Yeah, this Jenkins instance was just to show the two different places like the Jenkins job that is running the byte test and my local byte test run that will be able to debug the test. Okay, cool. My next question, can you tell me what that shell you had, it showed whether the git was dirty or clean? It looked very nice, I liked it. It's all my set SH with some tweaks, yeah. Okay, thank you. Hello, thank you for the talk, it's very important. Well, my question is like this, when it comes in my understanding personally, when it comes to deploying your application, I can understand the benefits of using Docker. There are a lot of advantages like encapsulating all of the businesses inside your containers so you can ship it as is or make your application cloud native ready, something like stuff like this. But talking about testing only, what is the real advantage of using the containerized solution with respect to other available solutions that in my opinion might be more lightweight solutions like virtual environment? So have you considered this other solution? What is your opinion on this? As I said before, given the complexity of our testing framework, this was the best solution at the time. And as I said also, I don't want to preach or sweet-talk you into using Docker or containers for replicating your Jenkins instance, but this is actually saving us a lot of time for newcomers and also it's maintaining some sort of uniformity on our testing framework. Do we have any more questions? No. Thank you very much Gonzalo once more.