 Hi everyone, today we have a short overview of Jenkins development tools, we have a number of people on the call, so it will be a kind of interactive discussion, we are going to talk about plugin compatibility test, acceptance test harness, Jenkins test harness, and libraries and pipelines, we have an open source. So if anybody has any questions, just say something, so there is no need until the end, and yeah it's a general end at hoc session. So just to understand the level of knowledge, who needs a basic overview of what is Jenkins test harness and other such frameworks? It might help me, I've used all these things really, but I've never sort of seen them from a top down level, a lot of times it's been more like I've used them, something's broken, I've gone in and fixed it, but to be honest I can't say exactly what they're doing or what they're supposed to be doing, so that one would be good. Now let's, okay, then let's spend maybe 15 minutes on top level overview of these tools, and then we will go into details, just a second, I'll share my screen, okay, so do you see my screen? Yes. If I have nothing secret, except my playlist of course, okay. I don't see a screen, I'm going to reconnect, so I see an error message. Okay, let's wait for Kevin meanwhile, I'll just open the link. Ah, okay, much better. So Jenkins, we have a number of test frameworks, which we use internally in order to do all kinds of test automation. So one of the frameworks is Jenkins test harness, another one is acceptance test harness. So these are two main frameworks, we also have some additional tools like plug-in compatibility tester. So currently, Raul is the maintainer of this tool, it actually evolves pretty fast nowadays. And yeah, of course, we have a lot of internal stuff in order to run test automation on CI Jenkins.io. So CI Jenkins.io is a CI instance the Jenkins community has, so this is CI Jenkins.io. And in order to run test automation there, we have a pipeline library, it has JIN infpiplan library. So these are main things I'm going to talk about today, and yeah, let's just do a short overview. Do you want anything else to be added to the list? Okay, thank you very much. So let's start from here. So Jenkins test harness is a test framework we have. Actually it's completely based on JUnit and JUnit rules. But yeah, in this case when we talk about JUnit, it's not exactly about unit testing. I would say that Jenkins test harness is rather a functional testing framework. What it does, it offers particular rules, like for example, Jenkins rule. So this is a rule which you can use in order to bootstrap Jenkins within your unit test, for example, in a plugin, and then do some tests with it. Let's take one plugin, for example, Jenkins, okay, I'll just take our plugin, let's take my plugin, for example, hope it has tests now. So there is Jenkins rule, should be in test, yeah. So there is a number of tests we have here, and these tests are pretty recent. So when you see Jenkins rule, you know that it's JUnit 4 framework. Before that we had JUnit 3 could be based on Hudson test keys, which is pretty much the same, but for new tests you will see something like that. So here what you can see, there is Jenkins rule. On the test startup, it provisions Jenkins instance, in this case, it was my left plugin in the development mode. So all the changes you applied to your plugin get propagated, and it also automatically installs dependencies. So finally you get a fully running Jenkins instance, just by setting couple rule. And here there is an example of a test. So you may see that the only configuration he is pre-configuring, Jenkins location configuration is one of the components in my other plugin. So we do initial initialization, then we create project, and then what we do here, so we built and check this project, and then verify that something was done with the email. So it's pretty trivial, and yeah, this is really a functional test. Then with Jenkins test harness, you can also do a lot of additional things. So the framework is really big. There are special tools, for example, for round trip comparisons and for object comparisons. So this test, what it does, it submits configuration, and then it verifies that this configuration was actually applied. And yeah, there is a lot of such round trip tests, because they actually ensure that there is compatibility for all kinds of disk persistency, which is popular in Jenkins. So yeah, you may see that there is a lot of tests, and generally when you develop the stuff, you just use this rule, which is, yeah, actually it's an endpoint for the most of the tests. If you want to have something specific, actually there are more rules available in Jenkins test harness. For example, if you want to really restart Jenkins, so Jenkins test harness default rule doesn't allow restarting Jenkins. There is another rule called restartable Jenkins rule. It's developed in a bit different manner, because you need to really restart Jenkins, so you can just interact with the Jenkins object directly, and there is some logic there. But if you need, there is a rule for that. And if you want to do integration testing, sometimes just having a Jenkins rule isn't enough, because you also may need environment. And in order to do that, there are also frameworks included. But what we actually use nowadays, it's docker fixtures. So it's possible to provision some docker containers within your test. It's also a kind of rule. Yeah, probably you've heard about a project called test containers. So effectively, docker fixtures is a Jenkins implementation of these test containers, but it exists long before. So there are some attempts to refactor it to use the existing project, but that's what we have now. So it should be called docker, let's see, yeah, so here's an example, docker. So docker rule. So this docker rule actually provisions Java container. So you may see that what happens here that when the test starts, the rule gets initialized and you get the container provision, and then you can do something with this container to within your test. So let's see what happens here. Yeah, so there is a great ability test, I believe. So they get container, they initialize Jenkins agent with SSH launcher. It connects to this container and then starts operating. So this rule is really being used to provision containers. Currently, docker fixtures doesn't support multi-container environments. So there is no docker compose support. There is no Kubernetes support or whatever. But even having access to single containers is something really helpful. And as with Jenkins rule, every time you start the test, you get a clean environment because yeah, this rule is not only provisioning of the instance, it's also the provisioning of the instance. There are some pitfalls with that because even if Jenkins rule deprivisions the logic, there is no real way to clean up static variables. So it means that if your plugins or if your code base uses static caches, you may need to add additional clean up logic in order to get it working, or some data may be assisted between tests. It's something to keep in mind when you work with Jenkins rule. So it's still not a fully independent framework. We will talk about acceptance test harness later. But it's really helpful for development of short tests. And actually it has a lot of optimizations. So these tests execute pretty quickly. Okay. Jenkins test harness, any questions so far? No, maybe it's interesting that if you can talk about the local data, I think that's quite usual. Okay, local data. So local data is one of the ways to run the test with existing configuration. It's also an annotation. So let's go here, there is local data, and it's called XML file test. So what does local data does when you have Jenkins rule? If it finds a local data annotation, it goes to your resources. And if it finds a resource with proper name, it just loads this resource before the test starts. So in this test, we have whatever XML file, I believe this test has been created during last winter when there was migration to XML 1.1. So effectively, there is some code which we will try to find, but our test just asserts that Jenkins configuration contains whatever thing which probably comes from our configuration. So let's try to find this resource, find file, yeah, it's here. So there is a directory in resources, and we just have one small resource file which is being passed and there is a string which we check for the test. So effectively, we ensure that this configuration is being loaded in the test. So there is a lot of such smaller things in Jenkins Test Harness. If you want to find documentation for Jenkins Test Harness, you may have some difficulties with that because, yeah, this framework evolved sporadically. So on the documentation, you can only see the change log. We have some examples, but generally, if you want to develop something, usually you end up with going to existing plugins and core looking for examples and copying these examples. So, yeah, there is Viki page for unit test. Here you can find a lot of things like recite, local data. So the most common things are declared there, but in many cases, you still have to opt out and to just use examples. So I'll just put it here for a while. By the way, somebody has edited this page recently, so maybe there is more information nowadays. So regarding interesting topics, yeah, local data and preset data are more or less the same. Preset data is not much used, it's just configuration of our class. This plugin is optional installation of additional plugin on startup. By default, Jenkins Test Harness starts up only with dependencies, test dependencies and optional dependencies. So, for example, if you depend on a plugin optionally, you get it by default with Jenkins Test Harness. Sometimes it's not convenient, but that's how it behaves by default. Then there is without Jenkins. So this annotation allows to skip Jenkins initialization for particular test. So if you need to verify something in the class which doesn't need Jenkins runtime, you can use that. And there is also this timeout, which is just timeout for the test if you need something. So this is more or less it. If you want, I can show more particular examples for Jenkins Test Harness later. But I would propose to move to acceptance test harness. Yes? No, I agree. I think that's great introduction. Thanks. Okay, so if needed, we can set up more sessions to deep dive to particular frameworks because all of them are pretty big. So, acceptance test harness. Yeah, as you may see, it's another test framework based on Selenium and Docker. So this test framework actually implements everything we cannot do in Jenkins Test Harness. And primarily it includes web UI tests. So in Jenkins Test Harness, you can create a web client. A web client allows, for example, to verify web pages if you need, it allows to verify some REST API interactions, et cetera. So you can do some UI testing using Jenkins Test Harness. But if you want something real, for example, interaction with Web UI, then acceptance test harness is a way to go. It includes Selenium. So if you're familiar with Selenium development, everything is more or less the same. So in Jenkins Test Harness, actually, we have a number of tests. So this framework, by default, went to wrong repository at this. It's main Java. The most of the code is in test, of course. So here, what do we have? We have tests for plugins. We have tests for the core. So we also have self-tests. So here, for major things, we have some test automation. So you may see that for the core, there is actually not that many tests here. But it's still some. And it does some interactions. For example, we can take a look at create item test. So here, you may see that you want to verify that duplicate item names create errors. So you use Selenium in order to create a freestyle job. So there are common methods for basic steps. So you don't need to use low-level operations for everything. Then you look for new job fill in the form. You fill in some information. And then when you do that, you can also continue that, for example. Here, what we do, we created one job. Then we try to create another job with the same name. There are some checks. I don't really know what about. When we try to click a button, we expect that the submission fails. So yeah, this is just a Selenium framework. So you may see that there are some web elements which are direct access to Selenium and some top-level accesses. This is how it's implemented. And yeah, Selenium is also Jenkins acceptance test. Hardness is also powered by Docker. So if you need specific environment for some tests, you can get that. If we talk about plugins, one of the biggest problems of acceptance test harness is actually it's implemented as a test suite. So if you open plugins, you may discover that there is a lot of plugins here. So whomever creates a test for the plugin, then they create a separate test. They create a data model and then implement these tests. So it's actually good. And there are tests for critical plugins. So if you take a look at LDAP plugin, you may see that it also uses Docker. It uses a specific version of LDAP plugin. So it's also something missing in Jenkins test harness. In acceptance test harness, you can actually create tests for different versions of plugins if needed. And yeah, here we use Docker container in order to have LDAP running. And then we do some tests with LDAP plugin, I believe. So this is how it works. But here, everything is put in the same framework, accepting Blowshunt test. So Blowshunt test has its own test base. Actually, there was a project idea to detach this test and to allow keeping them in plugins. Ryan Campbell, one of Jenkins contributors, he was working on the framework for that a few years ago. And yeah, I believe there was a lot of enhancement done in this direction. But yeah, I'm not 100% sure it was finally released. So if you take a look at Blowshunt, you may see that there is a Blowshunt acceptance test. So yeah, Blowshunt guys decided that they have their own test framework. Well, it's reasonable because Blowshunt is largely about JavaScript, et cetera. So you may see that there is JS entries there. So they write some tests with JavaScript, which is not quite supported by common acceptance harness. So they have some other tools embedded. But there are still classic Java tests. And these Java classic tests, actually, they're also Jenkins test harness based. Yeah, so it uses the same framework. Yeah, if somebody is interested to make notes, please feel free to do that. Otherwise, I'll just do it after the recording so that we can focus on the call. Okay, so acceptance test harness is good. It still offers a lot of test coverage. The problem with that is that the framework is really heavy. So, for example, in Jenkins pipelines, we never run full acceptance test harness unless running before the LCS releases. And full acceptance test harness in Jenkins community has been executed by Oliver Gonze and Lucy, who are the maintainers of the LCS baseline now. So they execute these tests on their environments and just report back. As far as I know, we don't have acceptance as harness on CI Jenkins. Daniel, maybe you could correct me if I'm wrong. At least I'm not aware about that. Well, maybe we do actually. There is a test for acceptance test harness itself, but it only runs self-test. For Jenkins, maybe no. Okay, so with acceptance test harness, probably I should briefly switch to pipelines before we go to plug-in compatibility tester, because if we talk about acceptance test harness, we may want to understand how it's being used. And there are actually multiple ways to do that. One of the ways is to just run it as is. Then there is some packaging for Docker containers. So if you want to run it locally, now there is opportunity to do it in Docker as well. And there are some enhancement which have been done recently by Raul in order to make it available on bigger instances. So there is Pipeline Library. And Pipeline Library is actually the Pipeline Library being used on CI Jenkins Ion. So it includes several standard components, for example, build plug-in and also steps like run ETH, run PCT. So if you want to run ETH against your plug-in, there is a step for that. But generally what we use in Jenkins mostly is build plug-in. So if you take a look at each plug-in, you may see that there is Jenkins file in the root. And this Jenkins file usually contains something like that, build plug-in. So the rest happens in the library. So if you take a look at this build plug-in, actually you may discover that there is some parameterization. So if you want to test against particular GDK versions like GDK11, you can pass it here. You can also pass platforms like Linux and Windows by default we built on both. And yeah, there is a way to test against multiple Jenkins versions, et cetera. So this framework offers some parameterization. But actually what we do in this build plug-in step, we just run in multiple configurations. We do the build there, so it's here. So yeah, there is a lot of Maven options magic, but effectively it ends up with run Maven. This framework also supports Gradle. There are some plugins in Jenkins being managed by Gradle framework. I won't be talking about it today, but just for your information, it exists. Okay, and after tests are executed, we just do some archiving. So for example, Maven is being published. Findbox is being published. If ChequeSale is enabled, it's also being published. And we also published artifacts. So for each plugin built, we publish HPI files, JPAs, Java doc, et cetera. So for each build, this information is produced. So if you take a look at this SSH Slaves plugin, there is a green kick. So let's go there. So this is an execution of this build plug-in. So you may see it has been executed on Linux and Windows. Sorry for the Russian web interface. Because CI Jenkins iOS still doesn't install local plugin. Just a second, I'll open it in Chrome. If you have questions, just please ask them, because I do not see the chat. Okay, so now it's in English. Yeah, here you may see tests, you may see artifacts published. Yeah, and there is pipeline. So there are multiple stages, but pipeline doesn't display nested stages here. So we are fine. If you need more details, you still go to the classic web interface. Usually you have to do that when something fails. But yeah, there is information for that. And actually we have pretty much the same for Jenkins Core. I will return to that. So yeah, let's go back a bit to the pipeline library. I also have one step which I haven't discussed before is publish incrementals. So this spring we introduced the incrementals support. It's a kind of continuous delivery engine for plugins. So some plugins like Artifact Manager 4S3, they support incrementals. So on each release, we automatically publish the build. Okay, Artifact Manager S3. So let's take a look. So there is last pull request from JC. And we may see that there is incrementals deployment. So we can go here and just download the version. It's in Maven. And you can do the same actually for pull requests. And there are tools like Docker, Pegager, et cetera. We should already support these versions. So you can work with these versions directly in continuous delivery mode if you want. So it's integrated into build plugin. What else? Yes? I've got one question, Oleg. In your plugin, do you need to specify some dependency or is it gathered by the parent pump? To be able to use this library. You mean incrementals or pipeline library? No, the pipeline library. Pipeline library works for a plugin which inherits from recent plugin pump versions. So there is a plugin pump. It's a kind of universal plugin definition for Jenkins plugins. And it includes components like find bugs, check style, et cetera. So common definitions come from here. If you extend it, you get publishing. Okay. Is that what you were asking? Yeah, yeah, yeah. That's it. Yeah. So this build plugin is fine. There is ongoing, not that ongoing, but I hope to eventually get it over the fence. Also built Jenkins components because we don't have a standard framework for building libraries like remoting, step or whatever in Jenkins right now. It's still working progress, but I hope I will get it over the fence because we really need it for Java 11 work. So it's here. What else do we have in this library? So run ETH, run PCT, so it's pretty much the same. So history of these components. Last spring, we were working together with Raul on some automation for Jenkins Evergreen, which was called Jenkins Essentials at this point. And there is Essentials Test in Brune. So it's a kind of framework which allows doing end-to-end integration testing for continuous delivery. And it includes running ETH and PCT using this framework. So it's managed by YAML files. We've used it for several plugins nowadays, as far as I know. And yeah, there are steps like run ETH and run PCT here. So you can build it with custom war files if needed, with custom versions. And actually, this is just the steps which simplify using all these components. And there is some documentation here, by the way. So, yeah, if you want to see how it's used, you just go to this pipeline library and you see examples. So here, for example, in vacation of ETH, the configuration is passed in YAML. And you can see which revision of acceptance that's hard enough to use, which Jenkins version you want to use, how you approach that, which browsers do test, which tests and categories you want to execute. So, effectively, this step saves a lot of time if you just want to run particular tests in your pipeline. And that's what we actually do in Jenkins Core. So let me open Jenkins CI. Yeah, so Jenkins itself, it also has Jenkins file, which defines the build process. You may see that we actually do a lot of the stuff internally, which still needs to be refactored a bit. But what we have here, that there is run ETH step included into the standard build flow. And actually, we don't run everything. We run only particular components. So you may see here that there is a file called Essentials YAML. And let's take a look at this file. I believe it's in the root. It's in the root, but I have missed that. So you may see that here we only run a category of smoke tests. So it means that we do not run the entire test as you need because it takes about 24 hours or so on a single machine. We just run smoke tests. They take approximately 15 minutes, which is quite convenient for us. There is p3vision, so we know what we test with. And it's actually a part of Jenkins build pipeline. So we can go to the last release. And here what we have for Jenkins core. So we have run on Linux. We run on Windows with JDK8. We also run acceptance test harness in parallel. So there is some execution. Currently, we work on integrating Java 11 testing there. So there is a pull request, which is almost ready to go, I hope. Update to JDK to new Jenkins test harness and enable testing with Java 11. So once it's integrated, we will also have Java 11 testing as a part of our default test suite for weekly releases. So it's here, it's passing exactly the same graph, but also with Java 11. Okay. So going back to pipeline library. Effectively, this pipeline library is pretty convenient. We spent some time this spring on refactoring it. So now there are common steps in infrogruvia. So for example, if you want to run with Maven, you do not need to invent everything on your own because there is a step. Then there is some specifics because we run in Asia. We want to use mirroring instead of loading the world. So yeah, there is some settings files which we pass. We still have support of JDK7 if needed on our instance. And yeah, this is a framework which simplifies definition of steps a lot if you develop a CAE for Jenkins IO. Okay. Any questions? Nope. Yeah, I've seen something from Wadiq in the list, but I'm not sure whether it's actual. Yeah, it was for the softness. Okay. Yeah, Baptiste mentioned that Evergreen also uses Essentials YAML in order to define all these things. So yeah, this framework was originally created for Jenkins Evergreen, but generally there is no restriction. You can use it in other companies if you need. And yeah, I believe we will have to update it a bit for Java 11 support. There are some scheduled tasks in Jenkins JIRA so that we get support of Java 11 testing with all these components because having acceptance test harness and PCT is really critical. So speaking of plugin compatibility tester, should we press it there? Yes. Okay. So this plugin compatibility tester, actually this is a utility which allows to cross verify particular plugins. So Jenkins acceptance test harness, sorry, Jenkins test harness, acceptance test harness, they work with particular questions. But you cannot create compatibility metrics and it's still hard to cross test multiple development versions of plugins. So idea of plugin compatibility tester, that it actually allows doing that and it also creates a kind of reports. So this framework we also invested time into it last spring and now it has some documentation and also has Docker support. So if I have good internet, probably I could just launch that. Let's hope. I just had a meetup yesterday so there is a demo of something. Okay. So what do we have here? Okay. Let's just try to launch that. Okay. It's a good start. So unless I run out of disk space, maybe we should get it running soon. Meanwhile, I'll show you what it does. So this is a framework which can be used in Docker mode or not in Docker mode. It includes CLI interface. So generally it's a command which allows to do some testing. So you take PCT executable. You say which plugins you want to include into testing. You say what to test. So by default it runs with Jenkins, what versions defined for plugins. But you can also specify custom Word file if you want. And it supports Word files used by custom Word packages if you need some self-configuration. And then what it does here, it can build these components from local repositories. So it can check out the plugin sources and build them on demand. And finally it just produces a report. It may be the XML, but when you specify XML here, actually it also produces HTML. And yeah, this is something useful. And we invested into PCT for a while because last year we were working on JEP 200. It's a cluster disk realization thing. And we needed to test a lot of plugins. So we used PCT in order to run this test and to build compatibility matrixes. I'm not sure I still have reports for that. Maybe I do. So yeah, as you may see, it's still doing something. If I'm lucky, let's just take a look at my directory. I'm usually not that good at cleaning up old stuff. So yeah, most likely I have something for JEP 200. I do. So we have out. So this is actually PCT CLI. We were testing with Jenkins LTS release and with pre-release versions before. There are also some automation scripts, but actually it's something like that. So it just tests with a particular plugin and we were running through a list of plugins which we suspected to be impacted by JEP 200. And here we have a PCT report. So let's see what's inside. Okay, it's here. So this is a report produced by plugin compatibility tester. So you may see that I was testing with different versions and for each plugin we have some reports. So what PCT does, it takes your source codes or plugins. It modifies to run with a recent Jenkins core and it modifies plugin pumps, et cetera, if needed. And then it just executes the common build and common test. So it runs Jenkins test harness tests, included into your plugin definitions, but it runs with custom versions. So we were talking about SSH slaves. You may see that SSH slaves was compatible with JEP 200 at the very beginning. So you can open a report here and you can see how it looks like. There is another unrelated issue, but generally it passes the build. All tests run and it runs with JEP 200. For some cases we may see that there were failures. So here you may see that there is a build. So it even failed to check out the plugin because there is some expectations from PCT of how plugins should look like. So for example, if you require testing with a plugin, the plugin should be pointing to HTTPS GitHub in its POM definition. Otherwise it will fail. Finally I'll show ownership plugin because it was one of the failing plugins. So yeah, there is a patch by Devin. Before this patch actually PCT was failing against this plugin because it was defining SSH here and PCT was unable to check out the source code and modify them. So there is some issues there, but the interesting thing is JEP 200. So let's try to find something. This report fails not due to JEP 200 but due to parent POM updates. We needed to do a lot of such updates. I may have difficulties with finding an example because when we were fixing the JEP 200 issue, we were actually rerunning the build, but maybe I'll be able to find something. Class filter. Unfortunately there is no search supported so PCT generates a pretty primitive report. It's still useful but you cannot do a lot with it. Okay, so here's a JEP 200 issue. We were able to discover issues in this plugin by running PCT. And this is pretty helpful because you can just run against arbitrary Jenkins core version and see what happens. So let's take a look at our... Okay. Probably I didn't pass maybe cache, so likely it will take a while. Okay. We can take a look how to automate that. So you see this manual report but actually there is an interest to have it generated out of the box. And that's why we have pipeline library step here. Okay. So this is pipeline library. And in pipeline library we have run PCT step and in this run PCT step, it's automation created by Raul. And this automation does some definitions. So you can also... So you may see that it's also driven by metadata file. And here's example of this file. So for example here we test this particular Jenkins version and we test this credentials plugin. Most probably there is a type because it should be lower case but double check later. And yeah, you can pass particular PCT URL, particular revision, et cetera. For PCT nowadays we ship official Docker containers. So if you just want to run a PCT on your machine, there is Docker file, there is automatic deployment and it's better to do that. But if you want to take a specific version, for example unreleased fix for plugin compatibility tester, like whatever support of library snapshots, I'm not sure what was that. But in such case you still can pass these options to your pipeline and you can get it running. And actually run PCT is used in some components. So let's just try to find them in Jenkins. But I believe that Evergreen uses it, right? Let's see. Yes, I know. It's not definite because it may be invoking Essentials test nowadays. Okay, maybe no. Okay, let's try to find another example. Yeah, so you may see that Git plugin does it now. So it's one of the plugins which was updated to Essentials test flow. And here we run PCT. There is also metadata file. So there is Essentials YAML. And here we may see that actually in PCT we just run Git plugin itself. So we run PCT only for Git plugin here. But if we take a look at the code base, why do we do it at all? So you may see that it also gets Essentials sources from other places. So maybe it actually does more than just Git plugins. Oh yeah, there is local plugins PCT. So like there is some logic behind that for generating a bigger list of plugins. So what we can do, we can just go to our pipeline and see what's going on there. Okay, so Linux. Let's try to find PCT here. Okay, it may be challenging. And okay. So probably I'll need to review this repository, but generally there should be PCT running some way here in this list or not. So maybe not. But if you want to do that, yeah, there are steps for that. So when we talk about Java 11 compatibility, we need to think about post acceptance test harness and plugin compatibility tester, because they allow us to significantly increase test coverage, especially for plugin compatibility. So what we have already done for Jenkins, we have Jenkins test harness passing. So we know that some plugins which are being used in Jenkins test harness are more or less compatible. But the most of integration testing nowadays comes from acceptance test harness and PCT. So we will need to update in order to get this test running on our environments. SRC test, Java, yeah. So I would be happy to get all these tests running with Java 11 and to see what we are missing there. And pretty much the same with plugin compatibility tester if we can run automatically compatibility testing against maybe 100 most popular plugins. It would also give us a lot of confidence when we release Java 11 support in weekly releases. Okay, this is what I wanted to say. Do you have any questions? I have one question about ATH in general. I mean, first, I think it's been great and it helps me a lot. One thing I've seen is there seems to be some flakiness with the ATH, so I'm wondering, you know, especially when we want to see if Java 11 is working correctly. How consistent ATH runs have been and how long does it take to do a full ATH build? So last time I tried running full ATH in a single machine. It was something like almost one day. There were some experiments with ATH realization. And a run ATH step actually supports that internally. So in a run ATH you can realize it across multiple machines. But it still takes a while if you want to run the test you need and there is no automatic realization logic in the Jynx pipeline library so far. Regarding flakiness of ATH, so Jynx is a web service. It includes a lot of various logic, including dynamic logic with JavaScript and whatever. And actually we use ATH specifically to verify this logic. And yes, some tests are flaky, some tests rely on magic numbers like timeouts, etc. So ATH is not exactly famous for being stable, but it still gives us a lot of information. So right now the builds in ATH are broken at all. I'm not sure whether it's infrastructure defect or not. So yeah, you may see that there is some parallelization split. It's parallelization of ATH internally. I should probably switch to Chrome again. That's fine. I'll talk to our infrared team to install the local plugin. Fine. So you may see that some tests fail. Take a look why they fail. That's the fork booter problem, the Maven 3.5.4 problem I think. Oh yeah. Nothing actually ran. Yeah, so definitely something needs to be done around ATH to get it stabilized because your current state is not that good. But for particular tests you still have options. For example, when we were testing against security 200, we know about particular message patterns we are looking for. So when I was presenting this report, I was looking just for class filter. So when we were developing JEP 200, we had two types of messages, and all of them redirect to this support link. So it means that we can just search across the entire test report and to find failures which are printed this warning. And then we know that it's something caused by JEP 200. Even if not the entire test is green, it gives us a lot of information. And I believe that's something you should try to do for Java 11 as well, though it will be a bit more complicated there. Okay. Okay, that's good, thanks. Yeah, any other questions? No, not for me. Okay, I hope this overview was helpful. There is of course more details about what is included in these frameworks. We also have a number of internal libraries. We have a number of internal experiments like find bugs detectors or whatever. So Jenkins has a lot more test tools than I presented today, but at least these ones are the main ones within the Jenkins project. And yeah, it's something to start from. And particular companies like Evergreen, they have their own test suites. So maybe we could have a separate techie about that at some point. And yeah, for script testing, there are separate frameworks like SHUnitBuds and some repositories. So there is much more than Java being used in Jenkins nowadays. But for Jenkins, it's the main source. Now I need to copy paste that. I will still add some details and then post it somewhere. Oh, okay. So yeah, there will be a wiki page, I believe. Sure, okay. Okay, if there is no other questions, I will probably stop the broadcast and we can sync up a bit more if you want. Okay, no, that's great. Thanks a lot. We really appreciate it. Yeah, thanks to everybody who was watching that. I'm stopping the broadcast.