 Welcome everyone. This is Jenkins LTS certification overview. Oliver, thank you very much for presenting to us. We so much appreciate this. Yeah, my pleasure. Would you like me to take this away. Yeah, if you'd be willing, I can, if there's something you'd like to screen share, we can certainly have you screen share if rather you would prefer that we focus on the notes I'm happy to screen share the notes. You tell us what what you think would be best for conveying the information. Right. Yeah, I'll definitely try. I see that you put together a quite nice list so with you guys I see that many of you have joined in the meantime so hi everyone. So, if somebody will get some questions or remarks I'll be happy to be interrupted at any moment and answer those, except for that I would try to give you as much of a concede rumbling as I possibly can. So, the LTS certification just just just very briefly. Our LTS are moving in four week cycles. We get two weeks for back porting we get two weeks for for testing and verification. Actually, this is documented in full length. So, let me attach. Let me attach a dog where this is where this is documented. So what we're going to focus on is specifically how this is certified. Since I am the single person that does the back porting perhaps will be interesting to cover in the end if we get some some extra time, because there's definitely knowledge that needs to be transferred for whoever is going to be, you know, doing this after no longer the release officer. Okay, so when it comes to certification, we are sort of following the Linux kernel model. So, the release candidate is being being made available to everyone, and we got this mailing list request for people, people feedback and reporting issues mark you very well known, because you've been contributing to this for years and years. So we, you know, just inviting pretty much everyone and I'm pretty sure that cloud we have been working on this very hard to provide us with the feedback and actually a lot of a lot of problems was captured this way by some of your colleagues so this is the way how we're, you know, receiving these from from the upstream. And at the same time, for instance, for us we've been running our internal verification based on a th which you know we also contributed to these results same as same as we encourage people to do. So this is what we've been, what we've been doing. And as I said, I will get back to the, you know, okay so one other thing is that the release officer role was not only just doing this backporting and orchestrating this feedback collection, but you know, keeping all this all this space going right since, you know, scheduling all these events in calendar, making sure that things happen happen at a particular time, initiating new new LTS cycles, etc. So again, that would I can also cover that in case that there was an interest. Right. So, Mark, I'm going to use your notes to keep me on track. So where does the certification process run actually multiple places, due to the distributed nature, nature of what we're doing and how we're doing it. It's sort of hard to get a complete picture. So we just, you know, rely on on on the people to contribute their own results. In the past, I was looking for some more structured way of how to do this in the past, because of the key page and it didn't work very well. In my opinion, and I haven't found a better, better tool to actually get everybody who is interested and who's got some interesting results to report them back. So we stick with a good old email. So the certification runs in both our CI Jenkins IO instance. So that's where we're running the full 80h so it against against the LTS. So the certification is actually a full blown CI. So even during back porting, we're just running the same same verification testing so it got an early back early feedback. Even before we cut the release candidate, etc. So this is the one place, you know, the advantage of doing this is that everybody can see the results can investigate a failure scan, you know, spot the problems, etc. There was a couple of ideas in the past to actually move this into one of our, maybe Red Hat, maybe maybe Clobis or even some somewhere else and to do a dumb stream. Though, you know, the obvious obstacle would be that this is sort of hidden and, you know, not everybody got access to the results to possibility to investigate and work on fixing the issues, which was one of the most tedious and time consuming part on on maintaining the acceptance of harness. So I felt really, really strongly keeping this in the upstream. So we can, you know, lower the lower the barrier for entry into, you know, getting new contributors to actually fixing and updating the test, because that was a real, real struggle. So, yeah, that would be the upstream environment where it runs. And except for that there is a, I guess, a number of internal environments. I know that the mark in order they're doing some, some intensive exploratory testing on these, which is very useful. Except for that. And you know, there's some other environments I presume in Clobis that I don't have that much information on. So speaking on the part that we've been doing in Red Hat, we've been testing this exclusively on OpenStack, both on the host on, with actually using REL7 for this, or, you know, previous versions before that, of course. So it was, you know, tested either on VM or into the container, the ATH container, not the Jenkins container. So those were the two environments that we've been covering and testing this and, you know, sort of comparing the results with what we've seen in the upstream, what we've seen internally, etc. Because, you know, due to the complexity of the whole test suite and the fragility of the approach of using the Selenium UI testing, those of you who play with that know what I'm talking about. So there was quite a lot of false positives and problems that wasn't really related to any bugs or anything. So it was truly necessary to compare it between the environments and see which of the failures are real or which of them are not. So yeah, those are the environments that I'm personally aware of. So Mark, you're on mute. Yeah, so you said virtual machines but orchestrated by OpenStack and then also containerized, again orchestrated by OpenStack, those are the two environments you were checking? Well, the OpenStack was not that much involved in that simply because ATH does not interact with it in any way. So it just happens to be Jenkins agents running in the OpenStack. So the point is that it was the VMs with the actual OS, right? Got it. Okay, so it's not in the relevant thing there is, the virtual machines there is to say this is an operating system, not just a Docker container insulating something from it's running directly on Red Hat Enterprise Linux 7 in that case. Okay, yeah, this is what we've been doing historically even upstream we've been doing this before we moved to CI Jenkins IO, so maybe four or five years back actually. But you know, we find out that the results are a lot more reliable on virtual machines than we ever get in the containerized world. Okay, so maybe maybe that's a dumb question and it's getting late here. So maybe I didn't just follow Clark correctly. Are you saying that what you what you're using the VMs and OpenStack stack is something you're still doing nowadays or was the past things before you migrated to CI Jenkins IO? Yeah, sorry, I might have misspoke. In internally we've been using both approaches to running it on VM and actually running it on VM and inside ATH container. And what we have verified is that the pure VM approach, you know delivered more stable and predictable results compared to the containerized, right? The remark on the CI Jenkins IO was really a side note that even there some years ago we've been using this pure virtual machines before we moved to containers and we also, you know, observe instability in there. Okay. Thanks. So, so about this question, are you still running I think I hear that yes you are still running. Is that likely to continue or does is Red Hat likely to say hey we're going to stop running those tests internally. So we will have effectively as a community lost one resource that could have found problems for us. Very good question. I mean, one of the reasons why I'm giving up on a release officer position is that I just cannot predict my allocation. So I would definitely love to stay in loop and continue to be one of the people contributing the results find, you know, found the found regression and possible problems. So I would definitely love to be that but you know, I prefer not to promise you something I won't be able to deliver. So, so, so in the future, Red Hat may reallocate and say hey we're not going to run those tests Jenkins project therefore I think is is well advised to consider how do we, how do we execute these somewhere publicly so that we assure we see them. Yeah, I mean, when it comes to allocation I would say that definitely this is a lot more expensive in terms of human time rather than a machine time right somebody needs to be looking at it keeping the age and it would shape prone of some infrastructure issues. Some random glitches incompatibilities between page objects and plug-in versions so it's a lot more expensive on that side, and I honestly cannot quite imagine that I would say no no more resources for running these tests. I mean, thank you. Thank you. Good question though. Right. So folks, once again I would like to encourage you guys, if you have got any questions just feel free to shoot. Yes, sorry. So regarding this open stack and red hat and so on. So it's still something that we cannot see, even though, I mean, we probably we don't interact with, but we cannot see as a read only access or anything like that. That's that's the problem with this, you know, downstream internal internal testing I was James who was, you know, doing something quite similar inside inside cloud base. And as I said, I've got pretty much zero information except for the fact that he sometimes reported that he found a problem so this was one of one of the issues and one of the reason I so I mean, I'm a huge advocate for having this, you know, I have a reference environment in the upstream where the full test suite is actually being run and provides the results that are visible for everyone. Thanks. Yeah, I'm just I'm wondering actually so right now those days, you know, do as far as I remember in the past it's very vague now but I think you were looking at the open source. I know the results of it yet, but then with your expertise you would actually know yourself what are the tests that are failing that should be considered and the one that should be ignored. So we would, I mean, we would always as far as I remember have a th failing, but you know, because of your expertise you would know which tests should be concerned and which tests actually should not be one. Kind of still the situation right now or I mean, maybe I'm not clear. Right. Let me see if I understand that so you're suggesting that at age is fragile in particular tests and only the people around it has an inside what is a symptom of such problems and which tests are more fragile than the other. Somehow, I mean, and that's my bad because I should have prepared this call a bit more but I remember that back in the days, most of the time the acceptance as harness executions would basically be red all the time. Like never be green really because actually many tests failing and you would be one of the rare people knowing which tests should be actual concerns and which ones are known to be failing all the time be flaky whatever. So not really be a concern of our other ever, which I think might be if this is the case, which I'm looking right now at if it's okay right now might become more and more of a concern because indeed we may lose that you know, expertise that's only in your brain in your in your expertise to know which tests should be looked into and the test should be ignored because they've been flaky forever. That's a good point. In a sense you're right. We never after easily five or six years of effort to actually get the ATH stabilized which I blame for lack of time and inherent nature of selenium. We are trying to verify something like I don't know I'm pulling numbers up from my sleeve but somewhere for the independently developed components, which you know maintainers not really know how it's being tested and what particular effect it's going to have. So we usually just find out as somebody had released a new version of plug in the changing UI a little bit. It started failing and it's sort of sort of tedious in this way. But yeah, so this is this is the part of what I what I mentioned that it requires pretty much constant attention and adjustment to the plug in and to their UI send to whatever specifics. So it very often happens that sometimes starts failing simply because something have changed. Usually it's UI but there might be different things that do change and needs to be adjusted. So it's not quite that we would know that I don't know some small fracture of test would be just constantly broken or unfixable or just fragile. We have a ways to deal with that, you know, actually fixing them, but it requires the time it requires somebody to have a look at that. And these tests are, you know, getting broken, you know, annoyingly often so to speak. Maybe I would make sense actually, you know, you think it would be a good idea maybe especially for people who may not be aware of what's going on or how it works to share what we're talking about on the screen because I actually opened it just now. So I guess it might be useful especially for people like are they recording more than us in the end maybe you mean the results. Yeah, just this is just what we're talking about right now. And so if we look at the situation right now. Yeah, we see indeed that we don't really open or ever have had. And again, I mean I hope it doesn't sound like, like a reproach because that's not the case at all I absolutely understand that we all have limited time and everything so just to, you know, kind of make a checkpoint on this. Yeah, so and when I was looking at the thing right now just a few seconds ago and that's what I was referring to indeed the number of tests failing and I assume that all your nose which ones you know have been failing forever which ones may be new and so which one are probably to be looked at right. Yeah, I mean, 100% get your point. It's fair to say that it's rarely rarely rarely ever see to have these 100% successful. And, you know, let's not lie to ourselves. So yeah, you're right. As I said, the thing is that, you know, like, you see a lot of like, let's say job DSL plugin tests are failing without examining closely presumably this is because of a single problem that has, you know, broke at some point in time. And, you know, needs to be looked at adjusted the test or the plug in but majority of the cases is this test. So it gets fixed etc. So it's not like that this particular test will be fragile or constantly broken and it's not fixable, but it just require. Yeah, sure. Somebody to actually adjust the test so it matches the code base in in these plugins. And it's sort of a, you know, constant struggle because of the way how this is architecture, we're trying to verify the let's say entire ecosystem to verify that the experience of a user at install Jenkins with all these plugins is actually going to have something functional. So in the end we're verifying quite a lot of components which, you know, I would have put it was not integrated before right nobody has done any any checking and verification on that so there is a various reasons why this can break most, you know, prominently because they're independently developed and released by individual maintenance. Right. And so maybe interesting question to ask to you, you know, if you had more time. If you had more time if you had like, you know, multiple people or people available or whatever. I would advise people to focus on to improve the process in general, maybe improve the process delivery. You know, maybe the test aspects or maybe focus on the test aspect first and maybe I was thinking maybe at the end of the meeting or something we could ask you whether you feel the whole process what you what would you keep what would you actually change you know accelerate whatever you know, I guess you have a lot of ideas or a lot of your question. Actually I put the put a paragraph close to the end of the agenda. Future of LTF testing where I got a couple of suggestions. And I guess we're thinking quite along the same lines in the end so would we like to, you know, operate on that or would like to keep it on the end. I'm, I'm great with going there I think if there's interest there let's go there. I do want to, I will beg for the privilege to come back to ask specifically about tables to divs. But I think Baptiste's question is a good one let's go there. And then, then I'll make sure that my questions I can come back. If I to realize I'm trying or so to kind of buy it by the request of Oliver which is to ask questions to go. No, that's perfectly fine I would actually suggest that we would go through the through the rest and then we will make sure that we can have a look at this so we all have as much information about the LTS and then we can, you know, potentially brainstorm and spend the time on discussing the future. Regarding the ATH, if I recall, James, I actually saw as well the commit so it seems that he's now somehow leading the project, the ATH one way, right? That's a very good question. I have announced my intention stop doing that some two weeks ago or so. And for that, we've got two people that from cloud business was helping me on this it was. I guess we even have a team within that. So, but they probably. Antonio maybe. Antonio James, Raul maybe. Let me see I'm sorry I'm terrible with names and I'm afraid to get them all wrong. For me and for Victor actually for some reason. I'm sorry come again. Yeah, I was just kind of making a half private joke because actually Victor knows those people because he worked with them so the name I'm mentioning he reminds them. Right it was it was Antonio. And at the same time as they're all with with one girl about sorry I Beatrice. I don't think so. Anyway, there should be a team of maintainers. I was the most active ones but I know that Antonio has stepped in and pretty much as everywhere else in components in Jenkins we haven't been very rigid or something like that. I remember of people that have to release permissions and been doing the releases. I remember that a couple of people from Colby's got to release permissions. I guess as you said James was doing multiple definitely merging not sure about about releasing. Yeah, actually hit it. So a couple of other people from it has should have the release permissions at the same time. So for what makes sense to me is to have a couple of people and that's what I try to achieve and to large extent I failed on that to have a group of people that would actually be you know, as pretty much in any component, you know, permitted to merge and release the ATH is happening but it's, you know, fairly informal at this point. So, and Oliver you use the term release. Can you help me understand what does the term mean to release the acceptance test harness. Is it actually a version test suite. And so we go backwards and forwards with version numbers or could you give me an overview there of the concept. Right. Another good question. Well, from that standpoint the ATH get two roles. It contains the test that actually verify the, you know, the product. But at the same time is a framework from composing similar tests that can be in other components so you can use ATH is a dependency and to use the framework and the page objects to actually write your own tests. We actually use this in a couple of other plugins. We pull it and use the same mechanisms, it permits you to do the UI testing in the actual the browsers and a couple of other, a couple of our features. I guess I can point you to some, when this was really priceless. Actually, I don't know that you need to point that I had forgotten completely about the fact that yes I can depend on a version of ATH and use it inside Jenkins core or inside a specific plugin. So that thanks that clarified why you do a release. Thanks very much. And what I can disclose is that actually we do that also at CloudBees to depend on this framework to add more tests that are on the proprietary side of CloudBees. Yeah. Right, so well, I guess it will be for a separate topic how to actually use this in the plugin I spent quite a lot of time putting it in a better shape some year or two ago. But yeah, that will be for some documentation effort or something like that. Thank you. Yeah, thanks for the question. Just covering the rest there is can LTS certification be run on a weekly release. Not only it can but at the same time it is when we're running the acceptance to harness if I'm not mistaken, it's CI is actually running against weekly. I think we run smoke tests, like the very recent set of not really like exactly because smoke test meaning the one that are like I've been qualified as really non flaky. But it's, I mean, people should be kind of aware that you know, the set is very, very limited on purpose because we don't want to be ever flaky. Yeah, probably we should actually consider, you know, fixing the flaking what is the rest and growing the street because in the end it makes the weeklies plus the LTS is stabler over time. It's a never. Right. Yeah, that's that's true. As I look at it, I see it run over thousands of tests which I believe it feels like the entire test suite. Yeah, I recall that except for the flakiness there was also another problem and that was the resource utilization of CI Jenkins IO, because there was quite a lot of this now we tested everything twice for Java 8 and Java 11. And there was this, you know, certification of LTS and this sort of CI that actually use the latest course, etc. So, yeah, I remember that in front of was not very happy about how much resources this is using because the entire test suite takes quite a lot of time to run. And at times it's picky about resources but you know not not using it very, very efficiently because this is UI testing. So, there was another, another consideration there, how to actually do this resource wise and I remember that, as you said, there was a push to use only a smoke test for this. But apparently it's not set this way in the moment but definitely something to keep in mind. I mean, you're thinking but so when I look at the execution time of the master branch on CI Jenkins that IO for Jenkins core. It says there that it is in fact running a running a th but it doesn't take nearly the eight or 10 hours that it takes to run a th elsewhere so what if I misunderstood. Right. Yeah, so you can hear it running. I just opened I don't know if it's my screen. This is the Jenkins core my star Jenkins file. This is calling the essential dot the ammo file which is here and then it's actually running a category the smoke this category. This is why it's so such. Yeah, that's that's a good point so but it's the market talking about the same thing right so this is the Jenkins repository CI which is actually only running the smoke test in stable branches for LTS. I actually replace the smoke test category with every test so on the stable branches after backport we're running the entire test suite. But what I was referring to was the CI for the acceptance as harness itself. We should talk about this as CI for for a Jenkins core. So, let me just give you the link. And that's this thing. So yeah it's a bit. Contra intuitive. But in fact we are verifying the latest ATH against the latest core, but it's running, you know, not as a part of a Jenkins core CI, partially because it would be too fragile and pretty much every, every change would be flagged as broken. On the top of my head how the plugin versions are set, and if they are even set. How does it work from ATH that does it like use or whatever latest version of every single plugins or how is this been the normal how it works. Right. There's a couple of user mode let me try to wipe the dog. There was a lot of discussion about this and in the end we come up with some set of use cases. We have this because recognize use cases for ATH. Let me feed it into the into the chat of the zoom meeting. So you've got a couple of individual use cases and all of them require something else. Presuming that you use this for testing your your plugin you definitely want these tests to run with the dependency versions as they are hard coded in your palm. So obviously you want this to be pinned, but it's not a requirement from from the ATH standpoint. Obviously when we are verifying LTS, we sort of want latest versions to be used from the upstream. Imagine that we release a new LTS version today. We obviously want it work for, you know, for the users or for the customers with whatever plugins version there are. Right. It doesn't, it doesn't make much sense to test it with the last year's versions. I mean it would give us more stable and predictable tests. No question about that, but the value that we deliver to our users would be would be drastically reduced. So, again, this was a subject of quite a lot of debates, the, you know, usefulness and value delivered versus the stability and maintenance cost. Am I making sense? Absolutely. Absolutely. I'm just thinking that, you know, we all know how hard it is to have things, you know, break suddenly from one to another for an unrelated reason, you know. And I'm wondering whether we could have something in the middle, which is like, you know, a Power Mixer or something that would pin every single version that we are using in ATH, but then enable Dependable to have a controlled and constant flow of updates that then would be vetted by PRs, you know, and we would detect which PR, which plugin is actually making, breaking on ATH instead of bumping and then all of a sudden because we've got 10 releases of plugins. We see 55 faders and then 100 and so on. And in the end of the world and, yeah. Yeah, I would have to give it a, you know, more of a thought to see how this can possibly work. But, you know, I mean, I can definitely describe the current state when we just grabbed from the update center what is available for a given engine conversion, either the latest or the latest LTS we test. So, yeah, I mean, that's, I guess, something to think about. You know, I see some issues of how to actually implement that, partially because, you know, there's a huge number of the plugins that are involved. There's also a lot of plugins that are being installed, but not necessarily tested, right. You know, it's like, I don't know, what is JDK Tool Installer. I assume that there are no tests for it, but it's one of the, one of one of these bundled or dependency plugins. So it would get there anyway. So, yeah, I sort of see it somewhat hard to implement how to actually get this get this pinned. But yeah, let's, you know, I guess that's definitely something that can be considered as a next step for ATH evolution. Okay, and just to finish my thought, the reason I share this contributing guide there is this list of recognized use cases. So oftentimes when we get these discussions. You know, people tend to forget what are the other use cases for the entire ATH and, you know, try to, you know, make things be sane in the use cases they care for. There's sort of a complete list of all the recognized use cases. So these kind of changes can be discussed without forgetting the other stuff. So, and I was not, I had never read that I'm embarrassed to admit I'd never read that I love the idea of recognized use cases as a way to remind us this repository has a purpose, and these are the purposes that it's it's addressing. Yeah, so refer to you. And at some point when, you know, the discussions went sideways because of reasons like that. Thanks for the question. Right, so let me, let me get through the dog to make sure we're not forgetting anything. What is the storage requirement. I wouldn't say that, you know, it's anyway demanding. You know, CPU and memory demanding as you can imagine right it's test suit Jenkins, the browsers, you know this slain him. He always got all got its, its footprint so it's somewhat memory and CPU heavy, but I wouldn't say that much for for storage. Are there collision do you generally have them running with a dedicated display device associated with each test or can they coexist on the same screen at one time is I'm not familiar with not familiar enough with Selenium runtime to be. You mean, if you would like to run several of them in parallel in a single host. Right, is that is that generally feasible or no it's rather you they really need to be one in one on a host at a time. Um, the thing is that you know, from the nature of of the right this thing there is a plenty of waiting it's very hard to write the test to be synchronous and you know there's a plenty of waiting when it's so such a constrained time to interact through UI, you click a link and you get to wait for page to be in the right environment so there's a lot of waiting. And obviously a lot of time out so because these things doesn't happen in time it can be a symptom of a problem but at the same time can be just slowness or something like that. So, we ended up never to parallelize is on a single machine to avoid this to be interacting with each other because there's quite a lot of going on. It's going to really happen and be very very unpredictable so we've been getting more predictable results and we just avoided that and you know sacrificing some resources and parallelize across a number of systems. But from the top of my head, I don't recall this would really be clashing or fighting over so over a particular display environment. What is real relevant for this is there's a couple of modes how to execute this. I mean how to execute the ATH and then to specify a browser which are somewhat noteworthy. I will do a very short detour to what these are. So the docs are all referenced from from read me so there's a selecting a browser which is sort of misleading because it does not only specify where the browser is, but there's a couple of, let's say, most substantial differences, presumably the Firefox and Chrome container. So what these do is that they just wrap the old frame buffer, this XVNC and the Firefox inside a container. So, and if you run number of tests in parallel, they would get it they separate containers for every test when the you are is running. So it will be hopefully impossible for them to clash. I'll be moving down the dog. Let's see what questions we have in there. The one that is next in the dark for me is the one on tables to give Jenkins to 264 to 65266 and 267. I was expecting major breakage there. Have you seen major breaks from the UI changes that have been made beginning with 264. I didn't have the resources to look into non LTS results. Okay. Good. That's okay so that's good to know that's a place that is a fertile field the rest of us can explore. Great. Okay. Because 263 is the next LTS. We've still got three months before we will get into LTS on something that has tables to give them. So, okay, great. I think that's, I mean, I know this is a thing we are doing here so on cloud besides so trying to anticipate weeklies before the LTS is as a continuous bow. I mean, the challenge of picking a reasonable weekly. Yeah, plus trying to actually use, for instance, you know, running everything on the weeklies in advance and continuously to be able to not end up taking everything in the face, you know, at the last minute when the LTS baseline has been chosen finally. And then, you know, executing it is at that moment instead of having tried to fix them continuously every time every week, every day, when things start to fail. It's hard because it's actually a lot of work. The ecosystem is huge. Right. Yeah, I mean, the entire ATH approach that that I've been leading for years. The goal was just this right but it turned out to be, you know. So somewhat, it required a consistent attention and work to be put into just keeping these tests up and running. And historically, we've been quite short on cycles. And for reasons that are absolutely understandable. It wasn't very attractive for the people to actually join helping you fixing analyzing resolving and situation like this. Okay, so let's have a look at the rest so we can spend the time on a future of LTS testing. Who would attack the failures right I guess we briefly covered that so in the in the upstream upstream CI it's pretty much whoever comes comes around. So that would be, you know, not necessarily the maintainers but whoever contributes to this so I must say that cloud based has been doing lion's share of working there. There's quite a lot of people that have contributed fix and I improved these tests as a time ago. I also know that we're working very hard on these, you know, making it our part. So, yeah, for the for the upstream. I mean, that's for, you know, just keeping the test suite healthy when it comes to LTS certification that's the mailing list approach I described so every four weeks there is an email that calls for for contributors to report the problem from their private environment that might or might not use at age. So, failure investigation. Well for for eight age only. It's done in a way that can be usually, you know, invoked on a little desktop so there's also a way to run this locally. Again, the documentation is somewhat complex and elaborate on how to actually do this, but in pretty much vast majority of the cases of you're able to rerun the test locally single tests and any to write on that. So, or even, you know, pause it and investigate so fair part of that is documented I would say the specifics are beyond the scope of this document. So, common failure modes. Right, I would say that change is one of the most common failure mode, which is sort of a counterpoint or, you know, sort of a disadvantage of using the UI base verification. Other failure modes that are not so frequent are various changes in in versioning, etc. where for whatever reason, when a test or an ATH runs with slightly different versions of plugins for some reason might be might be some dependencies optional dependencies or in the past this bundled plugins. Yeah, usually, it's some sort of disparity between the state of the plugins and plugin ecosystem and ATH, often the fact that the page objects of acceptance as a harness and the UI of plugin are, you know, not compatible anymore, but it's not solved with this. Right. And speaking of the duration. Well, it's been a while since I actually seen this executed in a serially, but last time I checked we were somewhere close to 12 hours. Since then, we've done some, you know, some reduction, which is a project that kickstarted some, I guess years ago, when the sort of, and I guess also in the contributing doc, sort of specify what tests does and what does not belong into the acceptance as harness. Let me see. Yeah, test contribution. So the idea is that, you know, while we're interested in having a reasonable coverage, we definitely cannot have tests for every plugin for all the use cases or even, you know, even thinking about the code coverage is close to insane given how expensive the test run it how slow it is. So we sort of specified what is expected to be in and what is not so test for plugins that are don't have enough installations or there's just, you know, occupying too much, too much time to try to reduce them or move them away, which we partially partially succeeded. So again, the idea was to keep this sort of constraint and prevent the age from bloating. And, you know, hopefully keep the runtime and the scope and the amount of coding there reasonable and easier to maintain. But, you know, I guess for the reduction is still something desirable. So this will be the full runtime. So when it's running parallel, it's usually what was it will be using something like 10 splits. So they usually fit into three hours. I mean, the slowest one, there is, you know, it's not just 12 hours divided by by 10 in this case, it's usually, you know, taking a lot longer than an ideal case but this is roughly the time estimate. Moving further, what types of issues does ATH certification detect? That's a good point. Oftentimes, we run into the situation that this is an incompatibility in, I would say in a substantial majority of the cases is incompatibility between the test suite and the plugin. You know, when the UI changes, users might be surprised or used to something else, but it's not necessarily, you know, considered regression from any reasonable standpoint, though the tests are going to break anyway. So vast majority of the problems that LTS detects are these incompatibilities or some flakes in the environment, which, you know, no amount of effort we were ever able to dedicate to this resolved. So I consider this to be inherent to the Selenium and the entire UI verification approach. So the problem this do detect are usually something that, well, I don't think that there is a particular category of issues that is detected we would, you know, we would highlight. So, when you say incompatibility, I like that one because there was a there's a current failure in a plugin I maintain that is due to a, a loading prior to the plugins own tests of a library that's older than the one that plug in it's a valid, it's a valid condition, it's a valid case, and the solution is restart Jenkins after installing a test or after installing a plugin and so it's those kinds of things that it surface those for you as well it's not just me that's benefited from ATH finding that kind of that kind of challenge. Yeah, I was thinking along on this line, you know, the NLTS standpoint it's hard to use typically the so called PCT plugin compatibility tester. But in the case you're mentioning mark might be that it's easier to use and less flaky or less you know full blown things Selenium related because it's full Maven. So, PCT is basically a thing that will look into your dependency tree and bump everything using the changes in the center data to, you know, use the latest and run MVN test the same way, somehow. So, it's going to be more Maven based than Selenium based. So, probably a bit more stable but I was thinking, you know, on the cloud and the side is different but indeed on the Jenkins side. We have 1700 plugins plus cloud counting. So, we can't really run. One of the 170, you know, 1700 PCT runs of Maven tests and everything constantly. The right way would be to actually do this on single plugins but it's a quality thing more than just an LTS thing I guess it's much wider than just the LTS subject I guess. Yeah, I mean, my, you know, visibility into the into the plugin compatibility tester is practically no, though, yeah, what actually can help is, you know, drawing a line somewhere like for his LTS we require at least 1% of all Jenkins installations need to have the plugin installed for it to be considered something that ATH is being run against, which, you know, I guess it could eliminate a lot of these, you know, potential runs. So, you know, it depends on where we put the bar, but yeah, we can focus on the most popular plugins or something like that. That's a good point. We could just use an obvious things like like non sentiment based somehow things like, you know, last release date plus number of installs. Something like this, like a mix to test the like, you know, like pipe because basically, for instance, if Jenkins if pipelines are broken Jenkins is moved. So, yeah, we should probably do it. We probably want to detect it. Yeah, precisely. Right. Okay, so is there anything else or we can have a look at this future of LTS testing. Yeah, Oliver, yes, a brief summary regarding statistics, just to get a sense of for the older releases that you've been dealing with and leading. How often do you see regulations and mostly all when those regulations are based on the UI changes, just to have like, you know, overall numbers that give us a sense of how is the, how is the releases behaving. Obviously, my chain between releases, but you might help me as well to understand. Right. Well, when it comes to the LTS is itself. I don't keep the keep the numbers but from the mailing list we should be able to find it out. So it's like in every one in three or four from my you know got feeling of the RC testing we discover something that something stopped working. Not always it requires further back porting sometimes it's just plugin that is incompatible, or something that can be fixed on a side of a plugin and we get to rush it in, you know, merging it and releasing it. Oftentimes it's a glitch that just happened and people need to do something in a reconfigured is stop using that or something like that. So it's being documented into into an upgrade guy upgrade guide that we produce with every LTS so Mark was doing a great job and composing these lately. You know, so short warning or guidance for people what to do and what to change. So yeah I would say that can be one in three one and four requires something right. So the fail test as I said it's probably a vast majority of the failures that we observe are some kind of incompatibilities or flukes. This is somewhat given by, you know, what amount of time we actually preventively invest into the maintaining the 8080 age. Right. I mean, if somebody would be looking at the at the failures every day and working on, you know, one hour to actually fixing them. It's not that much of an allocation, but we keep, you know, we discover we discovered these incompatibilities fairly soon is get fixed and it doesn't pollute the statistics anymore, which is quite far from the reality of the past years. You see quite a lot of issues that is being reported, and vast majority of these are actually you know just plug in has diverged at some point any at age haven't been adjusted yet. Right. But still, the statistics does not, you know, are not very encouraging in this regard anyway, because the number of time that something needs to be touched is still something like, I don't know, like nine times in 10 when something breaks is just the incompatibility which is otherwise harmless and one in this 10 cases would be, you know, some actual bug or some actual problem. Right. So, thanks for the questions that was really, really good. We discovered, you know, a lot of things I didn't even thought of covering so definitely definitely help me to to completing this. And thank you Mark on helping to capture all this in in doc so that's a good job done there. I focused, I mean I suggested a couple of things that should be done in the in the in the future of LTS testing with ATH or not. One of them we briefly scratch that would be the simplification and reduction to cut, you know, runtime to cut resources to cut the feedback time right you know one of the reasons why we don't do it is a part of the best thing is that we probably don't want to wait three hours for the poor request to get a green light, even if this would be all stable and all that. So, that's one of the, one of the, you know, possible ways to go. The improvement there is only linear like even if we just sacrifice all of the tests or make them running twice as fast, it's still get from, you know, three hours to one and a half, and it will be a heck of an effort to actually get there. It's sort of questionable to, you know, it's a linear improvement, I guess that's it all. Another thing is that what we do with ATH is becoming less and less relevant to modern Jenkins. We are testing the UI both, I mean, you know, to pulling data from it or observing it is still, still, you know, valid as of before, though configuring it and clicking through the UI in order to put things in there, mostly configuration is becoming less and less, you know, popular use case. So that's another thing to consider how to actually, I mean, this is, you know, we'll be encouraging user to do and, you know, speaking for Red Hat, this is what we are pushing very, very hard on to actually get every internal user of Jenkins away to actually get it configured without touching anything. So, you know, the complexity of putting things in Jenkins forms through page objects is just immense and, you know, ideally should be used less and less every year. So, one other thing is that I believe it's not putting enough empathy, sorry, enough empathy on on jcast can job DSL configuration, you know, simply the UI is not as used as before. So that's something that I guess it would benefit and make this a lot more, a lot more relevant. And also, it's just verifying the war file I mean there's a couple of other ways how to how to run this is also configurable. So what we are, I guess, only focusing is how the Jenkins bar is running, but not necessarily how it's running inside of a Jenkins container which again, sorry I don't have the numbers at the hand but yeah, I guess it's just, you know, being used more and more often and that's probably something we would like to verify, maybe as well, but eventually switching to this almost exclusively. So earlier, is there any execution on windows, I assume this is all Linux based is that is that correct. This is all Linux based that were fixes and improvements coming from James in the past so I guess he might have more information. I think the investment done from us on windows we you know, obviously work with the contributors and try to make sure that this is this is running but it just runs on windows. He likes to suffer so that's. But yeah, that's probably why he is the one pushing the most fixes specific for windows because I don't think we have a lot of others running windows. I mean, they have machine even. Is club is interested in this use case as a part of business. You mean running things on windows. On agents definitely on masters. Yes. Like you see the gradients. But we do not recommend it, you know, controllers on the windows should be. Yeah, this courage set up as far as I think we do we put in our kbs but don't cut me on that. Yeah, that's, you know, fairly similar so we are actually not that it would be not doable, but it's definitely not straightforward how to get at each run on Linux and get on the controllers on windows. Right, which, you know, if there is any windows use cases to be this one. So, you know, as I said, I've been running this project for Jesus, six, seven years, I guess, quite a lot. And honestly, it's been a big, big burner for a vast majority of the time. So definitely there are things to improve. Definitely do not hesitate to reach to me I would be more than happy to provide my guidance and experience of what can be done with this. And, you know, I'm currently in a state that I'm almost certain that something needs to be done with this for partially the reason that I've already mentioned. So yeah, I was glad that the, you know, plug the PC to us mentioned which I was looking at a possible alternative. Maybe I'm not going to say replacement but a possible thing to converge to to actually figure out how exactly they would like to certify the LTS. And also, and this was really more of a vile idea rather than anything else. And that's composing something totally different, based on container jcask job DSL, and some other form of verification which would be prone of Selenium, which would hopefully save us some problems, and it will resolve the thing that you know configuration through your eyes becoming less and less relevant. And that would be, you know, verification through some scripts which feels sort of white boxy and quite nice, but or alternatively using his HTTP client. What is called Jen, not Jenkins CLI, you know, using the HTTP client for Jenkins that is based in Java to actually very, you know, start build verifying states getting logs and things like that. So that's more of a vile idea how to how to advance further and avoid a lot of the hassle that we currently have. I mean, that we always add with Selenium. Oliver, thank you very much. We've used your hour of time. Is there anything else you want to advise us as we conclude here. Thank you very, very much. Thank you. Yeah, folks, definitely thanks a lot. You know you brought quite a lot of interesting questions. The thing is, as I said, I would like somebody else to take this over for now. So I guess it would be the number of contributors we currently have. James would like to step in and becoming the main maintainer. I will be more than happy. It's not that I'm leaving for good. I should still be around still can help with things and provide some guidance and some historical context if that's desirable. So I should still be available to do that. That's great. Well, and, and just to share Gareth Evans who is on the call is has agreed to take on a piece of this as part of his responsibility in the Jenkins community team, or in the community team at CloudB. So, Gareth has the unfortunate privilege of reporting to me. And therefore he gets to, he gets to listen as I try to guide and steer. So you will probably be pinged by Gareth or by me as we looked at how should this evolve. Speaking of unfortunate privileges, Gareth, do you owe an axe? Is it a big one? It's not that bad, unfortunately. I'll get a new one. I've known less lucky guy than, you know, you will be reporting to Mark. He's an adorable person, we all know. Anything else, Oliver? Not for me guys. Alright, thanks everyone. I will post the recording after its process. The recording usually processing takes on the order of 30 minutes or an hour. I'll also place a link to the recording and to these notes in a comment to the Jenkins developers list so we have them archived. Thanks everybody. Very, very much. Thank you. Perfect. Thank you, Mark.