 Welcome, everyone. This is the introduction to Jenkins 2.277.1, the new Jenkins long-term support release. We're delighted to be here with you. Thanks very much for joining us. This is being recorded, so it will be available separately later as well. I'm Mark Waite, joined by Oleg Nanashiv, and together we'll be presenting much of the material here, then in addition to those, the presentation part, we've asked three of the key developers involved in these changes to join us for a panel discussion, where we'll do question and answer, letting us ask them questions, gain insights into their experience and what they learned from taking a product used by 250,000 installations from previous technologies to newer technologies. We're thankful for Jesse Glick, for Felix Keroga, and for Tim Giacome, each joining us as panelists. Thanks very, very much for being with us. Now, by way of agenda, we're going to do these welcomes, and then we'll talk through the introduction to Jenkins 2.277.1, including things related to the user interface, how we improve the core, and then we will shift into a panel discussion where we'll invite questions from the audience and questions from Oleg, from me and from others, to gain insights into what it meant to make these changes, how they were verified, what sorts of experiences there were. We encourage you during that phase to ask questions of our panel as well. By way of reminder, during the meetup, it's best if you use Zoom's Q&A feature rather than chat. The Q&A feature lets us manage the questions more effectively, you'll find the Q&A button on your Zoom user interface. Now, we will reach a point in the meeting where we will turn off the recording, and after having turned off the recording, we'll actually grant voice to everyone so that you can come on and ask questions of our panelists separately. We like to do that because we want the recording to be have had the benefit of careful vetting by us as moderators, but we want you to have the facility to ask questions at the end if you'd like. And after the meetup, the Jenkins mailing lists are a great place to ask questions, and at the conclusion, we'll ask you to submit your answers to this feedback form, helping us better understand which things work well and which don't work as well for Jenkins meetups. We're so glad that you're with us and looking forward to a great session today. So Oleg, you want to go ahead. Okay, so I will do a brief summary of what was introduced in the recent LCS release. 2.277.1, this is a new baseline, we released the first LCS release in this baseline last week, and many of you have probably installed it. But still, we would like to discuss what's inside and talk about how to upgrade and what potential obstacles you may experience. Next slide. Just quick summary, what is Jenkins LCS? Jenkins LCS is called long term support, though historically we rather use stable release term when we discussed this LCS, because LCS release is selected every 12 weeks. So it's not a long term support, something like several years. So it's rather a stabilization release. And when we select new baseline, we have three releases every four weeks with backports and security fixes. Sometimes if we need to release additional releases, or we can maintain the release line all longer. But we normally expect the user to stop with the new baseline after 12 weeks, or eventually when they're ready. First, the process is coordinated by the Jenkins release team, and by Jenkins release officer, a team who's also on the call. And the team is working on preparing backpours, preparing challenge logs, and preparing upgrade guidelines for each release. Next slide, please. So, if you're Jenkins, next slide. Thank you. So, yeah, what do you get as a user? Firstly, you get all packages. So it's a Word file. It's a installers for different platforms including Windows multiple Linux versions and those docker packages, which you can use for docker and Kubernetes deployments, deployments in other container environments. So there are packages which are not maintained by the Jenkins project directly, but maintained by other communities, for example, free BSD, Gantt, Macos, OpenBSD. We don't distribute these packages on our own, but they are being released by other communities. And yes, you get to change logs and you get upgrade guidelines, which are specific to the LCS releases. You can use it when you prepare to upgrade to the new versions. You can go through changing logs between your current Jenkins LCS version and the one you want to install. And you can see all major changes and all steps you need to perform to do the upgrade. So, let's go to the next slide. So, for 2.277.1, it's basically a weekly release with some backpacks. So the previous brain line was 263. And in this version we introduced quite a lot of changes, but actually in 2.77 we introduced a lot more and some of these changes are breaking ones. The release is quite stable and it includes bug fixes which have been released since the baseline selection. So we selected this baseline six weeks ago and over six weeks we processed all the repressions which have been reported for weekly releases, we reported fixes, and when you install a CS release, these fixes are already included. So the release should be quite stable. Next slide. So, if you want to find a change log, you can go to the URL provided on the page. And if you go to this page, you will see some interesting details. There is some animation on this slide. Okay, we can do like that. So you can see that the current community ratings are not exactly great. So we have 124 successful deployments, but some users reported different regressions. So you can see that approximately 30% of upgraded terms experienced some issues. And it's quite a high number. So it would be nice to understand what happened. And, yeah, let's go to the next slide then. Okay. Yeah, so yes, there are breaking changes. And actually that is quite a number of them. It includes. Could you please go to the next slide. Yeah, thank you. So, in this release, we started at the recent technical depth we had in the Jenkins core. So there was a lot of changes related to visualization and dependencies. This is the most significant update of SG security to spring security replacement of all the extreme for by upstream version maintained by the extreme community. And we have the awesome library and actually we updated a lot of other dependencies inside the weekly baseline between these releases we have enabled to depend upon so actually, there is a lot of library updates here and there. And these are main dependency changes. Also, there was major changes related to configuration forms and forms in Jenkins in general. So what we call the table studies improvement for better accessibility. And yeah, this is also breaking change. Later we will discuss these changes in details. I just summarized them and glance. So, for those who use system properties that there were several system properties which were placed by new names. So no Hudson now it's Jenkins. In addition to that we had a few improvements. So, for example, plugin manager got a lot of improvements where you get available pay page so it lists plugins much faster. So that was synchronously. Then, now you can specify a reason for preparing to shut down. There is new color palette, which is a continuation of your UX improvements, which you may have seen in previous releases. And also there is a lot of performance improvements and events here and there. So the full change lock is much longer that one would present, but I just wanted to list key highlights. And, yeah, I think that we should just talk about the details, especially about the breaking changes. Mark, would you like to take over. The key components that we see here. The first piece for me is this user interface improvement story. And we have to offer sincere thanks to Josh Soroff who started the, the poll request. Well over a year ago to Felix Kerrigo who has been a major leader in in implementing it Joseph Bergen for his design work on it. And Tim Jacome for amazing fixes in all sorts of places. Daniel Beck for some really good testing and evaluation, and many, many plugin maintainers. Thank you. Thank you. This has been an amazing effort to bring us this new set of UI improvements. As we see the old UI before to 277.1 was laid out with HTML table markup, and that meant even on a wide screen so this screenshot that you see here is a 1280 pixel wide screen. You can see the scroll bar at the bottom, and notice that the fields are truncated over there on the right hand side. It just didn't feel like a reasonable use when my nice wide monitor doesn't fit the whole form that I need to edit. Thanks to these changes that have been made. Now, the fields fit inside the page. Okay, that's that's already a nice. There are also hints with dashed lines along the left hand edge edge that helped me see which things are grouped together previously I wrestled with hey how does this work and where does which thing does this apply to and that doesn't apply to you, and no horizontal scroll bar. So, those kind of things are significant improvements to the experience for someone editing in forms. Seven core pull requests, 40 plugin releases 40 plus plugin releases 14 weekly core releases to get here. 56 issues resolved. And yes, there are 15 issues still open. Well, a live demo as well and oh like I think that's, that's kind of a fun exercise let's take a look at my Jenkins on my live environment. So, here is my Jenkins installation and if I go to manage Jenkins note this is 263.4 so the previous long term support release manage Jenkins and I'm going to click configure system. And my screen is quite wide, and yet we're going to see when configure system arrives that I'll likely get a scroll bar lucky me. Meanwhile, while that is loading. We're going to do the same exercise with Jenkins 2.277.1 manage Jenkins configure system. So, here we have 277.1 wide screen and already I've got a horizontal scroll bar. And as we go down the UI is Oh look, picking the credential is really hard to do because that the pick list is actually off screen. Those kind of things made the UI more difficult for administrators in Jenkins versions prior to 277.1. Look, everything fits and yes I've got these nice little hints over here on the left that tell me what's going on, which thing I'm grouped with and how. Thank you very much Felix, Josh, Tim. Thank you all of you for doing the work to give us a better UI. However, you can open this configuration page, even on the mobile screen, and actually it will look quite good. And, and all like you make a very good point that it's quite a treat that now my narrow devices. Okay, I've got a Chromebook I use I've got my phone that I use. I can actually do better have a better experience on those narrower screens on many different screen sizes than I could before. I can barely imagine that somebody would manage configurations from mobile screens, but in practice you can do it. Right, exactly. So now the question is how do you get there. And, and this part we're going to reinforce multiple times today you get there by reading the upgrade guide. We've intentionally described in it things that may need your attention. So many Jenkins releases on LTS. You may have ignored the upgrade guide this is not one of those places read the upgrade guide. Second step, remove unused plugins before you do the upgrade. Many times older plugins will cause bumps and bruises that you don't want to experience. So if you realize hey I'm not using this plugin. This upgrade is a great time to choose to remove that plugin from your installation. Now, then, even before you upgrade to Jenkins 2.277.1 upgrade your plugins. Now you may be accustomed to doing Jenkins upgrades without upgrading plugins but because of the breaking changes inside 2.277.1. It's important that you upgrade your plugins. You do them before because there are many plugins that can be upgraded even without upgrading to Jenkins 277.1, and you get, you get that newer version. If you do the upgrade to Jenkins 277.2.277.1, then afterwards you again do the update of your plugins because there are some plugins that must be that are that are not available for 263. So you need to get 277 and then upgrade again so remember that's two plugin updates that you need to do. Now, Oleg, do you want to take on core component improvements or do you want to introduce it to Jesse, if Jesse is interested. Okay, Jesse, do you want to be voice. I can't actually talk about the ASM update but I can talk about these two slides so was listed as Jenkins enhancement proposal number 227. And this the idea here was that since pretty early days in the Jenkins project actually going back to Hudson days, it had been using this library called Seji security. I hope I'm pronouncing that right. And that was sort of an old Java based security framework. And Jenkins was using a lot of the API is defined in that framework to handle every aspect of login and authentication. So basically anything that you would configure under the security realm part of the configure security of global security screen in Jenkins. So that would be built in user database, LDAP, different kinds of SSO systems, active directory, all of those authentication plugins are all using this API. And a lot of other plugins were also using this API to pick up information about who the logged in user is and what kinds of groups they were assigned to LDAP groups, that sort of thing. And this all worked. But basically, we were stuck on this old library. And years and years ago, that library was totally rewritten. And it was, I won't say totally rewritten, but it was largely revised. And the revised version became known as spring security version 2.0. And that was sort of broad under the aegis of the spring project with a lot of other libraries. And by the time we came to this work, that library was on spring security version five. So you can imagine that there were a bunch of other major refactorings in three, four and five after that point. So we were using this very old library. We didn't really want to be using it. We would rather be using an official upstream version of all of these things to make sure we're getting all of the latest fixes, especially for something as important as user authentication. Unfortunately, not only did the rewrite to spring security change the Java packages, but it introduced a lot of other incompatible changes to the library. So it wasn't an option to just replace one line in Jenkins and say, yes, please use the new one, change this version number, and everything would be fine. Not only did we have to change hundreds of files in Jenkins Core to refer to new package names, but a bunch of other things needed to be changed as well. And it turned out that dozens, maybe hundreds of plugins were referring to the old library under the original package names and making assumptions about how the old library worked. So I'm not sure how much detail you want me to go into now, but basically this involved a pretty big patch to Jenkins Core to introduce the new security framework, keep some stubs in place for the old one without actually bundling the old library, just creating a facade that would let plugins that were still built against older versions of Jenkins continue to work the same way with only a very, very few exceptions. And so this landed in Jenkins 2.266, which means that it appears in this LTS line for the first time now. So I think that's, yep. Excellent. Thank you, Jesse. So this is my chance to, would you be willing to be the voice on this particular slide again, just to remind people what should they be doing? Upgrade plugins, yes. There is a list of plugins affected by this change that we know about. Try to best keep it up to date. We rely on community contributions to notify people, notify us and everyone else about when particular changes are accepted and released and so on. As far as I know, all of the changes needed for, they're actually required for plugins to work after this change have been released, meaning that you can, meaning the updating plugins before the Jenkins Core upgrade work. So they've not just been released, but they've been released in versions that are compatible with the previous LTS line. There's one exception, which is the CAS plugin, which is a sort of not frequently used single sign-on authentication plugin. That one, for technical reasons, we couldn't do that. And so for that case, you actually have to update the plugin at the same time that you update Jenkins Core in order to be able to continue to log in. So now, if, Jesse, if the user made the mistake of not updating their plugins before they did the upgrade to Jenkins 2.277.1, is there a risk that in some configurations they might be locked out? Yes, definitely. So if you, for example, if you were using an older version of the LDAP plugin, more than a few months old, I think it was 1.24, maybe, or 1.25, that the very basic compatibility fix went in, the minimum compatibility. So if you're using an old version of the LDAP plugin, you haven't upgraded in a while, and you just went and upgraded Jenkins Core without touching that, then after the upgrade, you would get all sorts of errors the moment you tried to log in. And that's especially a problem for a security realm plugin, because you need to be logged in as an administrator in order to update from the GUI. So that's why we really emphasize you should accept all upgrades before you do the upgrade to make sure you don't have this kind of lockout situation. If you get into that, you have to go into the Jenkins home directory and shut off internet access to your server, go into the config.xml file, and turn off security temporarily, log in as admin, do the update, and then undo all of that. So it's possible. It's just a bother. So don't miss upgrading your plugins before you upgrade your Jenkins. In the past, it wasn't always that way. This is somewhat distinctive because of the breaking changes in this one. So, Jesse, on the next topic, extreme unfork, would you be willing to share some further insights there? Sure. So this was another Jenkins enhancement proposal, kind of along the same sort of lines that we wanted to be using an up-to-date standard version of a library. In this case, extreme is a sort of an alternative to Java serialization. It follows a similar model to Java serialization, but it reads and writes XML. And basically everything in Jenkins that has any kind of stored configuration, any kind of settings, as well as metadata about builds, definitions of jobs, all this stuff is all stored in XML files using this library. So it's pretty central. And unfortunately, we were using a fork of this library that, I guess, goes to K-Fork way back in the midst of history to add a few customizations here and there. And the discrepancies started growing. And we had been making an attempt to pull in upstream changes from official extreme releases and merge those with our patches to it. So we had half a dozen little changes to extreme or something for Jenkins use. But the patch was falling out of date and it was really painful to maintain those things. It would take somebody a day to go through and analyze what needed to be done when we do the update. And so the result was we weren't updating it. And worse, one of those patches was actually the result of sort of a misunderstanding of how extreme is supposed to be used. So we were using an annotation feature that was convenient and was used in a handful of places in Jenkins core and a few plugins. And it was nice. But it had a prerequisite about the threading architecture of how you use this library that did not match how Jenkins actually worked. And we had tried to work around it with some patches in the library. But when we offered those to the library maintainer, he said, no, no, no, this is not how you're supposed to be using it. No chance I'm taking these. So we were a little bit stuck with that. So as part of this work, it was a little bit easier, certainly than the 227 spring security work. But we did need to basically undo that patch, throw away that facility and change some codes in both Jenkins core and a few plugins that relied on that assumption, basically make it work with the official version of the library cleanly. So that also cue the next slide. And require some updates. In this case, not quite so critical, because I think nothing is going to block you from doing the update as an admin after you upgrade, but you might see a bunch of stuff broken before that. You could have failing builds. I'm not sure what all could break. So that's being on the safe side. Thank you, Jesse. And thanks for your work on the Jenkins enhancement proposals and what you've done there. Much appreciated. So the ASM upgrade is one that I've been aware of. ASM is a bytecode manipulation library that is used in the product in various ways. In this particular case, the example was that we brought in a new version of the ASM bytecode library and the token macro plugin had to be updated to adapt to that new one. If you don't have the latest version of token macro, and you can install it still on 2.263, but if you don't have the latest version, when you upgrade, you may get messages and warnings and failures that are described in the upgrade guide. But this is our way of saying, please, this one has nothing to do with breaking Jenkins XML reader right. But what should you do? Read the upgrade guide, remove unused plugins, update your plugins before release, before they upgrade to Jenkins, upgrade Jenkins, update your plugins again. So why plugin upgrades? Because we've made breaking user interface changes. We've made breaking security framework changes, breaking serialization changes, and breaking bytecode changes. You need current versions of plugins. This is different than many releases, but absolutely worth it. So this is where we switch to the panel discussion, and I'm going to switch off and stop my share so that we can see the panelists. Before we continue, maybe it makes sense to ask Tim and Felix whether they would like to add something. Good point. Felix, Tim? There was a lot of work behind the scenes and a lot of work which hasn't been presented yet. That was a really good sample and a nice live demo from email and some of that. So, Oleg, I have not brought up the Q&A panel. So let me look at that. If we've got questions from the audience. Yes, we've got one. And yeah, please use the Q&A because we will answer any questions. And later, there will be a discussion where you can ask all the questions live if you want. Okay, so the first question. Do you have any recommended way of identifying unused plugins in environment with loads of projects? So unused plugins. That's an interesting challenge. To our panelists. I know that Cloudbees has a way of doing it. I'm not on the Cloudbees. So there is an open source plugin usage plugin which tracks plugin usages and configurations, but it doesn't track plugin usages in Jenkins pipeline. So if you just want to analyze configurations, which I involved, you could try using this plugin. But if you want to go beyond, yes, then Cloudbees plugin usage plugin. Well, and I confess, I've just used configuration as code and delete the plugin and experiment with a prototype. Now, that's that maybe that's a terrible way to say it, but many times I can look at the plugins description and say, oh, you know what? That is somebody's old thing that they brought in long ago. We should just get rid of it. Yeah, you can just have the disable button restart Jenkins and see what happens. See if anyone complains for ones that you think aren't in use and then just remove them later. Yeah, especially for plugins that are marked up for adoption or have other kinds of red flags. That's a good idea to check periodically. Yeah, I had just seen a recent issue come in where a number of plugins were flagged with security warnings. And if you've got a plugin installed that's flagged with security warning, this is a great excuse to consider removing that plugin. You've at minimum should upgrade it. And if I'm no upgrades available, now it's time at least to consider should I keep that plugin because it's got a known security vulnerability. We are always looking for contributors because for some plugins, especially in a long tail of plugins use just a few hundreds of installations. They know active maintainers and it takes time to deliver fixes there. So if you use this plugin, consider stepping up as a maintainer, and we'll be happy to help you to get started providing permissions and reviews we needed. We're looking for contributors. Thank you. Next question about future roadmap. Currently Jenkins uses a flat file system for its data and configuration. There are a few ways to attempt to keep the data such safe as a thin backup plugin or really raw if used to be the context of Kubernetes. But are there any plans for the future you're in far off to look using a database similar to SonarCube, how SonarCube operates? I can start answering this question because very good discussions about how we could switch to the database and moreover, there was some research about that. And if you go to Jenkins JEPs, you can find the JEP 2.13, which actually provides configuration storage API. I will put the link to the chat. So this JEP hasn't been fully implemented, but in principle, it's possible so that exporting color configurations to the database. Whether you should use that, it's a good question because my personal recommendation would be to just use configuration as code and infrastructure as code. So basically a combination of a jcask plugin and a pipeline as code. The job they said if you need it, and then you don't really need a database to store your configuration because your Jenkins becomes immutable. So it would be my recommendation. And I would differentiate between the configuration and the runtime data produced by build. So certainly for the configuration itself, the configuration as code is our best recommendation at the moment. As far as runtime build metadata, currently that is heavily file system-based, entirely file system-based in default Jenkins installation. We have some efforts in progress to let parts of that be replaced. So it is possible to replace artifacts in external storage. There's experimental support for keeping build logs in external storage. And maybe Tim can say what the status is of keeping test results in external storage. So all of these things combined at least reduce the pressure, reduce the amount, reduce the total volume of files that would be kept that you would want to back up. So if you're interested to find this information, you can just go to the Jenkins site. There is Cloud Native Sieg, which has been historically working on pluggable storage. And here you can find statuses for the main artifact types. So artifacts, credentials, system logs, test results. That's what available for previews, fingerprints. And she's talked to some on 20 projects. But for build logs, it's only partially implemented for pipelines, but you can use it. And task logs, configurations, code coverages and build storage is what we haven't implemented yet. But in principle, it's possible to implement. We are just looking for contributors. And if you're interested, please join the Cloud Native Sieg meetings because we can collaborate and get other entities implemented. Thanks. So we have an additional question. The question is specific to a particular plugin. I'm not as familiar with this as I might like to be. It is, is the UI for post-build action drop-down in jobs also resolved? So previously they were not able to scroll. I had to zoom out the browser just to see it. And I'm not sure I recognize this one. I don't know that I've seen it. Have any of our panelists or Oleg, have you seen any issues there that the UI improvements would have resolved? Post-build action. So it wasn't listed in the list of regressions. It means that it doesn't have the table layout. And presumably with the new version, it should get better visualization. I can probably test it synchronously while we answer other questions. Great. And then we can return back to that. Super. So then another comment. Thanks to everybody who's done the work to make the UI improvements possible. Now a question has been asked, are there any changes that would require code refactoring of user-defined declarative pipelines or scripted pipelines? So Jesse, this one might be one. Are you aware of anything in the serialization, the extreme code, or the authentication code that would require pipeline changes? No, there shouldn't be anything. Yeah, this should all be invisible from the perspective of someone using Jenkins. Definitely there would be changes for people writing plugins. But as long as you're writing regular pipelines using the sandbox, there should be no effect. Now, if you have, you know, groovy libraries or other unsandbox code that's doing strange things with Jenkins internals using direct Java calls, then you're in the same category as a plugin. And it's possible but unlikely that would be affected. Thank you. And Felix or Tim, are you aware of any places where the UI improvements would require any change to a user pipeline either scripted or declarative? Okay. No is a good answer. I like that. Great. Okay, then another question. I'm using CephsFS, so the distributed file system Ceph with the latest LTS version. Are there best practices to follow for using that file system? It's Jenkins on Kubernetes with JCask and using Ceph as the network storage. I confess I've not heard of many people using that particular distributed file system with Kubernetes. Are any of you as panelists, have you heard anything that could give any advice? Well, I had some support for Ceph recently, but the historically just works. So I cannot provide details about usage experience. I can talk for hours about GlastrFS, but what's the practical experience? I know that it works quite well. Thanks. Another question. And are there any plans to provide a build history per node for pipeline builds? Yeah, I can take that. No. And basically, because not that it would be especially difficult to implement, it would require some coordinated changes between Jenkins Core and One Plugin. It would definitely be feasible to implement and have it look like the freestyle version, but the performance would be even worse than the freestyle version, which is already sometimes a performance issue. And the reason basically is that if this question is about what I think it's about, if you go to the build history link under an agent, which is really only meaningful for sort of old style long live static agents, it's almost useless for a cloud system. But let's say you have some Mac mini in a closet where the Windows 2008 R2 server that you've kept online for a year and you want to see the list of all builds connected to it. The way that's implemented basically involves scanning every build of every project in the Jenkins controller in reverse chronological order going back to the beginning of time, loading the XML records for all of those, and then checking each one to see did it occur on this particular agent. And if so, show it in the list, which will work for a small system, but it can definitely cause IO thrashing and serious performance problems on big systems. And to implement it for pipeline, it would be worse because of the pipeline metadata is not just a single file because you can't because it's not necessarily just a single node. It's using every node block and there could be a dozen node blocks running in parallel. Each of those is stored with its own metadata. So we'd have to read all of those things. And basically, there's not a reverse index for all of this information. So this, you know, if we had going back to the pluggable storage discussion, if we had a proper relational database or something that held historical builds metadata going back through time, we could just say create an index by agent and retrieve from that index, but we don't have anything like that in Jenkins today. So this is something that would have to be left to some external dashboard, you know, Jenkins would stream information to and then it would render that information from its own database using an efficient technique. Thanks, Jesse. Thank you very much. Thank you. There was one question which I accidentally moved to answer about performance, where I can find performance reports about the Jenkins, for example, how many files I can handle in the Jenkins home and how many jobs I can run with a specific resource and integration. Does anyone want to take it? Okay, so that's as the documentation officer, that's an indelicate thing for me to painfully acknowledge, we don't have an awful lot of performance data on the Jenkins public site. We've got some rudimentary guidelines there, but that's about it. And in general, Jenkins at scale, there's been some interesting talks, a talk from Stephen Connelly some years ago on running live the largest Jenkins cluster at that time that had never been run. So there are certainly talks out there, but specific details on number of files or how much swap do you need or what are these constraints on IO throughput. The Jenkins project does not have that kind of information. Well, first of all, it truly depends on your pipelines, because Jenkins basically stores this execution context on the controller and handles the most operations on the controller. So resource requirements really depend on what tasks you execute with Jenkins. And if you want to do performance testing for your instance, there are some guidelines which you can follow or just use standard Java development tools and begin from there. But yeah, targets will be really dependent on the system you run. Thanks. So we've got another question that arrived for cloud images. I often see them become unavailable during the job and then the job just keeps running until it times out or is manually killed. Are there plans to add any pipeline scripting options to capture when an agent fails so that that portion of the pipeline can be relaunched on a new agent? Yeah, that is something that has been proposed. I mean, it's possible to do, but it's awkward, essentially, right now. So you can have, for example, you can have a retry block that just retries the whole chunk of work, including requesting an agent, waiting in the queue for it to become available, connecting to it, doing your build process and the cleanup at the end. But then, of course, if it's a real problem, a compilation error or something that's genuinely wrong, you would be wasting time retrying that when you know it's going to fail the second and third time the same way. It is sort of possible with some awkward scripting to inspect error messages from a failure to see if they look like something related to an agent outage. Right now, there's nothing provided out of the box that does this smoothly. Definitely something that would be interesting, especially if people use things like AWS spot instances where you know it could just be killed with very little notice, but just in general for any kind of agent, or you might have transient issues for lots of reasons that are hard to predict, and you would like to be able to retry a block. Unfortunately, nothing today. You can try using some plugins like local parts and build failure analyzer, which can catch particular events quite easily. But for these plugins, you can restart the job entirely for pipeline stages, et cetera. Yes, scripted pipeline with striketch blocks is the only possible response right now. Unfortunately. Thank you. Good. All right. So we've got another question about related to plugin downloads change to Artifactory. I'm not sure. I recognize that I'm not aware of any change in the plugin mirror infrastructure. As far as I know, we're still using our publicly maintained mirrors throughout the world mirrors in China, in Japan, in the US, three or four of them in Europe, three or four of them. So I'm not sure I do do others on the panel know anything that I need that maybe I've missed with regard to Artifactory plugin downloads. Maybe it's a reference to archive. Because yeah, when you download plugins using Jenkins update centers, you download plugins from mirrors. But if you go to archives, archives are being downloaded from Artifactory. Okay. Thank you. At least that's how I understand the question. Yeah, if you're using configuration as code or something, you would, I guess, normally be referring to some things stored in Artifactory. Configuration as code plugin. So if you talk about plugin installation manager, it downloads from update centers. It does good. Okay. If we talk about other custom tools, like let's say custom work package, install plugins to sage, all of them are also download from mirrors. So the only thing which downloads from Artifactory and all about is this thing. So if you go to the list, here you get the Artifactory link. Oh, actually, it's also updates Jenkins.io. So even that is not Artifactory. Yeah, so now there was a request to assure that we sanity check that the use of repose.jankins-ci.org was legitimately for the Jenkins project. And so we did have to change some things where we had mistakenly allowed some tools to be uploaded to the Artifactory repository. Those needed to be held elsewhere because they weren't strictly related to Jenkins. So we did that. But that's about all that I know about with regard to Artifactory changes. So yeah, please don't hesitate to reach out to the in the Gitter channel or stay until we switch the open discussion so that we can dive into this question. Thanks. Yeah, we have one last question. So for cloud images, I often see them become unavailable during jobs and then the job just keeps running until it times out and then they're killed. Yeah, we did that one. I think we got that one. So I've got some other questions that I wanted to ask our panel. So I'm going to shamelessly go to my questions. Is that okay? Maybe before we go that. So there was a question about post-belections. I can just show you how it looks like. So I have a test instance. It's basically the recent LCS release. And here you can see post-built actions like how they look like now. So the most of our post-built actions are compatible with the changes. They have been also updated. And here you can see that even message actions with a lot of configurations look quite well on the new interface. So yeah, there are still some plugins like let's say extended email plugin, which probably gets complicated even when we use these layouts. But overall, it looks quite good. So try it out. I don't have my next plugin installed. Excellent. Thank you. And just to remind everyone, there's some conversations in chat, but if you want something to be discussed, it needs to go into the Q&A window, not the chat window. Yeah. So the question, the most recent arrival was they're doing a lot of shared libraries right now for their Jenkins environments and reusing them. However, shared libraries, a majority of them, are not using pure groovy within the source folder, but rather steps within the vars folder. They have their own pipelines that run unit tests using JUnit pipeline testing framework, but because they don't have vars steps, they aren't able to pick up the tests covering any kind of percentage. So I'm not sure, is there a recommended way to handle how you develop pipeline shared libraries, I guess, is I think what the user is asking. And I'm not sure that we've got the right panel here, but let's go ahead. You don't have the right panel for sure. The description of right development approaches is what I would recommend. So, yes, write tests, yes, use vars if you want to have global steps, or use groovy classes if you want to delete them. Basically, both approaches are fine. There is no strict difference whether you use vars or whether you use classes except of encapsulation. If you want to see one of reference pipeline libraries, you can just go to the Jenkins project. We have a pipeline library here. So this is pipeline library, which actually builds Jenkins plugins and many other components. So here you can see the way out we use in the project directory. Jenkins files, we have Maven project, which basically triggers pipeline unit as well as some static analysis, etc. And here's our pipeline. So you can see here that we also use vars. So for example, build plugin, we involve in pretty much every Jenkins file, etc, the vars. But we also have some classes, not that many at the moment, but for example, document configuration management is stored as a class, which can be involved separately. But it's your choice how you use it. In our case, yes, we also store the majority of the code in vars because it's more handy. You know, the question was specifically about code coverage during tests. And I don't know. I will say that under the archetypes repository, there are a couple of samples of defining libraries. I can't remember whether using vars or source or both. That includes usage of the test system, but I don't believe that includes any system for code coverage. So what recommendations there might be, I don't know. I think you would have to go to the authors of that test library. Yeah, actually, it's talking about Jenkins CI slash archetypes, if you're doing screen sharing. Okay. This has a scripted pipeline and these both have these both have some sort of test infrastructure, but I don't think coverage. Yeah, we don't commonly measure coverage, though, in principle, pipeline unit should support that because pipeline unit doesn't really run Jenkins. It runs a groovy engine. So you can collect coverage from there. Yeah, I think the issue is with the vars directory, basically defining anonymous classes or, I mean, they're not actually anonymous and groovy. They are they are named classes in the default package, but perhaps the coverage tool doesn't work with that. Yeah, a couple of years ago, I had the similar issue with pipeline documentation, because I wanted to create a documentation generator for pipeline libraries. And somewhere in my GitHub, I have a converter code, which basically converts pipeline library, including class directory, the standard classes so that groovy tools can process them. If you're interested to reach out to me, I will try to find this repository. Thanks. The only higher level advice I would have is that if your code in the groovy library is complex enough that you feel the need for code coverage, then you should probably be working to simplify it and move most of that logic into external processes that don't require any complex setup to launch that's just run this Python script, something like that. And for the cases where you're genuinely interacting with Jenkins and genuinely do need some sort of logic, the use of a mock framework like the pipeline testing system is just going to be somewhat awkward because it's not going to give you all that realistic a view of how this is actually being used in the context of the system. So you might consider sort of higher level and to end integration tests at that point. There's not a single clear answer, I think. Yeah, for integration tests, we have two frameworks. One is Jenkins test harness, which you can use for testing pipelines, like we do for plugins. Another more advanced way is Jenkins file runner and frameworks around that. For example, Jenkins file runner test harness, which you could try, but this is a more complicated way and it doesn't fully cover all functionality. So my personal recommendation for common integration testing use cases, just consider Jenkins test harness, it works well. So basically it's the unit for framework with a lot of additional features like you can provision Jenkins, you can launch pipeline, you can evaluate restarts. Also, you can use test containers and other rules in order to provision environments. If you want to do it for the test, and this framework can be successfully used with pipeline libraries as well. So even if we develop it for testing plugins, it can be used for advanced pipeline library testing and Jenkins file testing as well. Yeah, many of our plugins basically just use it for testing pipelines anyway. So you're almost running out of time, but yeah, I will just show it quickly. So here, for example, you can see that we have Jenkins rule, which basically provisions Jenkins, and there we have a test somewhere in the bottom, which just emulates pipeline. I will look for that. You should have test, right? Yeah, here it is. So test pipeline compatibility, really simple example, which basically runs a pipeline on a Jenkins instance and verifies some results. Obviously, you can create more complicated examples with agents, with additional tools, environments, and you can also use configuration as code rules to automatically configure your entire system. The library plugin itself uses the same system to test pipelines using libraries to verify the different kinds of exotic edge conditions work as expected. Okay, and yeah, we have quite extensive documentation about this framework for plugin developers, which you can also use. Okay, so the next question, what will be the impact of the new version if you use Jenkins as code? Is it possible that we encounter errors with the plugins? And the answer is yes, it's possible, because when we talk about forms, yes, there are configuration forms, but actually there are many other forms. So for example, if I just start building these parameters, here's also a configuration for my class. And here, if something breaks in these plugins, the layout will be impacted. Another use case which you may experience is a pipeline snippet generator. So another test job. So this snippet generator, do you move for this snippet generator somewhere? If you just change the URL to top level to slash pipeline dash syntax, it will take us there without even going to, without the slash, yeah, there you go. And pipeline dash syntax. Okay. All right, we have to move to the site panel election. Oh, now that's interesting. Do you have pipeline installed? Oh, like I can certainly bring it up on mine. Well, whatever it is, just an example. So yeah, if you use snippet generators, not only pipeline, also job days, pipeline is YAML, then you may also experience issues because all these sections also create forms in order to provide your dialogue, which you would be configuring. Yeah, we'll find the link later. Just don't want to spend much time on that. Yeah, but I guess the, yeah, I'm not sure whether the question was about config is code of a job or config is code of the entire Jenkins system. But certainly if your entire system is being configured as code, then the same warning applies that you need to make sure you're using the latest latest versions of plugins available for the 277 line. Yeah, no, I was envisioning that maybe the question was focused on what if I've enabled extended read and intentionally not allowing people to do configuration changes at all in the system like Jenkins configuration. So Tim had done a bunch of great work on extended read. And that's now available in that case then. But that then is exactly what Oleg I think was answering to is that even with extended read, there are still presentation items in the UI shown to users that may be impacted by the configuration changes. And certainly the authentication changes need you to update your plugins. Also keep in mind that the configuration layout is just the first iteration and there will be more changes in layouts. For example, Tim has already proposed a pull request which changes run some risk page from tables to divs. So we will keep working on that to improve accessibility of the Jenkins web interface. And in the next releases, there will be more significant changes. Thank you. So Oleg you noted that we were approaching sort of our end. I've got questions but I'm okay with deferring my questions. I think we've got most of the question we've got the questions answered from the audience. Were there any comments that Tim, Jesse or Felix you wanted to share as we're as we're at this about one hour into the webinar Mark? No I'm good. So before we close, we will appreciate your feedback about the meetups so we can address feedback and future meetups, make them better and specifically interested to know what topics would you like to hear the next meetups because we want to organize more online meetups and any inputs would be appreciated. And if you want to present the Jenkins online meetup, please contact us because we are looking for speakers and we are happy to host any meetup related to Jenkins development, case studies and basically everything between those Jenkins. Thanks. All right. So I'm going to switch off the recording and then we're going to go live. Yeah. Thanks everyone.