 Welcome to the Jenkins platform special interest group meeting. It's the 13th of August 2021 remind everyone we abide by the Jenkins code of conduct be nice to each other. Agenda items that I've got for today include open action items Java 11 is default in our Docker images. Let me clear our Docker images, Jenkins interoperability newsletter coming from CDF, and a topic on Docker hans overview that don't know when we'll be able to schedule it, but just as a topic. Are there any other topics we should add to the agenda. So, Tim thanks for joining here are the topics that I saw for today's agenda so action items Java 11. Java 11. Interop articles and Docker agents overview. Anything else you'd like to add. No. Well, have you got so the Java switch over. So there's one thing about that the agent publishing is not working for. It's such agent and inbound agent I think. Okay. So that needs needs additional work to bring back. Oh, go ahead, Daniel. Just said hello, I didn't notice the meeting started because I wasn't in audio. Thanks to zoom. Sorry. Thanks. Thanks for appreciate your being here. Thanks very much. So, so we've got an issue then Tim you're noting with agent publishing for to the to the outbound and the inbound agents. Yeah, I merged changes to them a week or so ago and saw that no releases were published and then I had a look and no updates would be done in about seven months. Which I think coincided with the Docker hub changes and I think these were are on auto builds in auto builds probably stopped working. Okay, so now I assume that we can we can borrow from the techniques we're using in the controller at least in some portion to work on that. Yeah. So agent is working and agents more similar, but yeah. Oh, agent is working. Okay. Agent agent agent looks like just someone moved agent to building on trusted. Okay, good. And I think windows works and those repos so the windows ones is building on trusted but the Linux ones were using Docker hub builds. Okay, so that that we've got to change. Great. All right. Any other topics before we get into walking working through the agenda. Nope. Okay, so I had the action action item to open the jet pull request it's been opened, and requesting comments, we're actually executing on this plan right now, even without it having been accepted as a draft. We've been received from Tim Jack home and from Oleg Nanash of and from one other contributor and I think the proposal is currently up to date with those comments. I've still got the task to open a jet for Docker operating system support. And this is but we've got a code owners file now already. It's the jet is basically use code owners and the concept. We have one in the controller repository, but it's not yet fully expanded to cover all of all of the images that we have. We had an action item on plugin installation manager docs that PR is sitting idle now and I'm going to just pick it up and continue with it but it will be. I expect multiple weeks before I can approach that. And then is the hot one for me at least Java 11 is default in all our Docker images so the the story there is 2.306 has added JDK eight tags so there is a latest dash JD latest dash JDK eight, so that people could switch to use it it happens to be the same right now it's an alias for latest. Next week it will be an alias for it will be separate from latest. And because latest will become Java 17 or not 17 what did I just say that was a terrible thing to say latest will be Java 11 strike that seven Java 11 Java 17 is completely off the map for now. Agent images we need to switch from eight to 11 and the plan is to have that happen sometime during next week it certainly won't be exactly on Tuesday. We definitely want it before the LTS release August 25. Tim do you feel like there's risk or high worry about that as fitting that in given that we've got to rework the publishing process. I would say that well depends on whether someone's got time to get the agent one sorted. So they can control one to trivial but the some of the agent ones so Jenkins last agent is using Docker bake so it shouldn't be too bad. But the other ones haven't been swapped over. Okay, and conceivably, we could switch the publishing without switching to Docker bake. But Docker bake certainly makes it faster, easier, smarter, and more consistent. Yeah, we could the old scripts are very hard to understand at least for me not knowing them. And it's just probably not just well I don't think there'll be any point and not doing it because the old ones, unless unless you got Docker auto builds working. The only reason not to use bake would be if if you got the auto builds working again, and stayed with that, because there's no scripts for that those repos I would assume. Ah, got it. Okay, so switch switching to bake is the is the most effective path. Yeah, I'm assuming the tags were configured in the Docker hub UI on those repos. Great. Okay, thank you. All right, then we've got 2.303.1 August 25 will add a JDK 8 suffix tag and will switch to Java 11 as the as the default in the Docker image. Any red flags there are concerns from anyone of oh something that we we we should not go forward with this right now my plan is to go forward. Not a concern, but the states to be part of the upgrade guide and the change log. And we would not go there through the usual approach of looking at the core changes because it's packaging related. Right, very good point and I've attempted to already put it into the upgrade guide and the change log. So that's out for review, and would love to have more voices checking it let me put a link to that into the meeting notes. So it's, I didn't know I take it back I didn't put it in the change log I wasn't sure quite where to put it in the change like so Daniel maybe you can give me some hints there, but I definitely did put it as the first entry in the upgrade guide. I would consider it as a major feature on the top of the list or near the top. It just needs to make clear that this only applies to the Docker packaging. It could be merged with the existing entry that I just saw about the admin monitor recommending running on Java 11. Ah, okay. Alright, great. So I will, I will revise that pull request to include include that that's good. Okay. Mark revise the pull request to include it in the in the change log. Very good. Thanks. And Deraj Joda and I plan to collaborate on a blog post that will we hope to submit for for review before Monday morning. UTC, so that it could be reviewed, hopefully publish Monday or Tuesday in time for the weekly release 2.307. Anything else on the Java 11 transition. Okay, next topic then was articles do August 20 for the Jenkins interoperability newsletter from CDF. And I plan to write an article on my experience dealing with arm processors. Tim I believe you'd submitted a draft for review. This is a separate draft for. It's like pipeline visualization. With the pipeline graph your plugin. Yeah, which by the way I'm using more and more. Thank you very much. I can't tell you how grateful I am. It works. It's, it's delightful. I assume you want surprises or issue reports into the GitHub repository as issues as GitHub issues with pictures and ways to duplicate differences. Yeah, definitely. Great. Thank you. All right. And then we I've I've set a poll, trying to find a time when we could have an agents overview session with Oleg Nanash of unfortunately there are no matching times to fit my poll so the poll is invalid. I will. Oleg will share times that he's available. Based on his his vacation schedule and his family schedule, and we'll we'll look for it in the future. So the, the, I think it's a good idea to have that discussion, but my proposal proposed times for the poll just didn't work in the poll just didn't work. So there was on the Docker agent. So it was one thing that Damian suggested was that maybe we think about merging some of the repositories because we've got quite a lot of repos. And with Docker bake, we can kind of build them all together, especially the ones that build off of agent base so we can build the agent first and once that's built. We can build the other ones on top. I think it's I think inbound agent is the one that builds off it or SSH. Inbound agent I think. And I like that. I think, I think the intent there was discussion in the, in one of the mailing lists, but I think that's exactly the right thing to do because it gives us one more opportunity to accelerate those Docker builds and make them consistent. It would be a nice topic for this over agents overview session with Oleg to understand if he he identifies some reason why we should not do that. Excellent. Thanks. Any other topics related to platforms or things that we should be discussing today. So I joined this meeting. I think because like one or two weeks ago. I requested that the Docker build be sped up. I was with many image variants. I spend a long time waiting and then Tim informed me that this has already happened. So, because so I saw the meeting note meeting invitation earlier and I thought, well, that's a good opportunity for me to join and thank you for making the Docker build faster because waiting for that is always really something that I'm trying to deliver security fixes. This hasn't really been relevant in the last one or two months, but it will surely come up again soon. So thank you very much. Yeah, I mean, so they are the main waiting time is just on the is on the windows publishing. And that's because the windows base image is very, it's one large and windows Docker is small to pull. And then on the controller publish, which means that it that goes a lot quicker now. If the agent that runs on has a cashed. So if you afford every reason that running multiple builds, it's a lot quicker. We were seeing the other scheduled builds that we're kicking off where every single one of those was taking 20 minutes, and we're wasting quite a lot of compute. And that went down to three seconds if the agent was still around. But I mean, the main the main fix to the speed is to introduce staging and promotion. If security fixes an issue shouldn't really have to waiting should just be a button click. I agree, still it's a nice improvement over before when I regularly had to wait 4050 minutes. So, as I said, thank you for your work there. Of course, if you want to make it even better, I'm not complaining, but it's a nice improvement already. Yeah, I mean, switching six, whatever was five or six different variants to building at the same time rather than one by one, it's going to make a massive difference. And it did we had a Daniel thanks for highlighting it actually last week or two weeks ago Damien shared a an overview for us and we captured some of the choices in terms of why why bake is using HCL and why how that helps and how how the parallel testing helps and parallel image building helps and this was this is quite an amazing contribution from Tim and from Damien thanks Tim very much for what you did and and thanks to Damien as well. We get faster tests faster builds and now truly windows as far as I can tell is the prime bottleneck and then Tim, I think staging and promotion is a pretty big shift right that's a that's a notion of a completely different concept than is currently used in those scripts. Well, not I mean those scripts just care about a repository and credentials. So, you can point it to whatever repository you want. Okay, so we can use the scripts to do the staging and promotion implementation when we're ready. Yeah, and the scripts are massively simplified now I ripped out most of the code from the scripts as part of bake. Because all of that is encoded and declarative file now, rather than shell scripting. Especially it was nice and docker agent because I couldn't read that shell script was insane. Excellent. Thank you. Thanks very, very much. Yeah, the other fix that we did was I found that the tests weren't actually running on CI on the agent repository. There were tests in the repo but they weren't run and the tests actually failed when there was a arch Linux image was added recently and the tests weren't running and the tests were now failing since then. So Damien and I had the fun of fixing those. Yeah, now, part of me still wonders on arch Linux that's a different topic I think. Yeah, not sure if that should have been merged. That's a relatively obscure, at least to me that's an obscure Linux variant. Yeah, the reasoning given was something along the lines of being able to get as the rolling release and being able to get updates quicker, but yeah, I mean, there's not so many variants on the agents that's not such a big deal. Yeah. Okay. So, regarding that and I mean I want I joined to say thank you so I'm kind of changing course here, but the problem, a problem we have is is the image maintenance, right, because people. I noticed that because people send various security scans they do about outdated images and send it to the Jenkins security team and now we and then we need to handle them. And from my admittedly very biased point of view. It would be much preferable if we provided very few images or variants of images that we deem to be the best way to run Jenkins. Practices or whatever, you know, whatever it is and tell people well if you want to do something else you can always build your own images that are custom made just for you. And from my point of view, this would make it much easier for us to stay up to date with based images and various package related security fixes and all of that rather than having to adjust a dozen different variants all the time and keeping them updated. And I think we've, we've at least got a rudimentary beginning of that so the Java 11 changes will stop publishing the centos tag right. Because centos eight is effectively dead, and it will stop publishing the stretch tag on the agents, because stretch is Debbie and stretch is switched into its long term support mode and even that ends relatively soon. Now there's more work to do there Daniel much more I agree. So those are kind of low hanging fruit. The problem is there are so many Linux distributions that are actually actively maintained and are reasonable choices. And everyone has their favorite Linux distro. And so they submit a new variant for Jenkins because that's the one distro they're using. And I don't really see the point of doing that again I am very biased because I'm the one that receives the report says your package is outdated and then there's security vulnerabilities in the package nobody's even using. Make the image maintenance easier, especially given that we're historically at least not been great with continued maintenance without interruptions because you know people lose interest move on to a different project and then suddenly, you know, something that should be maintained isn't and you cannot fault the people for losing interest. But it does not look good in my opinion for the project overall. I don't really understand why we need so many different choices. The only thing I can see is should be a should be a reason why we need that sit like my architecture isn't supported on on it or it runs or this this image is so much smaller and we should recommend that as the defaults, or some other reason not just this is my favorite distributions. That's basically it and I mean the my favorite distribution there is one way to phrase it that almost seems reasonable and that is, it's my information security teams requirement, which is just business speak for this is my favorite distribution. But why make this a problem of the project right because everything they upstream suddenly becomes our problem. And so maybe, maybe it makes sense for us to consider a more opinionated approach they're saying we provide the image that we feel is the correct one. If you prefer a different one you're free to roll your own. Again, just my perspective as someone who admittedly doesn't care at all about the underlying system because it really doesn't matter to me. But perhaps something to consider. I think that makes sense. In terms of, if we were to take that approach. It seems like we would need to do a, at least an announcement saying hey, we're intentionally so conceptually let's let's take this what if we said. All right, we're going to stop supporting Alpine and stop supporting Centos seven and stop supporting Debian 10 switching only to Debian 11 or some some mix of those. There are certainly there will certainly be noise in the community we can't un-push those tags, but they would just stop getting updates right similar to what we're doing with these low hanging fruit So that could be a March 2022 LTS or go ahead Daniel. I think it's too early to talk about implementation detail when so far it's been the three of us. It was just, you know, something to my mind when I don't even know what it was thing. At that time, I was thinking about build times I also saw a Twitter thread about what it means for someone to do a poll request or request a feature be added, because that implicitly says, Yeah, this is now your responsibility to keep this working rather than, you know, my role in working with your tool and that sort of resonated. Now, most of what I've been saying applies specifically to controller images. Here's the thing we don't want builds to run on the controller in the first place. So you shouldn't probably shouldn't even use your package manager all that much because you don't need to install your tools because you're supposed to run on agents. So I see a use case for a greater variety of agent images to a certain degree that I don't see for controller images. So what I've been saying applies more for controller images than for agent images, but can certainly to some degree apply there as well. The interesting thing there is that we have a much, we have a smaller variety of agent images is at least on the default ones. Yeah, much smaller right. It's like DBN and Alpine and now, and then and then the other ones are just DBN and Alpine I think. Right, it really is only, I don't even got UBI in the agent images yet. Right, so it's just DBN and Alpine. Yeah. The problem is, I think that's a lot of people actually build on the controller which is a horrifyingly bad idea. But you know, the default way we deliver Jenkins makes this too easy to do. And if you look at the usage stats, the vast majority of Jenkins instances in executors, which are very likely the ones configured by default for the built-in node. So obviously we've done things like the admin monitor to recommend doing this otherwise and we've updated the documentation to explain exactly why this is a bad idea. But this might be just, you know, people actually building on the controller and I cannot think of a different reason to install Git LFS by default on the controller, which we do. So that would also be one of those things. If, do we actually need Git LFS installed by default on the controller, if you actually implement our best practices and do all of the actual building on agents. I don't know, but if it is not necessary, then we should not install it. The problem with that though is that the default one and the repositories is broken. It doesn't include newer features that get users or something. So if people download the default one from the repositories, it doesn't work. So we install the working version. And because of multi-branch, we multi-branch having repository on the controller, I think we have to have, I'd have to double check, but I thought we had to have LFS there. But your point is valid and it applies to LFS and many other tooling choices, right? There are several other tools like that where we ought to evaluate. Right, that's what I'm saying. If you implement the best practices and you don't need it, we should not deliver it because that sends the signal, yeah, just, you know, do whatever. And while we're at it, I think some of the packages grant other user accounts read access on the Jenkins home directory because that's a convenient way for people to do something like copy artifact plugin does. Which is also not a great default choice. I, but I don't want to completely high check this meeting. So this is a bit more than I actually wanted to say. It's just maybe something to consider for the future that if we as the project have certain best practices established, then we should that should also be reflected in the defaults that we ship. Right, good, good point. That's, that could be presented as the logical extension of. Yes, we're warning you not to have not to run agents are not to run builds on the controller. Now we're going to remove tools from the controller because they're not needed for the controller itself. And if you say you need maven, we say no you don't not on the controller. You say oh I need and your example get LFS if we can find a way to confirm for ourselves that we don't have to have LFS on the controller, then we don't install it. Yeah. Excellent thanks Daniel. Any other discussion there or topics we need to bring to the session. Okay, then I think we're at the end of our session. Thanks very much, both of you. I'll upload a recording I hope within the next two or three days I'm badly behind on recording uploads so refer to the meeting notes for now, recording upload will eventually happen. Thanks everybody. Thanks Mark. So