 Welcome! This is Jenkins Platform's Sieg meeting of May 23rd, 2023. Welcome! Today we have with us Damien Duportal, Kenneth Salerno, Mark White, and myself. Welcome! So today, on the Open Action Items, we'll talk briefly about something that does not progress, but that's not really a problem. It's the Docker container image for the Blue Ocean product. And we have to say at each and every meeting that it's not progressing, but someday it will eventually... And we have some progress on it. No way! Yes, we do, because there are now four approvals on the warning message in Jenkins Core that will allow us, that's the beginning of the step we need in order to kill that container image. At least for me to feel comfortable that we killed it with honor by telling people it's unsupported. So yes, there will be more changes needed. So we have several weeks before it'll be visible in a weekly and probably won't be dead, final dead until about the same time as we do CentOS 7. But we have the tooling now that we need to do the job. Excellent! Yeah, we have the tooling. As we should say precisely, we almost have the tooling to do the job when it's merged. Then we'll have the tooling and another pull request will update the tooling to do a little more. Sure. Cool, got it. It's almost a running joke, you know. We have to talk about container deprecation and then you come with some good news. Okay, too bad for my running gag. Now, two weeks ago, Damian, you told us about code factorization that you were working on. And I think I saw some more pull request this week and the week before about that subject. So would you have anything to share with us? Yes, the first phase that I described last week of factorizing between GDK version on each repository that has been already approved done, fixed, deployed, fixed again, fixed again. And it should be okay on agent and inbound agents, on SSH agent as well. So we were able to decrease the amount of code of three times. So that will ease the factorization. So now I'm entering the second phase that should be finished until next SIG meeting that merging inbound agent inside agents. We keep updating both images, Genkin slash agent and Genkin slash inbound agent, but they would have a release cycle that should be the same for both when you have a new change that will build and deploy both images at the same time. That's the scope I'm targeting, which does not change anything for the end users except that the source code of one of the Docker inbound agent repository once walking and release at least one or twice should be put as archived and deprecated and everything should be on the Docker agent repository. That's the target for the next time. I have everything ready right now. Walked a lot on the windows to link, but yeah, that's the status and then we will do a report and I will bring on the table the proposal for the next step after this is done. But right now that shouldn't change anything. Cool. Good job on that. That was definitely necessary before merging everything. At least that's my understanding, which is very partial to say so least. Now you have also worked on alpine images so we can use update cli and gelling with me. Thanks a lot for your guidance, your patience and your time. So we're progressing. We've already done what was it Docker agent, which is now using update cli and we're hoping to fix SSH agent sooner or later. We'll see. Mark, now back to you. CentOS 7 early end of life. I know you'd like to get rid of CentOS 7. So what are the fresh news? So the poll request has four approvals not yet been merged, but with four approvals, I think we're very close to getting it merged. And so the important dates for people here are to be aware that users who use the CentOS 7 image containers will see this warning message pop up already. If they're running weekly, they'll see it beginning May 30th. So roughly two weeks a week from now users who are running the long term support release who are running weekly on the operating system without a container if they're on CentOS 7 will also see it. Now LTS, they won't see it until August because that's when that's the first opportunity for 407 to be visible to any users of LTS. We can certainly actively promote it, but even when they see it, they'll have three months warning before we declare official end of life November the 16th. Sounds reasonable. Yeah. Good news. Thank you. So we still have to replace CentOS 7 usages in our containers. Yeah. And that's a complicated one, right? Because one of the things we do is the base JDK that we receive from Eclipse Temeron is based on a CentOS 7 image. We use it in all sorts of places. We and all we're doing is we're using the JDK from it. So we're not actually using that operating system, but it feels odd to have a reference to CentOS 7 in our container definition when we don't want to support CentOS 7. However, finding the alternative choice of container from Temeron is not as clear to me right now. So, sorry, I didn't get it. So is the JDK by Temeron built on CentOS 7 or are we extracting a JDK built with whatever operating system, which is part of the CentOS 7 image? So I don't know how to answer the first question. Is it built on CentOS 7? Because I don't worry about how Temeron builds it. That's up to them. But they definitely package it into a container whose name has dash CentOS 7 on the end of the container name. And we use that packaging of that container to get Java 11 so we can run J-link from that image. And so we're just using their container as a packaging step in our construction of our final container. But we use that not just for CentOS 7. We also use that for Almalenix 8 and for the UBI 8 image. And it really makes sense in those cases because CentOS 7 is a predecessor of Red Hat Linux 8. But again, it needs more work. Just a reminder is that the risks are really low of changing as soon as possible from CentOS whatever version for the Temeron extract to whatever UBI Linux because the J-link step will take care of resolving the dynamic runtime dependencies in Slashlib. So the generated package that we use and then copy, which is stripped from the debug symbol and multiple optimization should also be completely compliant if you have exactly the same operating system on both images. So I don't know for Almalenix but for UBI since I remember, I might be wrong but I'm almost sure I remember seeing an Eclipse Temer in UBI definition. So that one should be quite easy. The Almalenix is more the one I'm not sure. Well, an Alma is just a relatively simple derivative of Red Hat Enterprise Linux 8 so it's like UBI. But the UBI image I've seen is UBI 9 but I like your observation Damian and I think it should be easy enough to do the assessment and the test to assure that yes, the image we get out of a change of container looks the same actually as the image we got before we switched from CentOS 7 to the UBI 9 base container because I would assume Temer is probably not doing a different build on UBI 9. I bet they've got a single build that they package on the multiple Docker containers. Okay, we can look at that just point. We have another solution which is stop relying on the Eclipse Temer image and instead download the GDK from Adoptium website with checksum control and we can keep the multi-stage build now meaning from Almalenix, whatever image you download and you run g-link to strip so you keep the same pattern to have to gain this precious megabytes and that means you will be sure with that case of being able to not depend on the ability of Eclipse Temer information to deliver Docker image which are always published later than the Temer in itself. Okay, so you would be okay if there were a switch to still use multi-stage builds but use one of the one of the phases of that multi-stage build would be download the binary from the distribution from Temer and unpack it and then run J-link. Exactly, the only pattern here is that tracking the version of the GDK might be a bit more touchy. That might require switching absolutely from Dependabot to Update CLI or Renovabot because I don't know if Dependabot is able to get exactly while Update CLI and Renovabot are. So that's the only prong answer. Yeah, that's really interesting. I must say, but I don't think I understand everything. When you say our containers mark, what are these? Jenkins controller. Yeah, the Jenkins controller. Sorry, I should be more precise. The Jenkins controller container definition uses a layered approach where it first brings in a layer that it calls something like Java and that layer is the thing that it uses to copy in the JDK and then it runs J-link on that. Okay, thank you. It looks like what we have in the agents. I must say, if I'm not mistaken, Damian. Yep, there is an opportunity here to see we have a common foundation for all the images here. That's something to have in mind. I don't say we have an immediate actionable because that's quite some issue, but still we could think about that, but yes, you are correct. That's the same thing we do on all the images. Yeah, could be reused. Yeah, because I've been working lately with update P and so on and the multi-stage Docker for the agent. So I don't think I'm able to do that, but maybe I could have a look at least open an issue so that we don't forget it and then maybe address it later on if you don't mind or if you want to open the issue or do the PR. Anybody is welcome. Of course. Thanks a lot, Mark. That was the explanation was much needed for me, at least. Thanks a lot. And then so we'll have a block person on things and there was seven end of life in Jenkins. And I guess you already have it all written ready to post. No, but, but I will write it with pleasure. Yeah. Why is smiling? There will be there will be there will be serious joy while I'm writing it. How about that? I don't get it. Okay. And then we have a few other petering system and of life near us this year. So Alpine, Fedora Ubuntu, Alpine 315, Fedora, and so on. So I guess all of them, I've had a look lately at your pull request mark. And I think that all of that will be handled by the code that you provided, which is not yet merged. If I'm not mistaken. Well, the, so the, because there's a six month notification window by default. That means beginning with 407 May 31st. The top five of those will already immediately begin seeing warnings. So Alpine 315. Now we don't deliver any containers based on 315 anymore. But if someone's making the mistake of running on Alpine 315, they'll see it. And we certainly do deliver to people who are running Ubuntu 1804. And they will begin seeing that with, with the, with next week's release, assuming that pull request merges. Okay. I won't be naming names, but I have someone who always tells me friends don't let friends run latest, you know, so people may use a fixed version, but maybe they don't have a bit key or dependent about whatever. So some of them may have the surprise from 407 thing. Oh my, I'm using a deprecated container of some sort. That's the, that's our hope. We would really be pleased if people saw the message and actually took action on it. That would be a really great result. Yeah, I confess that I run N minus one Fedora for that reason. I'm always six months behind. And so long as you, so as you keep that pattern, it's a healthy pattern. It's when you become 18 months behind that it really hurts. So you become 12 months behind that it hurts. Right. And minus one with the latest patches. Right. Cool. Thanks. By the way, Kenneth, because you were talking, thanks a lot for all the work you did for PPC 64. Because your controller PR finally got merged. Since the last meeting. So congrats on that. And welcome on board. Then I've got what, yeah, go ahead, please. Oh, thank you. Thanks for all the support. Oh, thanks for coming. I'm giving some help. I would say in French. Now we have a list. A la préverse because we have lots of updates on the darker images of the agents. So I saw lots of releases for the CCH agent. Lots. Meaning two. I think. So there were lots of fixes about windows. I saw on it. For example, Damian, you do. You did a lot of work regarding JDK JDJ. Sorry. And of course we bumped things to depend about. Debian on a more recent version for the Docker agent. Mark, you'll change something. So that we get the right version number in the all pine container. And then Damian and I took. Your work and change it for that. Now we have the all pine version. Updated things to update clip. So that should work now. And by the way, I think somewhere in here. Yeah, we have now the alpine linux version to three 18. Thanks to our combined work. That's pretty cool. Kenneth, you got upgraded as code owner for the Debian Docker file. Congrats. Yes. And we also had some JDK version tracking working now, which is pretty cool. And for the Docker inbound agent, the parent image agent has been bumped and bumped and bumped. So now we are having the 15 version of the parent. And Damian, once again, you made some work about JDK factorization though. Now all of them have the same JDK definition. They all are working with the same tag. Anyhow. Anything to add? Go ahead. Anyone wants to help. I've tried to open atomic issues for the SSH agents about different elements that made improvements. I will try to do the same each time I see something because that will help. I wouldn't say beginners because that's still most of the time complex issues, but any maintainers or people with Docker skills could get started if they want to contribute on these elements. Yeah, I wouldn't say being a beginner would be a major issue because there is help from the community nonetheless. I took an issue that was way too big for me, but with your guidance, for example, it's progressing and I hope to get it merged in a few weeks from now. So all is possible. I don't want to use your time. So if people don't know anything about Docker and Jenkins, of course, that's not a good idea to start trying to resolve an issue. But if you've got, if you know something about Docker and Jenkins and the agents and so on, and maybe about update CLEE, that could be done. That one could be a good first step for beginners or people who want to get started. We still have dependencies that are not tracked on each image. I'm thinking right now on the Git LFS version, for instance. I don't want to go on the Git loop or because that one is a complicated topic. The Git LFS, for instance, is an example of dependencies that are installed on all the images, but that should be tracked with a system like Renovabot or update CLEE already in place. Yes, you're right. And I've seen an issue about dedicated tracking with update CLEE, for example, so that's something that could be done by somebody who doesn't have that merge. Absolutely. Anyone interested can get it. Second element that I forgot to put on the table is that we have one major change on the way we deliver the controller image. Now it's based by tag. We create a tag and the tag triggers the release. We don't have anymore the script that do things because it was a pain for the infrastructure port. And right now the question that we discussed earlier today on infrastructure SIG is if, whether we should give permission of release members to be a maintainer on the Genkin CI slash Docker repository on GitHub, so they could create the tag manually, which is the case today. Or if the infrastructure team can work on automating the tag creation, that's the road we want to take. But maybe we will fail or we won't have time. In that case, we will need the maintainer to create the tag at least once a week after the weekly release and the person in charge of the LTS release of the code. That will be a temporary measure during the transition. And second thing, we realize that since... Yeah, sorry. Raising hand, sorry Damien. Before we go off of that one, we've got an immediate situation with the 2.406 release. Alex and I discovered that the change that I had preemptively merged four or five hours ago was necessary but not sufficient. And therefore we merged a new change to 2.406 or to the current master branch and a build has just started. We are therefore probably three or four hours away from having that build complete. The challenge is three or four hours away from us is a bicycle with me on it. So I won't be ready to apply that tag. I will apply the tag after I get back, but it could be five or six hours from now before that tag is actually applied. Don't panic if in the worst case, the 2.406 container image waits until tomorrow. Thanks for the notice. Yeah, that's one of the major issue we have right now. Sorry for this one. But that allowed us to deliver away more in months. I will come back on the Jenkins controller next design meeting with two major questions for your alert. First one is to start having weekly branch and an LTS branch. Maybe not named like that, but to have the Docker image build base just to be sure that we don't update the same thing at the same time because LTS need to be LTS and need to be treated even on the Docker image. And the tagging version that that will be another topic, that will be another topic. Second major topic I was reminded earlier today about the Jenkins CI Docker inbound agents plural form repository. That's also some homemade images for agents with some popular languages such as Ruby, Go, etc. There were there weren't really a consensus during the past year about that repository, but no one is maintaining it except one user that did a pull request a few months ago and the user reminded us that this change wasn't delivered to the Docker hub for good reason and infrastructure. We are not deploying these images on trusted CI since at least one year on the half. So we will take care of delivering that change now because the user did a contribution that we merged. I want to raise the topic and people to think about I would want to include in the depreciation program that we are putting in place the depreciation of these elements because it's dangerous to run these images at home. No one is maintaining them. We don't know the version. I'm not even sure Python has been removed for the version too. It's too much work and no one stepped up. I already asked people complaining that we wanted to deprecate and then suddenly when we asked for maintainer everyone started to be silent. So I think it's dangerous for the community to keep these elements and I would want to advocate for either by default deprecating them unless someone stepped in to maintain these elements and by maintaining that mean ensuring that we keep the languages up to date even go along. GDKs in Boundary John Burrante, Ruby, Python, PowerShell and there is another Terraform I think. So I propose to deprecate it. Think about that. One of the proposals I had was to write down a Genkins.io webpage maybe part of the GSOC program or not but at least start with the read me here that say if you need these images then here are Dockerfile example that you should follow. I mean it's still the Dockerfile out there. People can build themselves and host themselves but we shouldn't have to worry about these images because they have opinions that no one use them almost. Okay, thank you Damian. No one uses them almost makes me think of the last subject but I don't think we'll have the time to dread that. I'm sorry, you opened the door. So on those container images, what was I... Oh, we have a page already on Genkins.io that talks about Java requirements, that talks about Windows requirements, that talks about Linux requirements. I think it might benefit us to have one that talks about container requirements and what policy we have in terms of containers that we are willing to watch and containers we're not maintaining. And our expectation that containers that the Genkins project delivers have active maintainers because containers without active maintainers are not healthy. Same idea as the plug-in ill score but for containers. Yeah, I agree. Do we have an exhaustive list of all the containers that we deliver in supply? I can get you one based on the Docker HubExport. We can look at the Docker HubExport and see from there. And I think in that article Damian, you want to link to the hacking doc of the Inbound agent because this way you don't have to rewrite those instructions. Yes, good point. Absolutely, that will avoid duplicating information and will be always up to date. Exactly, and if we change our steps in one, we remember it's all in one place. I have to say if you're interested to help, I wouldn't say no because as you can hear, my French accent and you can bet that my French writing in English is not the best one. The mark can confirm. But yeah, having people English natives would help and also with a different point of view because things are in my head. I'm trying to structure them and it's not very well structured within. So having an external someone outside my brain would help a lot for this one. Yeah, I can help you. I think if we work together to put together that matrix of what's supported and what's not, and then I could help you with the wording around it. That would be wonderful. I'm going to start the exhaustive list then, which will be the same subject as the Docker Hub stats. I was able to see that the export that I can get as an infrastructure administrator on the Docker Hub are anonymous. So having a shared spreadsheet where we put the CSV data every month, we have two kind of data that can be get the statistics per image and per tags for each image of the downloads. So this information could be interesting to take decisions. So my proposal is to start the document and to put everything and I share it with you so we can start reviewing it and discuss that in the next meeting if it's okay for you. And that includes the exhaustive list of images, of course. That could be a starter for us again. Yeah, good idea. Great. We're already one minute late. Would anyone have anything to add a question to ask, comment something? Actually, I need you to rewind on the notes because I inserted something earlier in the notes after we passed them. Okay, let me know which one. Oops, back up just a little... Whoa, right there, just above what's been done. It's the in progress thing. Right there, whoa. Nope, nope. Sorry, further down. After CentOS 7, end of life. Remaining work. What has been done, yeah. Okay. Proposal to switch Alma Linux container from 8 to 9 is open. Oh, cool. Damian and I have been discussing it in the controller because people should be aware it would be nice to include that in the LTS change log with 2.401.1 next week because LTS change logs are a great place to put that kind of thing. But that would mean we would need to merge it before next Tuesday so it'd be visible in weekly and then visible in the change on Wednesday as well in LTS. So just be aware. By the way, Mark, because I wasn't... My message was confused. But my proposal was to switch that pair into adding a new Alma Linux 9 first because that one can be merged quite quickly while removal of the existing Alma Linux 8 might trigger more difficulties. That could be slowed down. While the first one, as soon as we have a new version now we don't have the issues that slow down, can it work? Now that we have fixed that problem we can quickly deliver a new Alma Linux 9 and start testing as part of the weekly process. So the LTS could be focused on the removal of the 8 and if we don't have the time then we can target the next LTS for the removal without breaking people's usages. Mark, what packages are people using in Alma and CentOS that they're not getting in UBI? What's the purpose of those? I truthfully don't know. The fundamental story is why bother with Alma at all but that's a bigger question. That's what I'm basically asking. Unless there's some packages not in the UBI repo that they need but binary compatible why not use UBI? Exactly. It's a valid question, Kenneth. And it's a question for which I have no answer yet. That's a bigger conversation about shall we end Alma Linux completely? We're ending CentOS 7. We've got an end of life process there. I don't have an end of life process for Alma Linux that is container specific but we could easily do it with the tooling with a little bit of an extension of the tooling that's about to merge to Jenkins Core. So we could pretty readily declare to people Alma Linux container is ending life at this date in the future and they would then be warned. Okay, you need to get off this. Go to UBI. Which answer the question about Eclipse Timurine? We can use UBI Timurine package for Alma Linux that should be binary compatible. Yeah, except that the Timurine package is for UBI 9 and our Alma is currently Alma 8. So long as they don't rebuild so long as they don't actually build for Linux 9 we're probably okay but it would need testing to be sure. I missed that important pile. Thanks. I forgot. Okay, anything else Mark? No, excuse the delight. That's all. Fine with me. I don't have a bike ride to do so. Your bike is waiting for you. So I think that's a wrap up. Thanks a lot for coming to this meeting. The recording should be available from 24 to 48 hours. See you two weeks from now and until there, enjoy Jenkins. Bye-bye. Thanks. Thanks for the work that everyone put there. Bye-bye. See ya. Bye.