 Welcome to the Jenkins Platform Special Interest Group. It's the 30th of July. Remind everyone that we abide by the Jenkins Code of Conduct in our meetings. So it'd be nice to each other. So today's items that I've got, oops, open action items. Open action items here. Java 11 as default in all our images, pending Docker changes, and that's it. Aditya, is there anything that you would like to add to the agenda? No, no. Okay, all right. So first action item for me is that I'm going to open the Jenkins Enhancement Proposal today. This document has been out for review for an extended period. It's still got a lot of things that I need to add or I've got an open question from Tim Jacome. That's good. So that we can address there. Then we've got an additional JEP, which won't happen as today that's proposing Docker Operating System Support Pattern. And, but we've already got it started thanks to one of the users who started using the technique. We've also got Plugin Install Manager docs and the PR that's pending to replace install plugins.sh content with our Java program. Any questions on the action items? No. Okay, so then the next piece of the story is Java 11 as default in all of our images and that's the Jenkins Enhancement Proposal that is this thing right here. So this will be submitted later today as a pull request to the Jenkins Enhancement Proposal GitHub Repository and we'll start the review and discussion process. And let's see, any questions there? Yes, so let's just say that, okay, everyone gives a thumbs up and plus one token. Now we are going to have this. So what is the process and how will all the Jenkins images all start using JDK 11? So will it be like a new update is available and then everyone will download that update and use it? So how does the process basically, what is the flow of the whole thing? Very, very good. So the proposal is first we will, so the images that are changing is those images which do not include a JDK explicitly in their image tag. So for example, 2.301-alpine, if the new version, and now let's say we will release next week 305 and then 306, so this number should actually probably be 307. So 2.307, the previous week, 2.306 will deliver JDK8 inside the Docker image. Then when we deliver 2.307, we will deliver JDK11 inside the Docker image instead of JDK8. And the idea is that two to four weeks, two weeks before the LTS release, so mid-August, we will make the change in weekly. We'll watch it for two weeks and then we'll make the change just before LTS so that the LTS release August 25 has Java 11 in its base images. And this now I need to change from the 289 to 2.302, like that. Okay. So Aditya, did that answer your question? Is that we will transition, we will begin the transition in weeklies mid-August and we will then complete the transition end of August with the next long-term support release 2.301.2. Yes, yes, it does answer my question. This is what I was looking for. So I had a follow-up question. So from the user's point of view, it's just that it got a new update. So they do not worry about the rest of it. But there might be a case where some functionality might break, right? So if some software does not support JDK11, so how are those cases being changed? Exactly, you got it. That to a user, when they transition from 2.289.3 to 2.302.1, they will also implicitly transition from Java 8 to Java 11, running their controller. Now that may surprise them. That's why we're going to communicate it. We're going to blog about it. We're going to inform them because those changes, it's important that they know this is happening. And even with all of our communication, we still expect there will be people who will be dismayed or say, oh, I need Java 8 for this specific reason. And so one of the proposals here is that we will add image tags for Java 8. So that if a user says, oh, I cannot upgrade to Java 11, they will have a place that they could change their Docker file or change their Docker image they're using to instead use a different image. So they could use latest-JDK8 or LTS-JDK8. That's amazing. Yes, that answers my question. Okay. Thank you. Now there's a good question here from Tim Jacome. He asked, should we retire the CentOS images? And it's a valid one because CentOS 8 is no longer maintained in the way that it was maintained previously. CentOS 8 has become a very different distribution and updates are no longer arriving to CentOS 8 in the same way they did before. Therefore, it's becoming out of date. And as it becomes out of date, that's a risk to our users. So Tim's question about, should we retire those images is a very good question. And I'll be addressing that because I think he's right. We should acknowledge that we're going to retire it and we're going to switch to using a thing based on a Linux distribution called AlmaLinux, which is an open source or a community-inspired distribution based on the same concepts that CentOS used to use. Okay. Does that address your question? Yes, it does. Great. All right. Hi, Damian. Oh, is Damian here with us? Yes. Yes. Oh, very good. Damian, we're going to have you describe. That's excellent. We're just about to get to your topic. Okay. Sorry for being late. I mixed up my time zones. I took it was one hour later. My apologies. It's wonderful that you're here. Thank you very much. You couldn't have better timed it. All right. So I was just describing the Java 11 transition that we're going to be doing. And it's described in this draft Jenkins enhancement proposal that I'll be submitting today. So Damian, I'll rely on you and others as part of the review team to look at this thing next week and we'll have discussions about it. I'm sure Tim Jacome today has provided some good feedback. I've got some things that I need to tune in it before I submit it as a pull request. So it's an attempt to describe what we should do. And then we'll plan to implement it over the course of the next few weeks. Okay. All right. So that really covers it. So the next is pending Docker changes. So Damian, if you'd be willing to share with us what's happening in Docker changes and I'm going to just mute myself and type. Okay. So we did two main changes in the building process. The first one is regarding the build step. So we have a build process which is quite straightforward. If a contributor open a pull request on the Docker repository, Jenkins CI slash Docker, then there is a pipeline executed on CI.genkin.io which first built and then test all the image declination both Linux and Windows, all on Intel as for today. The build part was a shell script. It was iterating through a list of images and building them. And then the test part is using bats. So I will go back on that one and iterate over the same list and run the test harness on each one. That was before our changes. So the first step is most of the work on the build part has been done by team while I worked on the test part we split the, let's say the major person and we use each other knowledge for reviewing. So always two persons on the knowledge is shared and not only one person. And we had a bunch of review from Mark, from Alex Hull and from Olivier at least. So build part, what team did was using Docker Buildix extension, especially the new command which is still quite on the edge, the command Buildix bake. So first what is Buildix? Buildix is a plugin distributed on most of the recent Docker engine distribution on the city on both Linux, Windows and Mac OS. If you have an earlier version of Docker or if you don't see the Buildix enabled on the Docker info on your Docker engine, it means you need to manually install the plugin. And the plugin is a single binary to install on the client side. So there is no change on a remote Docker machine. You don't need to restart Docker. It's only a client side plugin. That plugin has underlined on the official Docker documentation is a new implementation on the way that Docker Build commands are implemented. Initially it featured a better cache implementation which mean faster cache and faster builds but it also provide additional new features that should solve some of the old problem that Docker had. So here we use Buildix not only for the improvement in performances that was just switching to Buildix without changing anything else was already 10% faster for us in the CI process. But we use Docker Buildix bake which will read the manifest file that can be written in G-Zone as a Docker Composier Mall or the choice we made with HCL the HCCorp DSL language used for Terraform and backer now. So you can find that the root of on the repository a new file named Docker Build, no, Docker-bake.hcl. Why did we chose HCL over G-Zone or YAML? G-Zone does not allow commands. So let's forget about G-Zone. And YAML has its own quirks and given the, let's say the heated discussion that can be over YAML, we decided to not go that way. Personally, I don't really care. YAML allows commands but HCL allows something really powerful variable injection without templating. So HCL seems the first class choice especially when we have a bunch of images like on our repository. It's a monorepository that generates a collection of images. So by using HCL format, it's easier for us to inject a single tag and have integrated variable management inside a file. So what is the goal of that file? The goal of that file is to be a manifest a recipe that lists exhaustively all the images we expect to build on that repository on a single place which is Docker understandable. It only depends on Docker and Docker is the only actor that should be able to read that one. It allows us to specify all the build arguments that we usually have on the shell script. So Docker build allows to pass build arguments to the build that can be used then as variable inside the Docker files. It allows to specify the Docker file location. For instance, you have Docker file.windows for the Maven image or something. When you have a bunch of Docker file on different directories, you can have one on three that point directly. And finally, you can add multiple name for a given image, which means if you build the Docker file which is on the directory hate slash debian, then you can specify that the image build there will have different aliases. It will be Jenkins slash Jenkins column LTS GDK hates. And it will also be Jenkins slash Jenkins column debian dash LTS, et cetera. You can add as much aliases you want if you want to have different naming conventions, especially in our case where we have different dimensions, the operating system used as base OS, eventually the architecture, the GDK and the main line of Jenkins, weekly, LTS, latest, et cetera. So it allows us to put the naming matrix in an exhaustive and declarative way. So as we saw during the publish, we forgot to tag during the change. It was easy. It was one tag to add on a collection and that's all. And we just run the build again instead of a bunch of shell file with iterating over loops and another part of the list and iteration on the Jenkins file, et cetera. So the goal is to have something that is portable and repeatable. So team changed all the build process. We are still using some shell scripts, but the goal is to have as much information as possible on the Docker bake. And the rest is only calling the Docker build xbake command that will parse the file. And based on the entry parameter, we'll select either all the images or just as a group of images. The group are internal. We can do whatever group we want. It's only matter of adding a group inside a file or we can target only a specific image and filter out. So that's the main work. The good positive effect of using that is that it accelerated even more the builds because when you run Docker build xbake, Docker by default will try to parallelize every images. So whatever list of images you ask him to build, it will analysis and parse the Docker file, the multi-stage Docker file, see if there are images depending on each other and it will create a tree and then it will try to build in parallel. So while it's pulling an image, it can already run a run layer. So the efficiency of building with build xbake is really impressive. We're able to put down the build of all images on the same machine size from twice to five depending on the machine power. So five time faster on the best cases. So the two time faster was on our CI machines and the five time faster was on team and high machines. So team has a Mac M1 machine and they have a MacBook Pro high-end and on my Linux machine I was four time faster and on my windows it was free time. So every platform benefited. So that was a good first step. Next, the second part is related to the test. So we did first the build, we deploy the build and we did a full weekly release with that new build xbake. So the idea was one pull request we didn't want big bank changes, first one main major change, then one weekly release and then we can iterate. So after the first successful weekly release using build x, so we kept the same images, no more feature only changing the way we built, no functional changes. Then the next step was related to the test step. So the test step was almost 13 to 15 minute at least for each image. Even sometimes slower when the build was picking a machine that was already used with no cache or with low power depending on AWS or Azure supervisors. And we saw by reading the test different things. First, we were using an unmaintained version of bats shell framework. So we had to switch from the original project to the new maintained project. That's the initial author of bats say, okay, I give that to a community and they switch locations. So we updated bats and while updating bats we discovered that bats is now able to parallelize the test with a flag. You can parallelize per test file or per single test. So we said, oh, initially we wanted to at least run optimize the shell script. So that instead of iterating sequentially on each test we wanted to initially say, okay, let's run all the files in background and then the shell script should wait. But with that flag bats can do that for us. The only requirement was using new parallel. So we had to install this on all the build agents. That's the only requirement. And we have tuned the make file if you don't have that binary. It's just all the tests will be run sequentially on your machine. If you don't have parallel, it's automatically detected. Otherwise it will run two tests per virtual CPU on the Docker engine machine. So it's a kind of automatic and you can disable that with a variable if you see issues because maybe there are side effects that we don't code earlier. So if any contributor or CI maintainer say that, there is a variable to disable the parallel testing. So the consequence on the test is that almost all the tests started to fail because we had a lot of tests with duplicated logic or tests that depended on each other on a given file that was for reasons. This is how the test harness was designed initially a few years ago. So after a discussion with a few members, it looked like that in fact, there were no reason to keep the sequential because most of the time, the sequentiality come from a test was building a custom image. So it's like, okay, I have to test Genkin slash Genkin cell TS alpine that has just built with BuildX before. And then I have to ensure that I can build that image but it was already done. Then it was customizing and building a new image from that image with some specific test content and then running a container, eventually two. But each of these tests had the same pattern. So we said, okay, first we don't need to ensure that the test has to rebuild the image. What we did is that instead of having one machine where we build all the images and then another machine where we test and rebuild all the images, we shifted and I said, we have one machine for one platform. Let's say Genkins for alpine with GDK8. And on the same machine, Docker BuildX was only building alpine and then run in parallel all the test harness. So just stop building from scratch again during the test phase allowed us to gain two to three minutes per test. On 80% of the test. So we went to 30 minutes to six. And then by revamping the test and ensuring they were all independent from each other. So that was a bunch of shell script to do. And we cleaned up some tests that were making no sense. The sense that they were written, they were making sense when the hotel wrote them a few years ago but now they were duplicating logic or even testing nothing. So we removed these tests. That mean we had to remove 20% of the tests that were unused, false positive or not optimized. And then we cleaned up the rest. And so now the tests phase for a given image depending on the image is between one and two minutes and a half instead of the initial 15 minutes. So that was quite a huge improvement. If you disable the parallel testing it was, it run around six minutes. And finally an additional benefit that was quite an issue. And that was the reason why the test were not running parallel before. With shell script you were able to completely exhaust the resources in the machine if you throw a bunch of uncontrolled tasks that are quite heavy on CPU mainly then the CPU start to be over allocated. And we saw the consequences that some agent were cutting the connection to our Jenkins controller. And if you have a Mac Intel you know what happened it try to fly basically with the fan with bats and the parallel testing. It's able to control the allocation of the resources and to manage the pool of tests mainly based on the amount of CPU which mean you don't need to risk any resource exhaustion. Everything is managed with your kernel. And so there is no MacBook Pro or laptop train to run and no more disconnection of the agent. So stabilized pipelines. So yeah, with the combination of these two we had to revamp most of the make file. So there might be some side effects. We saw some contributors seeing issues that we tried to fix on the go. There might be still some. So if you want to contribute and see issues with make file don't hesitate to open an issue because we might have broken something. I'm not a machine. However, most of the heavy lifting on a make file now is how to retrieve the name of a given image from the Docker BuildXPake result and pass that image as a parameter to the test. So the test can reuse the correct image. That was the most complicated part that involves exporting the Docker BuildXPake configuration as G-zone on the studio and parsing that. But outside this, that was quite easy and we are really happy with what we learned and the results because the time for the Linux images went to build on test where at 20 to 25 minutes we are under four minutes now for the combination of both. So that's quite an improvement. Five times faster, let's say. Now the next step will be same for Windows. We're not sure about the strategy and any contributor help is welcome. Windows is building and testing sequentially for now. So we have to wait and find a way. We saw two roads that could be possible from here. The first one is naive optimization using PowerShell scripting. It's already a PowerShell iterating. So trying to parallelize where PowerShell is way more powerful than Shell in the ability to control parallel resources in background provide more features. So that could be a first step. But we need contributor at ease with PowerShell scripting. The second roads will be trying to run but on Windows because now Windows between Siegwin, Git Bash and the new WSL system. It should be able to use Docker Windows container built and tested from a Bash command line. So maybe we could directly benefit from the rest of the work. However, Docker Buildix is not ready to support Windows platform right now. So yeah, that's the state. We need to spend more time experimenting on that part. But there are some time that we could gain from here because now the Windows part is the slowest. That's all, don't know if you have question. What an amazing result. Thank you Damien. That is absolutely exceptional. So yeah, I am delighted with the results. One of the questions for me is, okay, this was for the controller. We've also got inbound agent, outbound agent and agent images. And I believe we've got Jenkins infra images. Do you have an insight there? Are we considering making the change on those or does it not make sense to make the change on the agent build process? Yes, for the Jenkins infra where we have a collection of images, we are going to apply that change. However, regarding the Jenkins inbound and outbound agents, we will have to discuss that with the community. Because Jenkins infra is something that we use for our own use case. So we can experiment and we have a bunch and we will have some the same, let's say a speed improvement for sure, given that we have a collection of images. In the case of inbound, outbound, I don't know. That has to be asked. Do we want to move all the images in the same repository because there are dependency between each other? The goal is between a change on the base image and when the change is released and deployed in a Docker hub and available to end users. What are the steps and how much time do we have? And here that need to be mapped and then see if we can optimize this. Okay, I hadn't considered that. So there is an attribute of that, that today's situation with three separate GitHub repositories, one for the base agent, one for inbound agent and one for outbound agent might be better suited by Docker build X bake in a single repository. I think that's what you're seeing. But then we've got to deal with the different tag, tagging patterns or naming patterns in each of those subsets. So there's an interesting, okay, thank you, great. That is amazing. Thank you very much, great summary. I'm likely to put a link to this recording into places and I assume that we'll eventually do a blog post sharing this because I think the results are already amazing. Thank you. I don't have any other questions. Anything else you'd like to share or highlight for us on results you've seen or what you see as next steps. So the next step is mainly handled by team Yakom right now. It will be regarding the building and providing other architecture for Jenkins, so IRM or PC, et cetera. That's right. And that actually, okay. So multi architecture. And that's really quite a cool story. So I've got to share that. I've got to show that. So on the Jenkins roadmap, we have had on the roadmap for an extended period that we would support Docker images for system 390 mainframe for PowerPC and for ARM. And thanks to this work on build X bake and further, we've got progress on that. Right. So Damien, why don't you share more what, what you're seeing there. So that's also a benefit of using Docker build X. So Docker build X by default is a standalone replacement to the default Docker build commands, which is you have a binary Docker, which is Docker client. And you have a Docker engine, which run on your local machine on Linux or inside the virtual machine on Docker desktop for Windows or macOS, which is a Linux or Windows machine. And sometimes you have a remote Docker engine. So Docker build is a way to build a Docker file, which is on your client side. It send the Docker file and the context of the Docker file on the remote machine Docker engine. And it's able to build the image. That was the Docker build goal with build X. First, the artifact output is not only a Docker image. It can be a target on your local file system, which means send Docker file to remote Docker engine, build the image and retrieve a target image instead of storing that on the remote Docker engine. That one is useful when you need to use air gap system or when you need to export the Docker image to the OCI standard compliance. So you will have an archive, which is OCI that can be loaded on any CRIO or any container engine somewhere else. So not only you have this kind of benefits, but also build X introduced the concept of a worker. A worker is a demon running on the Docker engine where you want to build or any container engine you want to use. And it allows you to build container without needing root access to the remote machine. It can run on remote machine with a different architecture. It can even build container inside a remote Kubernetes cluster because it's a demon. That means it spawned a Docker engine and then it goes. Excuse me, just a minute. Sorry, I'm back. So with the concept of a worker, that means that a local Docker client is able to send things on a bunch of remote build X workers, which can be a remote Kubernetes cluster. And so that means a single Docker build X command is able to send builds, requests to different machines. So one RM, one Intel, one PowerPC. So as soon as you have a collection of remote Docker build X demon, I'm not completely sure about the wording. The documentation is way more clearer than me. But the idea is that you have one Docker file, one Docker client, and you have a bunch of remote machine. Build X allows to manage that cluster. And so with that in account, Docker added a nice feature which is enabling KMU, the emulator. So build X is able to spawn on a local Docker engine, Docker Linux Mac Windows with an Intel machine or IRM machine. And it starts and enable KMU emulation, and it's able to run Docker engine with images for different architecture. That technology exists in two years on Docker desktop for Mac. It was featured on different Docker code, like, oh, you can run the Raspberry Pi Docker image on your Mac OS instead of a slow Raspberry Pi. So using that feature, Docker build X is able to spawn in five, six seconds with KMU, a worker which is able to build in parallel for all the supported architecture here. So what did team work? He only added the supported and expected architectures on the Docker bake file, saying, OK, we want for Jenkins, Debian, not only Intel, but also IRM, et cetera. So that was just an array to append. And we had to install KMU and ensure it's enabled on all the agents. And if you have both, so enabling KMU is quite easy. It's a Docker container to run with the privilege that will enable an instruction on the kernel, and then that's OK. And with the bake change and the KMU emulation, then we are able with a single Docker build X bake command to build in parallel a given image on different architecture at the same time on an Intel machine. We don't need a specific architecture machine. Just a minute. Sorry, it's the war here. Too much person at home, sorry. So with that in mind, sorry again. OK, sorry. So with KMU installed and the bake change, it allowed us to build successfully during one week on CI Jenkins.io, all these architecture images without requiring any specific architecture agent, which is quite a good step. So it's an emulator. So maybe one day we will find issues because we built an image with some binary built or whatever. But most of the time we only download binaries and packages for a given architecture and we rely on the upstream operating system, Debian packages, Alpine packages, and West packaging. We compile on our home, so we should not have an issue. So with that in mind now, the work is to be able to publish these images first as experimental. So translating our work from CI to trusted. So from the CI parts to where we publish officially and sign the images. So we had some hiccups on that second part. We discover why it's because depending on the cloud provider and the kind of hypervisor using the virtual machine we use to docker build, sometimes you only have to enable KMU one time when building the custom virtual machine templates. And then it will be enabled always. And then some other cloud provider or hypervisor, you need to re-enable KMU before each build. So we decided that the pipeline had to be updated to force updating KMU because it's important. So if you run the docker run, blah, blah, blah, privileged KMU enable in any image, you can run it one, 10 times. It will have the same result on the end. So we run it before the build and it's okay. So that was the last changes and fixes that are quite hot because it was this morning in Europe. So thanks for our team for all the effort and the heavy lifting here. That's really interesting to see the, our ability to go forward on that subject. So anyone interested on that part, don't hesitate to raise your hand, propose help or reach out on discourse. Same if you have any question. Thanks very much, Damien. That's excellent. I don't have any further questions, although I admit I am looking forward to having an arm image. I'm a little jazzed about system 390 and PowerPC just because they're so exotic to me. So thank you very much. That's great. Now I assume no plan for arm 32, for instance, that this is all 64 bit work. I don't know what were the decision. Yeah. Well, I guess, I guess that's a good point. We had long ago chosen, I guess it's a year or two ago chosen that officially the Jenkins project is actually only running on 64 bit JVM. So I'd answer my own question. Really, we don't support. Yes, it runs on 32 bit virtual machines, but fundamentally the project does not test windows 32 bit, for instance, and we certainly don't test any Linux 32 bit. And so no reason for us to make an exception here. It's all 64 bit all the time. Unless there is a real use case, like at large scale, but most of the IRM provider now provide 64 IRM architecture, especially in RM, I remember that 32 bits. There were a bunch of different variations. Raspberry Pis depending on the serial number were having different CPU architecture that was kind of nightmare. And since the Raspberry Pi three, which is now four year olds at least. Yeah, we have that one. I don't know for other amateur or professional platform, but most of the IRM provider are 64. So that makes sense. Great. Excellent. Thank you. Thank you. Anything else you wanted to share today? That's already a lot. Thanks very much. That covers all the topics we had for today. Any other topics we should bring to our agenda today. Given the amount of work I've added, I'm going to write a mail. I will still have. I'm not the maintainer of the image and I will want to volunteer to be part of the maintenance team, especially given the amount of code that changed there. So I suppose it's a request to the community that has to go through email and. Yeah, just, just send a, so I asked to become a volunteer, maintain the Docker images so you can find that was a while ago. I think that's a great example. I think it just sends to Jenkins developers. I will immediately plus one that. I am confident that Tim Jacob will give his immediate vote. Yes. And then you'll be granted permission. Thanks. All right. We'll end our meeting now then. Thanks everybody. Recording will be posted shortly. So within 24 hours. Thanks all. Have a great weekend. Thank you. Bye.