 Hello everyone! This is Jenkins Platform, Sieg meeting March 20, 2023. Today we have Kéles Salerno, MacCway, Damian Dupoto, Kevin Mordens, and myself. We have quite a few items in the agenda, so let's start with the Open Actions item, if you mind. So last time we saw each other 14 or so days ago. My agenda item was to get in touch with us, the Jenkins security team, to discuss getting back the PVC64LE back to the Jenkins controller image, and I didn't get any feedback. So I guess they're okay with this new image. Let's hope so. We'll talk a little bit later on of this. Just for information, tomorrow Vadek for Lenny and I, we have a planned meeting during the European morning. We will discuss the impacts for both the infrastructure and the security team, because as infrastructure officer I'm responsible for the security of the platform, while Vadek is responsible of the security for all the items generated by the Jenkins project from the war to the Docker images. So he is the person that will be responsible if something goes wrong with new architectures, but also the existing images, architectural operating systems and stuff. Initially we wanted to discuss the fact that Windows controller might or might not be deprecated, because no one used them. I have a proof it hasn't been released since one year and a half, and no one complained. So maybe if we have users, but the main issue here is who is responsible for the security and when there is a CVE, what's the process? So that that interests the SIG meeting, because it's nice to have thousands of different declination. But yeah, what happens if we have a Git CVE, will it be fixed for, let's say, our clinics next week for RM64, and who is responsible for auditing and helping the security team? So that's the challenge. It's not about usability or should we have it or not for the need of users. It's if we don't have the ability and the bandwidth to maintain these images and keep security item, we might have to deprecate some. So yeah, I know that Vadek was out the office for a few weeks, so that might explain why you didn't receive any feedback. Maybe yes. My fault. I think you're the ideal person to discuss it with him, Damian. So that's precisely the right group to discuss and great status to hear today. All right. So you've got a plan to have a discussion with Vadek. So the infra officer and the security officer will be discussing it already a big win. Thank you. Yes, and we heard last time a month ago maybe that how painful that was for them to correct a CV for tons of images, but at the same time, we also love to give a reasonable amount of different images for our end users. Anyhow, anything else on this subject? Okay, take that for a no. So next subject about older Docker images. So to Blue Ocean, no progress as far as I know, Mark? Well, so actually, the progress that's happening is me learning something about we have two places on Docker Hub that use Blue Ocean in the context of Jenkins, not just one, two. Both of them need to die. It's even a bigger get rid of it than it was before. The first Jenkins slash Blue Ocean has no tags in the repository. Should be fairly easy to get rid of it since it's probably publishing nothing. The next one has tags, but thankfully or not, the last time there was an update was 12 months ago or more. And Blue Ocean, of course, has revised many times since then. Jenkins has revised many times since then. And the operating system has revised many times since then. So you can imagine that that image is full of problems. And so yes, it's we want to get rid of it. Yeah, okay, because I think I received some Blue Ocean plugin update this week. So this thing is still alive, but the Docker image is not. Correct. And that's the that's the important attribute there, right? Is that that the container you are welcome to and it's easy to configure Blue Ocean in the standard Jenkins container. It's a trivial thing to do. And we tell people how to do it in our tutorials in our online guides. We stopped using this special dedicated Blue Ocean container almost 18 months ago now. So it's long been unused in our documentation. Therefore, it's it shouldn't be that hard for us to say this thing is gone. We're not going to we're not we're not maintaining it, but we've got to get an announcement out. So I guess it won't be part of the bigger, you know, the jet modification for global deprecation is something else in other effort. It should be. I think it would be a really good one if it could be, but but that means it's going to be part of a bigger project. It'll take longer. I'm open to either Damian. My tendency right now is to leave leave this one alive for now with the idea that we will put a proposal out how to announce the end of life of things through the Jenkins admin monitors, because announcing end of life is not just a problem for the Blue Ocean container. It's also a problem for Ubuntu 18 for Cento seven for Alpine 3.14. You know, this end of life is not an uncommon event for us. So I've got an idea that I'm going to bring up as a JEP eventually. Okay, I got a technical question on that area. How do you plan to make the detection? Because when you are inside the Docker image, a process running inside the Docker image is not able to read the metadata, and it might be able, and it depends a lot on the container runtime, but it might be able to know it's inside a container. But knowing the tag or the image name is tricky. So that means inserting a file somewhere on the file system. That's what my thought was is for containers, we insert a file into a special directory that says this thing is end of life as of this date. And there's some metadata in that end of file record in that file that says this thing ends life on this date. If the actual date is after that, we tell you it has ended life. If the actual date is before that, we tell you it will end life on such and such a date, that kind of thing. But that would mean that we would have to make a new release for the Blue Ocean container. That's correct. Okay. And that's the compromise we make there. The other is we could just decide it's dead. And that's what CentOS 7 did, right? They just said we're no longer maintaining this. And that's okay too. Does your monitor could just say if I cannot find the expected file with the expected data, then I say you are most probably running in a supported version. Yeah. I'd rather not put that kind of heuristic into it. It's already awkward enough, but we'll have that discussion in the Jenkins enhancement proposal. Thank you, folks. Anything else about this subject? Okay. Then let's go with ongoing. So I've learned today that Jenkins 2.397 and 2.387.2 We'll need a new Linux repository sign-in case to be usable. So first trivial question, why? Yeah. So why? Because we intentionally configured our PGP key that we used to sign the install packages, the dev and the RPM to expire. And we did that because it's unhealthy to have PGP keys that never expire. It just dangers from a security perspective. Because they expire, we hit the expiration date or we hit it two days from now. And so what Damian has done is created a new key for 2023. And that new key for 2023 is now documented, published, available, and used in the construction of 2.397. It will be used in 2.387.2. Damian, I assume any changes have been ported to the stable branches for that. If not, we may have to double check that we port them to the stable branches. Yes, we'll need to back port now that the weekly is out. The deadline was let's validate the whole process goes correctly for the weekly, wait 24 hours and then back port to the stable line for next week. Great. So does that answer your question, Bruno? Yes, indeed. My next question would be when is the next time that you will create a new key? So how three years? Yeah, three years. Okay. Right. And the reason we chose three years is we keep on a cadence that the code signing certificate we use for MSI expires about the same time as the PGP's private key so that we know when we renew one, we need to worry about renewing the other. Cool thing. And I guess everything is documented except that it may not be public. No, no, actually, well, the key is certainly not public. Of course. But the public key is public and the process is public. And thanks to Damien, we have an excellent checklist for the communication and the steps to do. So his issue report now has this very nice checklist inside it of how to do this job well. Excellent. I had no doubt, but just wanted to check. I had another question that has already been answered maybe earlier. So there's nothing to do for the Jenkins Docker installation as far as I know, because everything is contained into the Docker image. So am I right, Damien? I would go, I would refine it even a little bit from that. We don't require the key because we actually don't use the operating system package manager to install Jenkins on a container image, right? We just drop the Warfile in there and use it because we don't want SystemD managing the service inside the container image, right? We want to manage the container or the service in the container ourselves. Yeah, I tried, I think, to use SystemD within the Docker image and, well, this was painful. So yeah, that was definitely a stupid question, but I like them. Thanks for answering nonetheless. Then if nobody has any question or comment about that, let's go with Docker and of OSS images. Yes, what sad news? We had one week or two weeks ago, but I think there's a happy head. And if I'm not mistaken, I've read some images, some messages, sorry, about Docker saying, okay, that was not a good move. Let's revert our decision. Am I right, Damien? Absolutely. I'm adding the link on the Zoom conversation Docker announced a few days ago. There are no longer some setting the free team plan. So all of our organization not already under open source program will keep at it. We still ask them to move Jenkins for a valid Jenkins CI in Fra under the open source and just to be sure, they should have been already since one year, but it looks like that both them and us forgot about that. The goal is still to delete this organization in the future in favor of using per repository or Jenkins infrastructure manage repository, because it doesn't make sense to use a free service, even if it's really kind from Docker for, let's say, test container or infrastructure only container. In particular, Jenkins for eval is particularly dangerous because it used to be and might still be an organization to push images built from pull requests by CI Jenkins, I use the public instance. So consider any content from Jenkins for eval to be potentially dangerous for your own usage. The only goal was to be able to share a built Docker image to another contributor or reviewer to test the proposed change. While this use case makes sense for easing contribution, especially in a synchronous world where people don't leave and walk and spend time contributing at the same time of the day, we might want to restrict that area to GitHub container registry, which provides per repository permissions. So the amount of people able to write could be restricted. It provides token isolation per project. And finally garbage collecting, which Docker up doesn't. But it's only an idea. But the goal right now no more danger, but we should get rid of this organization to keep only the official Jenkins organization. Got it. Thank you. So we're not in danger of using everything anymore with Docker until their next move, I guess. First time meeting, we are fine. Thank you. Mark, Kevin, Sam, maybe. Hello, Sam, Kenneth. Anything to add about that? Nothing for me. Thank you. Then next subject, and we'll look enough to have Kenneth with us today. So I've seen a very nice progress for the PPC-64 LEPRs. So I've seen some on Docker agent and SSH agent and nothing yet for inbound agent. But last time I checked, I think it was even not for the agents. Kenneth, you had told us about Jenkins controller and that all. By the way, I didn't find if you made a PR on the Jenkins controller Docker image. I did. So let me share my screen. I can catch everybody up. So I was able to build and test all three agents. The reason that I did not submit a PR yet for the inbound agent is because to build it, I had to tag my own Docker agent build. Of course. So if everyone's okay with it, I could just submit the PR for the inbound agent since it is working and I tested it. And it's a similar patch to what I've done for the other two agents. So I don't see why I shouldn't at this point. Make sense. Maybe it might be obvious, but I prefer stating it aloud. The checks associated to that pull request will fail until we are able to release the parent Docker agent image, of course, but that should not restrain you to open the pull request, given the promising results on the other pull request that you did. Exactly. So that was the reason I was holding back, but if it's not going to stop the pull request and it won't create an error in the pull request, then I'll do that. So Kenneth, just to be clear, because I'm not sure I industry everything, so you worked on the three agents, but did you do anything for the Jenkins controller for pbc64? I did. And so if you notice this built-in node at the top. Oh, yeah, of course. It's my controller build. And I will show you. Just if I didn't know the PR link, I guess. Something happened when I added the PR links here. And by the way, I have two screenshots that show you exactly what I'm showing on my screen here. I added these to the ticket. For some reason, when I added the pull request to this ticket, it doesn't have the full description, but the other two did. So this first one is for the controller build. So the dirty trick there, Ken, is if you edit that comment and put an asterisk and a space at the front of those lines, turn them into a list entry, GitHub will then in their infinite wisdom reformat it so it looks nicer. I know that's really silly. It needs this asterisk and a space. While you're there, hit the preview and we should see it preview. Yep. Now, can I promise that GitHub will always do that for now and forever? No, I make no such promise, but that's the behavior I've seen. Okay. Any questions? Not from my side. I just have a big thank you to tell you. That's a fantastic contribution from my point of view. Having a new to me, a new platform back into the container images is super fine for me. Daniel, anything to say about that? So just as a reminder, I'm not sure we used to build a PPC 64 image until we were unable to have the agent. I don't remember exactly. Well, the issue actually wasn't the agent. We had the agents. They donated so that we had access to agents. We actually didn't need them. The thing that we were missing was we didn't have the QEMU facilities to run PPC and Ken's found a way to do that. And that's included in his progress. I'm sure we already did it because the images are all built using QEMU since one year. Yeah, there was a very specific problem with QEMU for PowerPC 64 a year ago that made it so it was infeasible for us to build. I don't remember the details, but it's noted that there was something really dark about that particular thing and Ken's found something that's much better and it works well. I guess QEMU static Ken is what you used? Right. I'm using the interpreter to wrap my container, so I'm building it in a container. Now I could get it to work in a full QEMU emulation. I know how to do that as well, but it's slower. So I prefer to build it in a container. And speed, oddly enough, speed is one of the crucial things that matter to the security team. So if we slow things down, the security team tells us that's unacceptable because they've got to, in days of security advisories, publish promptly. So you're choosing fast is a good choice. If I may, if I may, that's not the case anymore. I really want to emphasize something is that now we are able to either parallelize using Buildix, the ability to build in parallel on the same machine or different distributed machine. And the issues that security had with the images is gone because it was a time where building all the images took 20 to 40 minutes. Now we are under the six to seven minutes on the deployment trusted environment. So adding the power PC, that's why I ask for the pull request to see already on CI Jenkins impact. Clearly it adds two to four minutes depending on the images on the builds. That's not really a problem because we have uneven machines. The machines are spot instances. So we cannot do a full realistic benchmark because sometimes it will be faster. Sometimes we sometimes it's just on a few builds. The main problem though with security on one of the topic we're going to discuss tomorrow is that the release environment inside the private network doesn't have access to a PPC-64 machine. And the reason is that we cannot trust a machine that we don't do not manage. That's why having a agent for testing and building on CI Jenkins.io sponsored from another organization is absolutely fine and a good thing that allows any contributor to both publicly open pull request and run acceptance test on a real PPC-64 machine. But the deployment environment must have a way to build the associated image. That's why we build currently the MD-64 using KMU and Buildix because we need something we trust and we trust the machines inside there. So we might have to find a solution. But the pattern we have right now with the CPU-Z machine and the former PPCs machine is that we SSH to an unknown machine remotely. So the level of trust that we can put on this machine is clearly low. It's enough for CI Jenkins.io and public contribution. It's clearly not enough for the environment because this machine if taken over could in some way inject code to a million of Jenkins instances. That's why we need to be really paranoid on that area. That's the agent that we have an infrastructure and security and the console I want to show there because speed is not a problem. In the case of KMU, we could spin up a machine that builds only PPC-64 with KMU. That will be clearly the same amount of time than the Intel with all the CentOS, Debian and images. So we can't parallelize. So the added overhead here is not worth optimizing. However, if you pull up any of the full requests I had, I documented how to do it with KMU. So I wasn't proposing that we use some power machine. I built it with KMU. Yep, that was clear. It was just a reaction to what Mark said, just to be sure we all align on the same concerns because we already had the asynchronous discussion. And that's really cool. Right now, the build you did, as I understand, are building successfully. I saw on the logs of all the pull requests, it used the default build X KMU loader. So that works very, very nicely. So the current state is that for technical needs, we can release or prepare the release of these images. Outside the discussion that we will have with Vadek, because I don't know the status, and we need to study on the Debian-based image, the status and life cycle of the CV, how does it work. Today, publish the same CV life cycle as the Intel and the RM1, that will be the main concern that will be a blocker. Outside this one, technically, it's clearly fine to start with that machine. And then as we discussed, if we are able to have a real PPC64 machine that will help to run smoke tests on the main branch or pull request. And that one will be useful to counterbalance the KMU emulation because then we could run on an untrusted machine, pull the image freshly built and try at least a smoke test to start a Jenkins controller like you did on your own machine. Does it make sense? Yes. Cool. So, if I understand the well, KDEV, you proposed a way to build very efficiently, but that's not what will be used by infra to build the end PPC64 image, because we will use Docker, BuildX and so on. But you proposed something else that people could use on their own machine just to test. Am I right? So, well, I did use Docker, BuildX and... Was it QMU static? Yeah. So, QMU static is a package from Docker that allows you to launch a container for a foreign architecture and then it'll use the QMU static interpreter to execute the container processes for you. So, is that not how BuildX is running right now on your builder hosts? Yes, that's how it runs. There is a prerequisite steps that load the QMU binding inside the kernel first. And then one, as soon as these bindings are available on the kernel, Docker engine is able to report that it's able to emulate some additional CPU architectures. Then the Docker bake build says, okay, I want to build that list of image on that. It creates a matrix of the different images, declination, tags and architecture. And build everything in parallel using a BuildX agent, which is loaded implicitly on background. Right. And that's exactly the steps that I had in the PR request. So, anybody could emulate what you're doing at home, basically. And that was the technique I used to build it myself. That technique is really useful for people running container engine inside container engine. If you have a Kubernetes cluster where the outer Docker or container engine or container D is able to run in username spacing, you can safely run Docker builds with the method that you shared. That one can be efficient for a lot of users. We use virtual machines so we don't have that kind of problem, but that's still worth sharing because that's really a nice work. Okay. Thanks a lot, Kenneth. And thanks for your input, Damian. Next, I know we're already late, so feel free to leave or if you have something else scheduled. A quick information about Scalingway, for example, I reached them last week asking if they still had ARM32 machines running. It's been seven years also that they had that kind of machines, but I don't know if that works or not. And I know they have an open source project, you know, helping other open source projects you have dedicated hardware, but I haven't heard from them yet. So we'll see if we can do anything with Scalingway, which happens to be French, I guess, or partially French, by the way, so they are neighbors of mine. And now, we also had this week, somebody asking for Alpine or 64 images, that subject that was interested in a few months ago, but maybe Damian could tell us more about that, but I was almost sure that we could get pretty soon Alpine Timurine-based images, but no, I think it's kind of stuck. Damian, do you know anything more about that? Yes, Timurine project needs hands to help them on that area, otherwise it's not going to happen. We have Debian IRM images. So inside Alpine Linux, there is an OpenGDK 17 build, a package that you can install as far as I can tell, but it's not Timurine, because we also had that kind of request on the official Docker ASCII Docker image. Right now, what we do is walking on Alpine for IRM 64, using the OpenGDK 17 package inside Alpine. We cannot use Timurine or another distribution. So we need help, at least Timurine need help to provide an officially major, at least an official package that we can install on any Alpine. The background is because of the JLIP used in Alpine as a reminder is named muscle. It's a strict implementation of some of the JLIP interfaces, which doesn't play well with most of the GDK features or low level features of the GVM. So that's why they are having issues. Back to the Jenkins project, the performances and the behavior can be really risky using Alpine on IRM. So that's why it will be, if you are running that kind of CPU, better to use the Debian image instead. The difference in weight, at least for the controller, is close to zero. So there should be no blocker on that area. It's more for the agent that could be problematic if you need an agent using Alpine for whatever builds you are doing. Yeah, okay. So Mark asked, why would you need Alpine for IRM 64? And the user answered that he needed very recent version of Podman and Chrome. It is what it is. If Timmering can't supply a version of GDK for Alpine, it can do much. Thank you Damian. Mark, anything you would like to add? Nothing for me. Thank you. So next subject, digital code signing for MSI installer and jar file. You said, told us a few words about that earlier because it was kind of linked with, not linked, but the same timeframe with repo key that we had to renew. Yeah, so the story goes to what Windows users expect of their installers. Linux users sometimes are a little more forgiving that, hey, something's not signed or something's not marked as this detail or that detail secured. Windows users tend to panic if you ask them to install an MSI package that has not been secured, not been signed with a digi-cert or comparable key. And it's reasonable, right? They're accustomed to as the major part of the infrastructure that they are as the major part of the world that runs Windows computers. They are the big target for malware. They're the big target for all sorts of dangerous things. And so Microsoft raises big warnings if you attempt to run an installer that's not been signed. The certificate we use to sign our installer expires March 30th. So we've got two more days. We squeezed in 2.397. It's signed with the unexpired certificate, but next week's LTS MSI installer, unless we get this replacement certificate, won't be signed. And that's not a good thing. The problem, of course, is digi-cert demands some additional authentication to prove that we are who we say we are. And we had to invoke attorneys from the Linux Foundation to answer the digi-cert request. And the attorneys have taken a little longer to address the question than we'd hoped. So it's still in the hands of the lawyers. And we hope that the lawyers will solve it. We've appealed to the Continuous Delivery Foundation, our Representative at Linux Foundation, to please accelerate that. And they've said, hey, they're accelerating it. So we are hopeful, but we'll see. Wow. Okay. It's not as easy as it seemed at first sign. No. Chain of Trust has to be established. And of course, digi-cert correctly says we want legal evidence that you have a chain of trust. And legal evidence requires a lawyer. And we've got the lawyers involved. We've had them involved for several weeks, but it's not progressing fast enough to meet our deadlines. It doesn't seem to be yet. Oh, thank you for the feedback and best of luck with that subject. Okay. Now, Weston, I haven't spotted many things. The latest updates I've seen for the agents' images were just bumping the Tivian and Oclinus base images. And I haven't yet seen the changelog for the main Jenkins docker image. And yes, I just wrote right on the old release. It was just to say that the release that we can see on the website, on the repo, is not linked to the release on the docker hub. Damien, I know you have already answered me in a previous meeting about that, but could you please just tell us a few words about why the release on a GitHub is not the same as the release on docker, if that ever makes sense? For the controller image, because that's how it has been for years. I don't know the why and the initial reasons, but the process for sure is a script that, when called by the deployment environment, will check the, it used to be 10 or 20 past versions of both weekly and LTS lines. Now it's free of them. And it checked, it then parsed the docker bake the list of all images and check on the docker hub before the manifest are published for each tag. And if it's not published, which 90% of the time is, oh, there is a new weekly published like it was today or it will be with the LTS next week, then it build and publish all the images in less than 10 minutes. However, that has one main drawback, the 10% left of the cases is when we had the new image declination. For instance, when we will merge the PPC 64 work, because I'm sure we will, then that will rebuilds if we don't take enough care, that will rebuild all the images of the free past versions. That's why we were restricted. Additionally, that script is a bunch of shell script, which means it doesn't work like this for Windows. So the good news is that it doesn't work at all for Windows, but still, that's the problem. The main concern now will be switching to at least creating a release. But that means part of the Jenkins release process will need, hey, someone need or something need to publish a tag inside that repository that will trigger a build, create a release and push the images. That one will be a first important step to avoid wasting our time when adding new version of a given image. So it's still doable right now. It's just we need help on doing that part. And that help must be able to spend some time with infrastructure and security team for the release process. Here we are. That's the reason. Okay. Thanks a lot, Damian. It's pure now for me, at least. That's all that I had on the agenda. As anyone, something to add a comment to make a question to ask? There is a proposal, but right now it's still just an idea. I need to write a GEP, but I'm missing time. So anyone willing to help will be welcome. My proposal will be quite a change and will have impacts on Docker images, but also some of the packages, at least the Debian 1.2 and RPM. The GEP will be defining a new versioning scheme that will involve the package number. Right now we have a one-on-one mapping between a given version of Jenkins and let's say the Docker image that is here. But if you change, if you add a new something inside the image, which is not directly the Jenkins word, for instance, you want to upgrade the Timurine version or you want to update Git because a CVE happened, for instance. Right now, either you rebuild the images, which happened accidentally, as I said earlier, with the Docker image, but that's not a good thing. And we rebuild a given tag and we override the existing image. Or we need to trigger a new Jenkins version that will be first a new weekly, then will have to be back ported to LTS and eventually other lines. That nightmare, especially since Jenkins is only a dependency. We have the same issue for the packages. It's less visible because the packages are quite stable over time. We had one major change one or two years ago, switching fully from initv to systemd. But we don't have the same amount of changes in the Docker image. Still packages, package manager, have a nice solution, the package number. You can build a package with the same application version and increase the package number. So you will have, depending on the package manager, that's an integer or that's a timestamp or it depends a bit like the snapshot in Maverick. And so the GP will be to add that concept of package build number on at least RPM, Debian and Docker image for the controller. So that would allow us to have a proper release management. And that could have the benefit of considering Jenkins as a dependency and not as let's push each time we have a new version. The consequence will be automated process by Dependable, Train of Abort, Update CLA, whatever tool will detect the new Jenkins version, the new war version as soon as it's released, and then would open a pull request saying, oh, there is a new Jenkins version. And also the same dependency system would open pull request saying, oh, there is a new Git version, a new Temurine version, and will allow us to release the same Jenkins target version but with a new package number. Of course, that new scheme or that new way of using packages would have impacts on the users. They will need to update their own system that track the version of either their packages or Docker image to kind of reg X or not saying, okay, do you want to track the latest minor version of the LTS Jenkins line, but always the latest package or do you want to be really paranoid and track even the build number because you need to check everything, which could make sense. So that one need a GP to explain the pro and cons. Maybe there will be other and better solution. That's why your GP is required to have the advice and point of view of a lot of different person explain the why and what. And so if it's okay, then the how will happen in the form of pull request changing the release processes and then proceeding with plenty of communication, of course. So that's not an easy task, but that would be a valuable one. That's why I'm asking for help on that area as well. Cool, got it. Thank you, that was clear to me. Any other questions, feedback? Okay, it looks like that's a wrap. Thanks a lot for your time, folks. The video should be available on YouTube from 24 to 48 hours and see you two weeks from now. Bye-bye. Thanks, all. Bye, everybody.