 Now recording is on. So welcome for this new Jenkins Infrastructure meeting. I put several actions that I would like to discuss in this meeting in the Google Doc. So the first one is obviously the ticket Infra 910, which is about automating Jenkins score releases. We did up to now, we did three releases. The third one was really a success. So I think it's really that we start looking at the security and the stable one. I have a meeting for the security releases. I have a meeting with Daniel tomorrow to discuss how we can implement artifact promotion. So the main requirement that the security team has is that they want to be able to release a new Jenkins version but be just in a private maven repository. And once it's approved, they can promote that from the maven repository to the master one, to the one that everybody is using. So we just have to modify little bits to Jenkins for a parameter to support that. And I also would like to introduce real artifact promotion. So not only we push, so we could also use that for the weekly release, which would simplify the process so we can, for example, release in advance, for example, on Sunday and then officially promote, let's say, on Monday morning when everybody's available. So that would be the idea. And once it's supported on the security part, then we can start working on the stable. For the stable releases, I think the most important work need to be done is on the communication. So we have a secure and reliable way to communicate about any change on the stable releases. But yeah, otherwise, from a technical point of view, I do not expect any major difficulties. Any question regarding this? So you said you'll discuss with Daniel. So now, sometimes security things have to be kept relatively quiet. Do you have a sense of how much of that you can share publicly later? I mean, certainly the code will eventually be public because we can see there are many portions that are public. What portions are you sensing? Oh, we won't be able to discuss that publicly. It'll have to be kept private because of security processes. So basically, in the security process, the person who reports a security issue will have a private communication with a security team so that they can work on security issues. And so the idea here, and that's what Daniel is already doing with the current process, so the idea is, when you want to release a new version, you just trigger the release on a specific branch and push the artifact in a private repository. So it's still on repo.jankins.ca.org, except that only the security team and the people who contribute the security, who reported the security issue can download the Jenkins version to do whatever they need to do. And once they're ready, they promote that specific version to the master branch. And then it becomes public, and everybody has access to this. So being able to really have promotion of artifact is something really important for the security. We kind of already support staging environments in the current release process. So if we just want to change the maven repository where we push the artifacts, this is something that we can do. The same for the packages. So right now, we can generate the packages. We push those packages in the staging directory. And the last step, the packaging job, is to move those artifacts to the right website. So OK. So the idea is really to discuss with Daniel how we can modify the current release process to also have package promotion between different maven repositories. So I mean, that's what I'm saying. It's important for the security process. But it would also be interesting for the weekly and the stable releases, because it means that we could push to a staging repository. And when we are ready, we just promote from that maven repository, which is not publicly available to the official one where people can access. Because the way I understand it right now, when we trigger a new release, once the Jenkins is published on the maven repository, people receive a notification in the Jenkins instance saying that a new version is available. But it cannot install it yet, because we first need to package for the specific, let's say, Debian or whatever. So the idea is really to have a real staging process. OK. So it sounds like you're aware then and ready for the things like packaging the Debian, packaging the RPM, doing the Docker image generation, things that are all downstream of the Warfile creation, but really shouldn't be disclosed publicly until the security release is ready. Great. Thank you. So the next topic, I mean, the change over the last week, we started using Fastly for multiple websites. So we first configured for plug-ins at Jenkins at Ion, accepted minor glitch. I think it was a success. It was easy to put in place, and it scaled very well. So we started using it for packages Jenkins.io and Jenkins.io. For Jenkins.io, we had to switch from the Apex domain to the subdomain, www. The main advantage of doing this is first, we support IPv6. So you can go there using IPv6 addresses. And otherwise, the main limitation of the Apex domain, at least in Fastly, is that because you cannot have a C name on the Apex domain, you have to be redirected to a specific IPs. So there is one IPs that have all the loads. But if we use a C name, we can redirect to four different IPs on Fastly network. So this makes it more robust. And so that was the main limit. And then regarding packages Jenkins.io, that was also easy to switch. The main challenge that we have here is packages Jenkins.io, the endpoint, is in front of a machine called packages Jenkins.io. I know it's not easy, but it is really something that you have to understand here. And the package, the Jenkins.io machine, has some, I mean, it's not able anymore to handle the load that happened during the morning Europe time and morning US time. And so basically, every week, around the same periods, basically what happened is the Apache is not able to handle all the requests. And so we have a lot of time issues. Because we switch the service package of Jenkins.io and we put that service on Fastly, it's reduced the loads on the machine. But we still have a smaller period of time where the machine is overloaded. And so one of the other things that we've changed for packages Jenkins.io, from a Fastly point of view, and now if the origin is failing, Fastly will return artifact from its network. So the idea is it just put an additional prediction. Using Fastly for packages Jenkins.io, it's give us more time to basically work on the machine so the machine can be able to handle the two other websites. So basically, this is Mirrors, the Jenkins.io, and the Update Center. The Update Center, I use most of the traffic. I need to be refactor or whatever. Maybe we just have to find the Apache configuration. There are some improvements that we need to do with that machine. One of the other challenges that we have with that machine is that because of the number of services running on there, we cannot use the pipettes. I mean, we cannot easily configure it using pipettes. So that specific machine is still in a state where we need to improve it basically. Any question before I move on the next Mirrored Brain, which is a Mirrored Jenkins.io, which is kind of related to this? So another thing, yeah, something that I also forgot to mention. So in the past, packages Jenkins.io was configured. Someone is typing on the mic. Mark, your keyboard. No, it's not you. I don't think so. I think I was muted. No, sorry. I apologize. That's OK. Another thing that I have, that is another change that I introduced on packages Jenkins.io. So basically, packages Jenkins.io is a website that's returned metadata information for Debian, Red Hat, and Zeus repository. So if you want, let's say, to install Jenkins from Debian, your installer will go through packages Jenkins.io. And so basically, what's happening on the back is that if it needs to return you a package, it will ask, let's say, the specific Debian version that you want to install. And so since four years, that package Jenkins.io was configured to fetch artifact from an Azure Blob Storage. So basically, the request was sent to packages Jenkins.io, then sent to an Azure Blob Storage and so on, instead of relying on a mirror infrastructure. And so what I did here is I stopped radiating and asking to the Blob Storage. So I'm planning to just remove and deprecate that Blob Storage. So first, it will reduce the time to upload packages to the Blob Storage, because we don't need that steps anymore. Then it will also help us to save some money. So first, we don't have to upload the packages there. We don't have to pay for the storage and some. So because it appeared that the resource group on Azure, the next biggest cost that we need to reduce is that specific Blob Storage, which is something like $3,000 per month unbanned with nStorage. So I just changed that configuration today, I think. Maybe yesterday, I'm not sure. And nothing was broken. So basically what's happening now, it's instead of fetching the artifact from Azure, it fetched from the mirror, and then those artifacts are cached on Fastly. Olivier, so that means that there is a chance that our Fastly costs will increase, potentially even beyond the costs of their sponsorship, but we would cover those costs through the funds that CDF has agreed to cover? Yeah, so basically, instead of paying on Azure, that cost will move to Fastly, basically. So we just reduced the Azure bill. And while we're mentioning about the costs on Fastly, right now, I had a look, I mean, since the last week, we used less than $100. So right now, the cost is not that huge. So I'm going to wait for the full months. So I have the full picture, but we may add more services on the Fastly accounts. That's cool. Just to be sure, you just said that we've not yet used even 20% of our allocated sponsorship. That is exceptional. Okay, thank you. So considering that the plugins or plugins is running since, I think, more than a week, and package of Jenkins.io and Jenkins.io were added last week. So I expect to have enough budget maybe to add an additional website there, but it need to be confirmed in one month. And again, if you want to have an access to the Fastly accounts, we still have some stats there. So if you want to play with the stats with information there, I can also invite you. So I need a question regarding Fastly. So then the next topic that I want to highlight is that, so now I fixed the release process in order to also upload packages to get the Jenkins.io. So it's getting more stable. While we still have to change to reset the storage, at least now weekly releases are also uploaded to be our brain, we still cannot use it because stable releases and security releases, I mean, the current process for security releases and stable release do not upload packages to get the Jenkins.io, but at least we are getting closer. And something also important that that website currently could be used, for example, as backend for package, the Jenkins.io, but it cannot be used for the update center because I didn't work on the update center to upload also an artifact there. So it won't work for that. So just for my clarity, get the Jenkins.io, I still am mentally thinking of it as a pure prototype that you might, if you needed to destroy completely and recreate that anyone depending on it should not expect 100% reliability. If they need reliable, they should go to pkg.jankins.io is the official face of the project for right now. Is that a safe assumption or have I misunderstood? So you understood almost everything correctly. So basically get the Jenkins.io is just a replacement of Mirrors.jankins.io. Mirrors.jankins.io is using a tool called Mirrore Brain, which does not seem to be maintained since years now. And that requires a specific environment. And that's one of the reasons why the package to Jenkins.io machine itself cannot be updated anymore because it requires specific Python libraries and stuff like this. And so each time I try to update the machine, it broke either of the reason varments or the Mirrore Brain. So that's one of the things. And so get the Jenkins.io is using, sorry, we have a Mirrore Brain. Get the Jenkins.io is using a different tool called Mirrore Bits from the VLC project. And so the approach is a little bit different. It's still a service in front of multiple Mirrors. So it's using the same Mirrors. It just like, it provide HTTPS. So it supports HTTPS. So what, which Mirrore Brain does not and a few more and a few other features. Just like a more, more recent. The thing is the biggest challenge that I'm facing with that service is to just upload the right information to the service. So it's definitely a prototype. It can be deleted at any time. And more importantly, it's not, I mean, it's not because a package has been published to get the Jenkins.io that it's to the right package. So it's really like, it's not the most urgent thing that I'm working on. So I just try to have it stable enough. And so once it's stable, the idea is to replace, get the Jenkins.io, just use Mirrore Brain, a Mirrore.jnk.io, just switch. I mean, put it back to the Mirrore.jnk.io. So the idea is to replace Mirrore Brain by Mirrore. But we are not there, we are not there yet since here. Thank you. So I think it's pretty old from my parts. And I think the last topic that you want to discuss is CIO Jenkins.io. Mark, do you want to? And I don't know that there's much to say there, but let me, I'll take the brief, brief highlight. So special thanks to Tim Jacome for the guidance on safe, safely guiding me away from doing a premature upgrade when it would have been dangerous. Plugins on Jenkins, that's ci.jenkins.io were upgraded this morning. So earlier today, as part of that upgrade, we avoided a problem that was known to have occurred in the Azure credentials plugin. Tim was kind enough to flag that. We unfortunately did encounter a problem with the EC2 plugin. It has some unknown bug that Gunther was able to highlight for me very quickly and we rolled back one version, restarted and everything was fine. I think we've all agreed that we want safer upgrades for ci.jenkins.io in the future and helm charts, et cetera, or other forms of upgrade tests are coming eventually. So just regarding our recommendation for upgrading ci.jenkins.io. So right now ci.jenkins.io is still using the stable version. So I think it's a type like a GDK11 dash, stable or something like that. Which means that each time we need to upgrade it, we just have to pull the new Docker image. So upgrade is not yet automated and it's not managed by helm at the moment because ci.jenkins.io is still running on a Bermuda machine. While we use Docker for it, it's not running on Kubernetes. Tim Jacker opened a PR to deploy its on Kubernetes. The main thing here is that I don't want to use public Kubernetes for the CI environment because the current Kubernetes cluster is quite critical and we don't want to use it for ci.jenkins.io. But we have to deploy a new Kubernetes cluster on Amazon. This is something that we need to work on and it's in the current pipe. So some help is needed here. I mean, not needed but would be useful. So basically the idea is it would be nice to have some terraform codes to deploy and configure the Amazon environments. So that's what's needed at the moment. I'm happy to help there, but I think the access is still limited to cloud Bs on AWS accounts as far as I know. Yeah, it's still the case. So while, so yeah, so this bring another topic. We are currently in the process to transfer the Azure account to the CDF. So while it does not immediately affect the Amazon accounts, it means that we are starting the transfer of the Jenkins accounts to CDF, which with some, so I hope it will simplify then the move for the Amazon. And it's almost, I mean, it's also important that billing transfer for the simple reason is that when we had a new service currently on the Jenkins project, we usually put our own information. So for example, in the case of Fastly, I put my credit cards, in the case of Azure accounts, it was KK credit cards. And so this is something that we would like to, this is something that we don't want to do anymore. Yeah. So for example, in the case of Fastly, it's not a big risk because sponsoring cover everything, but I would feel more comfortable once everything is moved to CDF. So, and that sounds great. Thank you. Anything else then, Olivier, that I'm, did I miss any notes on the ci.jenkins.io topic? It seems like we continue as we are for now. We have a vision that we want to go to Kubernetes and that that Kubernetes would be on a new cluster, not on the existing cluster. And that new cluster therefore won't be hosted in Azure. It was thus biasing towards AWS for the current choice of infrastructure. Yep. Now, if we could get sponsorship from another cloud provider, are we open to consider, I guess not, because we've got sponsorship already from AWS. Exactly. So we have sponsoring from AWS for the, for the, at least for the coming year. And so we have to move for our CI infrastructure and it's, so that's, I mean, that was, that was why, that was the topic of our discussion with the AWS guys and they were happy to help us. And so, yeah, so we are definitely planning to deploy CI.jenkins.io on Kubernetes. Great. All right. If help is needed to set up any automation for deployment to Azure, I can help. To Azure or to Amazon, you mean? Amazon, excuse me. Okay. Yeah, sure. Yep. I'll try to see if I can have an exception so I can have more people, external people on the cloud, on the disk of these accounts. This is something that I will investigate. Okay. That would simplify a lot the management of it. Yeah, although we've still got, I think core release automation is still our first focus. So CI.jenkins.io right now is just not as critical as getting core release automation, running security and LTS builds. Yeah. So my focus is still the core release automation project. I do not expect it to be a big work, depending on the output of my meeting with Daniel tomorrow. I have a better view, but I think we already have most of the components. I mean, the only thing regarding the core release automation, the only thing that I'm not sure yet is how to call the different profiles. So it's better to have multiple Jenkins file with specific parameters. Should we move most of the Jenkins file to a shared library and then just have Jenkins file for the weekly, for the stable, for the security? Should we just call everything from one Jenkins file? So that was the design right now. But now that I realized, for example, that the security need to change different parameter than the weekly arm. Yeah. It's more like, that's the only thing that I mean, but it was that the skeleton is the same for the three different releases. We just have to adjust the different parameter. Yeah. Otherwise, I think we are almost getting to the end. So is there any last topic that you want to mention? Highlights, we should focus for the next week. Yeah. I was just wondering if there's anything that we can do to help with the stability of package and updates. It seems to be almost daily where it's crashing. So it's probably not quite daily, but it seems to be almost daily. We're connection timeouts and there's users complaining or even I had it when I was updating our instance last week where you just couldn't download updates. So one of the changes, I don't know if we've only impacted the package. So the reason that I don't change the location is to read on content if it has to be cached. So I'm not sure if it will have the same impacts. Yeah. It's more for plugins, I think. So it's when Jenkins is doing the check now to try and get the updates, center.json file. That's the one that users normally complain about. So they hit check now to update plugins or they try and boot. And it's getting connection timeout to updates.json.io. There was users complaining this morning about package.json.io, but it's not as often. So Tim and the FastlyWork, will the FastlyWork help with the plugin update center download? Or no. It's been set up on package. So they're on the same VM. So in theory, it should help, but we're still having issues. Okay. So the updates.json file is not actually delivered by the Fastly Network? No, no, it's nuts. Okay. And it's nuts, and the thing that I fear here is that, but what I fear is that if we use it for, if we use it for updates, center, I just feel that we use too much bandwidth. And so it is. Yeah, I wasn't proposing we should use it. I think Tim's got a good question though. Is there anything we can do to stabilize that the delivery of those updates to improve them? Are there steps we can take that are obvious to you, Olivier? That might not be obvious to me yet. So basically the two places where I looked. So the first one, so yeah, I started looking at multiple times. And so we have different ways that we can improve the situation. The first one is to increase the VAM size. This is something that I did last year in June because of the same reason. So the issues that we have on package of Jenkins.io updates to Jenkins.io is something that affect us since more than a year. It's just that at the beginning it was, I mean, it's just that the load is increasing once after month. So the downtime is just bigger and bigger. So we need to find a way to solve this. So first I increased the VAM size. It just delayed the problem and now the problem is back again. The second thing is I had to look to MirrorBrain to see if we could find you in the configuration. While I see that there are a lot of errors regarding MirrorBrain's and the database on the machine. So we had like 500 gigabytes of error just saying like some cache information were missing. I don't know if I shared this information. The biggest challenge with package of Jenkins.io is that it's a really critical machine with a lot of services running on it. Is it, would it be a lot of work to split them off onto different machines or even moving some of it to Kubernetes? So that was something that I started with package of Jenkins.io to get the Jenkins.io and update center. The biggest challenge here is that, so we have three services running on that machine. We have, so that's what I said, package of Jenkins.io, update center, and your house. On top of that, we also have the release tools for Redats and OpenSUSE on that machine because when we generate Redat package with SSH on the machine, generate the packages there and that, and that machine is also configured to upload to the different your house. So we have quite a lot of stuff there. And so it's something that I would like to have just remove package of Jenkins.io from there. Same for the update center. And so just keep that machine just like a big storage and that's it. Yeah. But I think once we can get rid of our brain it will already solve our issues, but yeah, I'm currently not sure the best. I think the best way maybe is that I could grab access to Team Jackup to that machine and so maybe you could see how we could help maybe. Yeah. Yeah, I think that would be useful. At least then I can possibly fix the issues before others come online as well. And just understand the layout of it a bit better. So I just specifically for that machine I proposed was to set up a call with Team Jackup so we can see how it can work on that specific problem because yeah, that machine is quite big with a lot of different services and it's quite hard to, yeah, it's hard to understand if you don't see it and if you don't have access to it. And it also, and also something important is because that machine is running for a really long time you have a lot of scripts there. You have a lot of tools. And so when you modify something it has some sites. So basically what happened in the past multiple time like each time I try to modify or remove one service there it broke something else. A simple example that happened last week when we switched package a jinkie.io to Fastly. Initially it sounds like a good idea and then I realized that multiple scripts were calling package a jinkie.io to SSH on the machine to upload files but because package a jinkie.io was not used by Fastly basically all those scripts tried to SSH on Fastly machine to upload file which is I mean it does not work. And so that was a small change. And yeah, it took me quite a lot of hours and several hours to just review the script and fix those scripts and discuss with the different people involved there and especially because okay. And so yeah, that machine is something like it's an important machine and it's difficult to not modify it without breaking stuff. All right. Yeah, let's talk about this. Definitely would it be good to not have such important machine and break it down a little bit I think. Yep, definitely. So I propose to stop. So we are running over the time so I propose to finish the meeting here. If you have anything that you think that we should prioritize, feel free to look at the Google Docs at its hands or just ask an RSE. Thank you for your time. Thanks Olivier. Thank you. Have a good day. Cheers everyone.