 Okay, so what's recording has started? Go ahead, Olivier. Hi, everybody. Welcome for this new Jenkins infrastructure meeting. We have a bunch of things to discuss today. So the first is a few announcements. The few things that you need to be aware is we are still in a process to remove hand shots that we are using from GitHub.com slash ham slash shots. While I may create a few of them like graph and primitives and so on, we still have to update the nginx ingress controller. And it means that it will require a little bit more work because we have quite a lot of application using it. And so we can just upgrade the chart. We have to remove it and then reinstall it. So we have a bunch of ways to do it without having downtime, but it will require a little bit more work right now. It's not enough for some help on this, but I still have to work with them to deploy something new. We may also take the opportunity to use maybe traffic or something else. But yeah, so that's the most important one. And the other two hand shots that I did not migrate are out proxy index. Those were used for last year for the polling application that we used for the election last year. This application is not needed anymore, so we just removed those hand shots from our infrastructure. So any question so far with this topic? Yeah. And then again, so we can continue. The second thing that I just want to highlight is we have, we had a bunch of instabilities on Seattle Jenkins.io over the past few days. Those were for different reasons. The most one is we enable, I'm not sure yet who and how we did that, but basically we enable the wrong disk for open to machines. So the method of this that we use were just 8 gigabytes of disk, which is pretty low. And we should have been using a 100 gigabytes of disk because this is the way we use biker images. So now this is fixed. So far we should not have any issues with open to machines anymore, at least for the coming days. And the other thing is, so sorry, yeah. I was just going to say, when did you fix this? Because I was still having lots of issues up until like less than an hour ago. I had like 10 builds fail with full disk space. I went around letting some agents and then just some of them I just kept rebuilding until I got ones that worked. So yeah, so maybe something is not fixed yet apparently. So basically what I did is I worked on that this morning, my time, European time, and I deleted all the old machine. I reprovisioned open to machine from Seattle Jenkins.io and I double checked that they had 100 gigabytes of disk space and it was a case. So yeah, if you have a job that you can share, maybe I can investigate. And I was also looking from the Amazon console. So yeah, I would be glad to investigate if something wrong is still happening there. Probably deleted now. Nothing looks too bad. There's one EC2 agent with four gigs free, which I think can quite easily go very quickly. I can look at it. I can look at anything. So we know that we still have the EC2 agents that are unstable and we still need more windows disk space. This isn't this your change didn't do anything for windows builds, right? They're still, they're still on smaller images. Yep, definitely. Last things regarding Seattle Jenkins.io. Yeah, so just to continue this one, the next one is more for Daniel. So I investigated what was happening in Seattle Jenkins.io. And for some reason, the machine was in a wrong state. It seems to be a lack of memory, but I couldn't find any out of memory issues in the logs. So basically what I did is I just increased the size of the machine and everything seems to be working just fine now. I don't see all the weird errors from the logs. Well, I saw omkiller messages in the system log. So I would agree with that. Yeah. But I also saw a lot of weird error logs related to the Docker demon. So initially I thought that it was like some disk space issues or something like that. But yeah, it seems to be running since last week. So I would consider it as solved right now. Thank you. Next topic is regarding the Docker hub terms of service. So the Linux foundation replied last week. They basically said that they would be happy to sponsor the project for that. But also at the same time discovered that Docker hub had a sponsoring plan for open source projects. So I sent them an application, but I haven't received any response at the moment. So yeah, I'm still waiting until the very last minute. And if nothing is nothing clear happened there, I'd probably just pay for at least one month's and expense that to the Linux foundation. But the good thing is Linux foundation is aware of that. And they are directed to find a solution for the other project as well. So I'm still going forward on this topic. Now, when this happens in it's November one, is that right? Is November one there? So the deadline is November one. So does work to then go and make sure we Docker log in everywhere and configure the secret everywhere and yeah, and another thing that needs to be working on is we are still using a bunch of Docker images from my own accounts. And so we have to update those as well. I was I was reviewing my Docker hub account this afternoon. And let's say I have five, for example, is published on my own. I mean, I have some a bunch of keynote to do to push to the right location and use the right the image. But I'm not sure yet if either I push them to the hub or I use the GitHub Docker registry. So these are also another possibility. Probably Docker hub. There's been a number of issues with GitHub's implementation, I think. Which kind of issues if you have any document, because I haven't huge GitHub issue with interoperability problems between anything that's not Docker, things like container D which Kubernetes is moving to default in the next it's not going to AK SSD folder to container D in the next version. Okay. Okay. Good to know. I have to get that topic. The next one is more for Garrett regarding Docker images and windows. Yeah, that's progressing pretty well. I have the packer image with the Windows update provisioner correctly working now. So that's pushed up to Docker hub. The build's been updated to use that. And I'm just pretty a few images now just testing that they're all working. It's going to take about another hour or so to get through those. But once they're there, we should be able to start trying out the builds again to see whether or not they build Windows Docker images properly. Okay. Yeah. Just on that topic, we discovered this afternoon that InfraredCI did not reload the configuration and it should. I'm having stack trace related to that. So if someone is able to help me looking at what's wrong there, yeah, somehow this needed here. Because if it's broken for InfraredCI, I would guess it's the same for really that CI and the other Jenkins instances running on communities. Hi, Sledin. Hi. The next topic is regarding the JFrog Artifactory. Daniel, do you have any updates here? I mean, what it says in this item. So we got an analysis from Baruch from JFrog about traffic, popular artifacts, stuff like that. And when we looked at it, we discovered that just a handful of artifacts make up 30% of the total bandwidth used. When those artifacts are referenced in the tool installer metadata for the Allure plugin, which seems to be some sort of build tool or something. And what happens is if you run a build that uses Allure, your agent will download from the Jenkins Artifactory directly. And these agents, if they're ephemeral, they will not have an M2 cache or anything like that. But they will download every time. And that means we have hundreds of thousands or even millions of downloads for some of these command line tools, 15 megabytes each. So that accounts for 30% of the total bandwidth used. A week ago, I filed an issue and pinged the maintainer of the plugin Infra 2772. And we have not gotten a response until yesterday. So I basically blocked downloads from repo Jenkins CI org of this tool. So anyone who relies on it will see their builds breaking. And then we got a response from the maintainer. This last yesterday afternoon, offering to file a correction pull request. In the next few hours, I restored the downloads. I looked today around noon, no pull request files. So I blocked the downloads again and have no intention of restoring them. Because if there is a proper download location, we can do that on the crawler side, which maintains the metadata rather than ever having to allow downloads from repo Jenkins CI org again. And that's my current plan here. I have still not heard back from the maintainer again. I don't know what's happening there. But given that this basically looks like it might actually be enough to make JFrog happy based on what the other usage stats looks like, I will not allow any compromises here. If we maybe don't actually have to lock down things further. But we haven't discussed this with Barok yet. Okay. Thanks. Thanks for the update. And I'm really surprised by the amount of traffic used by Andrew. Any question for Daniel? Nope. So let's move to the next major topic, which is the JIRA upgrades, JIRA migration and JIRA deprecation at the same time. Mark, can you do an update on this? Yes. So this is test week. Anton Baranov of Linux Foundation reported late last week that he had completed the first migration or he completed the first copy from our version to his 7.13 version. He was doing the upgrade. Earlier this week, he reported during the upgrade, he detected some missing images. They were the images for our, what are they? A logo in a bunch of it. Right. Logo and avatars. That's what it was. Logo and avatars. And so I provided him two gigabytes of a zip file of the logo, tiny and avatars enormous. And hopefully that's enough for him to get us ready. This is test week. They had agreed to be ready for us this week. We're waiting for him to tell us, Daniel, I'm assuming you're a volunteer to help with the testing. The other people on this call were already nominated last week and consented to be part of the testing. Yes, please. I have a question about the test setup. Does that also, so first of all, are all projects going and all data going to be present in the test environment? Yes. The intent was that all projects and all data should be visible. So for example, a crucial thing I was hoping you would check is, are the security reports correctly visible and correctly hidden to people that shouldn't be able to see them, those kind of things. So this should be a full and complete migration. Yeah. I would like that to be, that there is a basic validation of that being the case before the test environment is made accessible in the first place because of the sensitive nature. I mean, just have an unprivileged account or go anonymous and type in any security issue ID and see whether you can see it. That would be very bad. Right. Good point. So let me take that as ask Anton to validate, to validate that trivial security exposures are not possible. So just to clarify, so the migration they are restoring exactly the same machine that we had when weeks ago. So it's supposed to connect to the ADAP. So we are supposed to keep the same group, the same users. The only thing is the data will be audited because we keep updating the tier instance at the moment. Okay. Normally we should also be using the current DNS record. So issues that Jenkins had at the org because we provide the information so they could generate the less encrypted certificates using DNS so it will not mess up with the current instance. So we should really be able to test it just enforce the IP for a specific host name or whatever. So you were saying, Mark? Just that you described perfectly the intent. I agree that Daniel's got the right approach to be concerned that while the intent is good, we really do not want this thing to be visible on the public internet with some trivial exposure of our security information that would be bad. And it should be easy enough to confirm that that's not the case. It's just, you know, if we miss it, then we've saved a minute of time and potentially lost a lot of confidential data. Exactly. Right. I think that's a brilliant brilliant thing and I will certainly work with Anton. I need to talk to him anyway. So this is a great excuse to do so. And also something to clarify here is we will not use in prediction the instance that Anton restored. So if we validate that the migration is okay, then we create a new machine and we restore. So we do a new backup and restore again. So and then when we decide to do that backup, we'll have to plan a downtime because we have to be sure that nobody updates the old instance while we are restoring the new instance. And that's a good pointer. The week of November 9 is the planned week that's trying to avoid the LTS release week, which is the prior week. That's all for me. Thanks, Olivier. Any questions? Yeah. I guess we can just skip to the next point, which is Miros. There is nothing renewed here. So we just removed it from the notes. I haven't worked on that, I think. Oops. Sorry. The latest information that we're not burning money is still correct, right? So the fallback mirror does not cost a lot of money. That's okay. No, no, no. Yeah, it's still the case. And that the peak is fine. While you're talking about Azure money, we still have a broken process right now where the Linux Foundation is not correctly notified when they have to pay the bills. So we still have two bills that they have to pay. But yeah, it's ongoing work. I made a few changes to the permission that the Linux Foundation has. So they should be notified in the future, but we still have to actively monitor that to be sure that they receive the bill in the future. Last time I checked the Azure, yeah, we were really good. Any questions? So we can move to the LTS bill process and release candidates. Marc, do you have anything that you want to bring here? There are two things. So first, sorry to see, but congratulations to Oliver Gonca. He's having twins. As the father of twins, I want you all to know that that was the most instructive thing I ever had experience as a parent. Twins changed my life completely. So he's having twins. And therefore, he won't be our release officer after December 3rd. That means that we need to figure out find what we'll do in terms of where we execute the acceptance test harness and how we construct release candidates. And I think we want to do the release candidates inside our release automation info we already have, but it will require some development work. Do you have an idea what are the requirements for the acceptance test harness execution? Well, I know what I see on ci.jenkins.io, when it runs on ci.jenkins.io, it is eight hours of blocking use of all of three, four, or five high memory machines. So it's a very heavy demand. And so I think we may want to consider a separate cluster or a separate place so that its resource demands can scale up and down without disrupting typical plug-in builds. We already have the autoscaler on release at ci.jenkins.io, so we could add that there. The only question is, do we need to run acceptance test harness on release at ci? On release? I don't think so. I don't see what... There's nothing that I know of in that harness execution that is sensitive. So I don't think it needs to be on release.ci. At least that's my perception. Tim, your comments? I think ideally it could be triggered by it, like just as part of it publishes the candidates and then we trigger it would be nice just to so you don't have to do any more manual steps. It doesn't think it has to be hosted there, but it might be easier because I'm not sure whether we want... We don't really want to disrupt everyone on ci.jenkins.io when we're running it. Especially if you're running it, you don't want to get stuck behind 100 core builds. So release.ci could be convenient as it could be easy to run it there. The results would mean the results are only available inside the VPN unless we published them. Yeah, that's my challenge with release.ci. I feel like acceptance test harness is already high maintenance and low visibility. And my worry is that if its results are behind release.ci that it will be very hard to get people to take action to correct failures that are detected in it even when the failures are real. I'm worried that there are enough failures in there that are fake, that are false failures that are misleading. That's a good point because we saw in the past that when a job is failing and it's not really visible to a lot of people, then it's more difficult to fix it. So, rc.ci.jenkins.io and it runs the RC and runs the tests. Oh yeah, okay. We still have to think about that. We can still build them, build the RCs on releases. It's probably easiest. Right, right. We could have a public instance that is rc.ci.jenkins.io that's visible. Yeah. What? What is it used for? When did you deploy that one? No, no, no, no. Could, could, could, could, could, could, could, could, could, could, could. Okay, okay, I was just like, what? I don't remember about that specifically. That's such a thing. Yep. And the other thing regarding the release candidates, do we really have, do we need anything more than what we already have with the release environments? I mean, if you're just adding a new profile, that would be, that would be easy. So, currently the RCs published to snapshots is the only difference. It's, so it doesn't get, it doesn't get published to any releases repo. Um, so it needs some changes in the get Jenkins version. Because if we're just, if we're just pushing the, the release candidate to a different Maven repository, um, this is something that could be done at the profile level. We just provide different settings. Yeah. It's basically the profile, just figuring out where things go. Yeah. Okay. So it sounds like an easy way. I mean, doesn't sound like really difficult to fix. It'd be good. Cause it's a last thing using merit legacy mirrors, I think we're going to stop seeing the HTTP links. So we could possibly have things like automatic mail in this publication of the RC as well or at least a script. Okay. Yeah. Let's take some more time to think about that. Any last suggestion regarding this topic? Um, yeah, then I'm really curious about the, the filling point, which is has your credit offer. Have you discussed about that with Microsoft? I mean, is there any opportunity there? Yeah. So I have a meeting today with Kayla Linville, our customer success manager. She initiated the meeting. She's trying to explore, is there a way she could submit a proposal to Microsoft to ask them to fund the project's infrastructure to some level. And I'll have that discussion with her. Olivia, you're welcome to join. I assumed I can talk these business things without bothering you. It's, we, we want them to give us money and she needs to act as our advocate to ask them to give us money. I don't, I mean, I already had this, this, this discussion several times with Microsoft. So I don't think my prison will be more useful. So great. Yes. And I'm not, I'm not holding terribly high hopes for success, but it's harmless to have the conversation and it helps, helps get her involved. Yeah. Okay. Good to know. Yeah. Just keep us in touch with that topic. The next one is something that I put on hold for quite a long time. So I fix a bunch of puppet codes in the infrastructure. So one of the things that you should have noticed now is when puppets change something, it notify RC, it would be easier to track when an update is not done. The main reason to that is because the puppet master is running in a hidden place. So it's not always easy to know if a change has been correctly applied or not. So the second thing is we still have a bunch of improvement that I would like to do is one of them is using let's encrypt with the DNS method. So we could have real certificate for certs, yeah, the Jenkins that I owe for puppet master and a bunch of other services. So yeah, so I hope more than welcome on this topic as well. But yeah, more and more things should be coming here. Though we still have a major work that need to be done for packages in Kizorayo machine, because the perpetrator is still disabled there and because we need a bunch of manual changes, we have to be sure that once we reapply perpetrator, we do not hold back changes and fixes on that machine. But I don't think I will have the time to work on that right now. I'm just looking for small improvement. And while I'm also talking about puppet maintenance, I'm planning to use update CLI to also did the current images there in the puppet repository. So I did the fix needed to use it. So I should be able to enable a bunch of configuration there. So yeah, that's all for me on this list. And we are running over time. So any last question before we end up this meeting? What's the backup situation of package Jenkins IO? Is that just getting backed up? So if you're talking about packages, they are replicated in several locations. One of them is archive.jnkci.org. And the other one is the communities cluster. So we have, I mean, they are back up in different locations. Okay, thank you. But while you're talking about this one specifically, and it's something that we have to improve on the puppet codes, right now, when we upload, we upload archive and we upload to the OSU S3MIROS. And OSU S3MIROS is the only mirror that has a nursing demon running, which means that when people, when someone want to add a new mirror, they just have to synchronize with OSU S3MIROS, which means that they do not contain every artifact that we have on package Jenkins IO. The only sync with artifact that were published over the last year, I think. So there are mirrors that delete automatically files older than a year, others not. So that's why some files are available on all VR and that this is here on every mirror. But something that I would like to do is I would like to enable her sync on archive.jnkci.org or maybe another service. But the idea would be to allow the mirror to synchronize to full content and not only those that were created over the last year. So we would have more people back up our packages. Okay, I'm going to go. Yep. Bye-bye. So I propose to end the Finnish meeting now, unless last topic that you want to bring one time. Two, three, then thanks for your time. Have a great day, a great evening for other people and see you in RSC. Bye-bye.