 Let's roll! Hello everyone, welcome to the Jenkins Infrastructure Weekly Team Meeting. We are the 21 of November 2023, and around the table today we have self-damaged portal, R.V. Le Meur, Stephen Merle, Bruno Verrarten and Kevin Mortitz. Let's get started with announcements. The new weekly release, I already forgot the number. It's 2.43. We have the war released properly. Package is released and pushed to mirrors. We are retaining the Docker image build, though. R.V. and the HIE have an operation to do on the windows, Docker images from last week LTS, and for windows. Once it's done, we will create the tag build and push the images. It's not done yet, it should be done, let's say, 30 minutes after our meeting. So we'll have a message on Jenkins release channel. We train as windows image re-tagging is required before. What's the status on the changelog port for you, Kevin? It's set to auto merge now, so as soon as it finishes doing it, it checks it'll merge. Cool. So I propose, Stefan, that you wait for tomorrow to upgrade in Frazier, is that okay for you? It's perfect, thank you. We have other announcements, folks. I don't either, so let's look at the upcoming calendar. First of all, next release last week, next week, as usual. Next LTS, I don't know when is the next LTS, since we had one last week. NA. Well, we have adversaries announced. We don't have any adversaries announced. So NA also. And next major event, let me copy past from last week. Because I'm lazy. If you don't mind, I'm going back to the announcement because we had an announcement. We were able to release a LTS last week, right? I've already forgotten the number. Okay. I was right. Released last week. We had an issue during the windows packaging. Scaling up windows. Kubernetes node. Failed with no abuse error. It's by manually scaling up. We're not sure why did we have that issue. So after more or less 20 minutes, we saw that the windows environment wasn't up. So we had to manually scale and relaunch the build, which was already failed. There were a word errors. Looks like AKS had some issue. And restarting the packaging build fixed the problem. So again, we know what to do here. That should be a program of switching from windows container to windows virtual machine. Because a windows virtual machine takes two to three minutes to start. It's way faster than scaling up, then pulling an image, et cetera. But that might need a bit of tests beforehand. So, yeah. We know how to do it. It's not that problem. It happened on half of the LTS release. I don't know why only on LTS on weekly. Usually it works very well. So maybe Wednesday is a bad day. I don't know. No other info issue. Everything else went really fine. Thanks everyone for the help on deploying that new version to the infrastructure on CI release, trusted on third CI. Following that update, I saw the G-CASK change the... We had some hiccups, configuration hiccups on G-CASK along with both new appearance section and data. Plug-in. So we have to be careful to check the G-CASK configuration for on Puppet. Because I don't think we have been able yet to fix it. So Stefan, are they having in mind that we should wait and restrain ourselves to update plug-in on CI and Kinsai on the other until we have checked that the new appearance section has been updated like it was on Kubernetes. I think if it's okay for you, we'll open a new issue for Puppet controllers. Is that clear or do you want clarification on this one? Okay, what do we have else on LTS? Do we have something else on the LTS, folks? Okay, so let's go on the work that we were able to finish or close. Let's start with plug-in details for Ogaton. No work there. Oh, Erwe, I saw you were able to change the closure. Should I be able to... Oh, yeah, I can close as not planned. You have to click on it. Yeah, that's what I don't. Oh, sorry. Oh, no, okay. Okay, good catch. Thanks. Okay, so that issue moved to the next section. We had a user who wanted to have their account deleted. Yeah, it's not automated. There are numerous flows of accounts junk inside you. You cannot remove. So we need to be careful on this one. In that case, there was a doubt about the person because they were claiming to have this email, which wasn't the email inside their LDAP account. So we had to explain multiple times that had to access the account and change the email. And then answer to the email I sent to the private email or to prove by changing things. In the end, they were able to confirm and change the email. So they had both access to the account and they were able to access the email and answer my emails on the private and the new one. So we removed the account. Always be careful because we have people trying to social engineer accounts here. And that could be an account takeover of a plugin maintenance. Be careful on that. Any question? Okay. So we were able to finish the updates on LTS for all the account, all the controllers. As we mentioned earlier. Again, two issues closed around Jenkins account. That's always the same. We explain and no more answer at all. So we closed them. Now let's roll on the walk in progress. So let's start with the date center. Hervé, can I let you report on that topic for everyone, please? Yes, so we've got our last test up and running. The caller is also updating our file share and our packets. The pull request to add the synchronization to update center job is ready for review. This additional thing are behind an option flag. So we can merge it and use it when we want. We'll probably have to plan a run out to test it in situ. And that's most of it for me. Sorry, I was muted. Nice, that's clear. That's a really good news functional test is success. Even though we had even though we had different behavior between Jenkins UI and Jenkins CLA, which was expected somehow. So Jenkins CLA. So next steps. As you said, verify update center to once per is merged. So just a reminder, you wrote it, but don't forget backups. On the on the PKGVM. As you said, plan a brown out. Yeah, and start the reduction of C. Yep. Which should happen before even planning the brown out. To start. If we get enough approval on the. Jeff draft. And also for later or in parallel performance test. Of the new. Is that okay for you folks. I have also. To not forget. Copying the. How did the plugins tools installer. These are tools installer crawler, which are not working anymore. So. These are not in the generated content. And we have to ensure we've got initial copy of them. Installer and. Installer. Installer. You did already had an item on the issue on the issue, right. I've reported it on the issue. Yes. Oh. As well as not updates. Cool. Do you have any updates on the issue reports this fifth five element. Do you have other elements in the UC. No, just to note that we disabled. The. Automatic scan. Which was set with a crown on your orbits. Because since the update center job will run every three minutes. So I already have scanning progress. So. We don't need the. Cron scan. Avoid. Good. Good one. That was an hypothesis. We initially initially stated that. We don't care, but in fact it's creating a lot of reproducible error. Because every three minutes, if there is the five minutes automatic. Instance at the same time, it failed the job. So better. Since we updated every three minutes, that doesn't make sense to have a background. I think one better to have it synced with the updates enter generation every three minutes. And also the last point in the. Full request. I've also did the. Ability to add. Many buckets. For now, we only have one, but. As is. Bucket synchronization are running in parallel. It shouldn't impact the. Total job time. Nice. Okay. Which mean. Plan adding. US East one. Let's say US East mirror. So we will start and put in production, at least during the brown out. And you update center with two different mirror one for us. Because today it's already on the US East. Well, so we don't want to change the latency of the US users. However, the European users should benefit from the initial. Brown outs because they would already have a West Europe. Which is what we have today. And in the future, we can have more. Is that okay for everyone any question clarification. Okay, cool. Let's continue with. 64 migration. Stephen. Yes. From the top of my head, I am. I did the move the. All the, the three. I've got the name. I'm sorry. Yeah, I need that. Plug inside plug inside issue and plug inside API. To the. I am 64 not poor. And I tried this morning. To move the CI, the weekly CI Jenkins.io. But it seems that I was not patient enough. So we'll try again after that meeting. You also migrated. And. Oh yeah. Or the new. You see charts. You see it's a big center for the. None. And so the last. The last mile for that issue is weekly CI. Later today. Nice. Next step after this issue. So you have hoping. Two new issues about that. That's correct. Yes. We will have to state the first one is. Infra CI. Migration. I am 64. And the other one is CI Jenkins. I. I propose that this new issue will be stated and had it on a milestone or not a bit later stuff that. Operationally, that will be the next steps. I also have a next step on top on top of my mind that I haven't mentioned earlier. Public. Gates. Resize not. Because. We used to have big Intel machines. And we started with tiny. Machines that. Scaled quite often. I think we have now seven machines running. Oh yeah. One for the technical. One for the technical services. To Intel and for. Now the Intel are almost empty. While the RM are packed. So the goal will be to decrease the size of the Intel machine increase the size of the IRM machine. And they'll resize that will be a separate. It should be done. But that will have a direct impact on the bill. Does it make sense? Is that clear? Yes, it's perfectly clear. Cool. We can take care of writing the shoe. Both of us in Peru. I don't mind the writing it. Because you already did the heavy work here. But this. There for now. And we'll come on. Okay. Is that all for RM 64? Do you have other. Points to add things, things you learned. All things I learned a lot. But. No, nothing to add. Okay. Next step. It's the contributor. Where is that? Okay. Your turn. Yeah. I haven't. I have no progress to report. For now. I work on it this week. Okay. Okay. Okay. Same for the digital. Sorry. I was traveling. Same status. No progress to report. And I need to write the mail and make you validate the email. Any question. Okay. So now. I got some report to do though. On setting up the secondary subscription. First of all. Permissions. Human admins. So you should all be able to access the subscription. And service principles. So the service principles are technical accounts used by our automated system to manage. As your resources for terraform or Jenkins to Azure. For. For. So that's a good thing. And that also demonstrated the ability to have a single terraform project. Manipulating resources on two different subscription at the same time. For that we use. Two different. Providers. They both have the same ID as your RM for Azure resource manager. But one other specific subscription. And for each resource that you define on terraform you have to select. The secondary the new subscription provider otherwise it's the default one. So I've kept the default one right now. And you must explicitly set resources to be created on the new subscription system. Virtual network. For. Public zone. Created. So that was second step on Azure net project. Created a new public network with subnet. And it's paired with the actual one. Even if it's a different subscription. Is. Back to one. Peering to network allows request to to be sent or to traverse to to cross both. By staying internal. It's as if you had to submit. Without having to go through the internet and paying for the bandwidth. We still have a fee for the whole bandwidth and gateway data. But it's. Close to nothing. It's 10 to 15 bucks per month. That allows us to have a machine on the current public network to be able to communicate with the new public network. For public. So now the work in progress is. Creating resources. To allow CIG in SIO. To spin up agents. In the new. The net of the new subscription. That one is taking quite some time because I've had to. Let's say to increase the. But to fine tune the modules we use in terraform to define controllers agent and stuff. I there is an open pull request in draft. I have to find the modules. To fine tune the modules we use in terraform to define controllers agent and stuff. There is an open pull request in draft. That is being reviewed. That should do the trick. And then I should be able to start creating resources by. Adding a new call to the module but on the new provider. Initially I wanted to do it manually but the risk of errors in copy and pasting were too high. So right now it's a big terraform module split. But that looks to look really, really, really good. And if it work as expected. We should be able in less than half a day to also do the same for trusted and third CI. So all that these workloads could move immediately. To the new subscription on peer networks. I'm really happy with the outcome. Then. Next step. Rested. And so that CI workloads. That's all for this one. The goal is to be able to decrease the bill. And see that we can consume a bit of. Of data on the new subscription on November and be able to pay the sponsorships. With the sponsor credits. Any question. Cool. Let's continue. I saw you were able to start diagnosing doing some diagnostics. With the plugin. Plugin site built failing on infrasy. CI. Can you report with us? Well. I'm still. Diagnosing it because. It is really flaky. Flaky. Only one built. Of five is passing. The young fetch. Step. So it's a bit difficult to reproduce your. I've got it on. I've heard it. Currently looking at difference between. CI. And I've heard it. I've heard it. I've heard it. Different. I might have found something around. I stay home. Environment. Variable. To. CI. And to. Jenkins on. I've heard it. And I'm wondering if I stay as a right to. Have on. The home. The Jenkins. Order. So I'm looking at it right now. The good news is. I've. Successfully. Running on. Without using. The curve in the pipeline. So. If. When I. Got. The running. Probably. Merge. One. And remove the docker. I'm on. CI. Good job. Thanks. Is there any. Question. Things to confuse. Okay. Yes. Yeah, no, it was about the contributors. But life is your. Report but. Question for Kevin. I've asked him. It's. We can wait later. I was thinking about the duplication of. Image. For the avatar, for example, they are duplicated in a static. And a source. Folder. In the robot. I see that later when I. Put in place the. I see this. Thanks. Stefan. Can you report for the. Sponsorship. Oh, I haven't done much. Just calling fewer. Few contact in there, but I. I have to fulfill a form. That have not done. But yes, the content said to wait a little because of that. Even they had. Huge haven't and everyone was focused on it. Had. Last week. Last week. In France. We for president. I mean. Good news. Their event. Had president macro. And. Things that you open source was important. Speaking about. Open source. Yeah, I said. Mark that he used his. Presentation. He used his presentation in New York to. Deal with the open source. Good way of working. That Macron. Said that Mark. Had a good way of presenting the open source. It's too bad. It's not here. But that's a good news. I mean. With the. The marketing and the noise it's doing. Let's. Fulfill the point. And. Push the contact. Hey, we are open source. And the president of the company. He said that's good. So. Do you want to help? Because that's your life. Right. We'll see. Kevin. Where are you able to work a bit on the. Chinese website or is it still. Working progress. And to find out where the. The web page. I built this. And I also built some charts. Kubernetes management and C. And followed the instructions and the Jenkins IO helmet chart. Configuration. The instructions were a little off. So everybody was able to help me figure out where the build. Archive. Zip file would end up and how to get that. But. I had a couple issues trying to set it up. So I changed course. And I started. So I ended up finding a handful of like stack overflow questions, Reddit threads, other resources that were explaining the rewrite call. So, yeah, so I actually understand how that works now, kind of a little bit. So Mark hasn't had a chance to set it up the last time I spoke with him at least. He still needs to do a little bit of work there. He still needs to do that on his end. But yeah, I think at this point, we're probably going to need another session to go over it with you some more. And don't hesitate to ask if you need any explanation or help. Of course. Cool. Good job, folks. Thanks for the help on this one. Stefan, your turn. Again. Oh, God, yes, I did move all the sanity check from the Linux part from the shell script to the to God's. So we got a God's Linux file. And I updated all the data line manifest to point on that new file. So it's done. Next step is to check on the windows part and do the same. So moving all the rest residual, the leftovers of checks to God's windows, and then to update all the data line manifests. And after that, we would have to factorized and get out every common parts to a common file goes common. And, and of course, after that update all the update CLI manifest again. It's kind of time consuming all that. Okay. And my side, I have to report that I had to do a quick fix up on one of the CLI manifest. Oh, I did a mistake. No, you forgot the git version. And it's fixed. Because there is a new 2.43 oh git version for both Linux and windows that has been released recently. It's already available. I guess there might be a security issue we haven't checked yet. And it's, it has been merged on Packer image and realized that yeah, we forget it for Linux. So just have it in mind for windows that you might have to add the kit as well. I mean, given the list of tools for getting one tools is highly probable. That's, that's absolutely normal. There is no problem. And as soon as we catch it, we fix it. That's the only important thing. I'm pretty sure it's it's Bruno's fault. That's fine. I'm joking. Okay, I tried to delay as much as possible waiting for March. First of all, was not working. So his VPN was not working. So for the LTS required team help. So I had to help Mark did not have time with him to check more check with him on his windows machine. I believe there is months configuration or something. VPN search is signed and deployed them. So I believe there is misconfiguration on client side or an unplanned issue. Last one plugin install stats. I haven't followed. It was between Mark and spin it. And I believe no work was done since last week. So no progress to report. So, unless I mistaken all the issues from the current milestone need to be moved to the next milestone because we should work on all of them. Is that okay for everyone. Okay. There are two issues that we should assess if we have if we're able to work on them or not. New issues. Here we are. First of all, I am 64. Stephen. Yes. I am 64 not pool. Will you feel like you can start working on this on the upcoming milestone. Given the current workload. I think so. Yes. Okay. Not the second one, but the first one. Yes. Work on it. So I'm adding it to the new milestone and I'm removing the triaged and second issue. Thanks for opening both of them. So the second issue require some work from Bruno. You didn't know, but yes. I'm sure you already did the work. The goal is to test a plugin builds on a pull request and see I can see you for using a Linux RM 64 virtual machine. Yeah, I have migrated one plugin for timing just to test. Oh, and it's already working. Yep. Okay. So let me add it to the milestone then. No estimation, not triage. May I ask you Bruno to have the comments linking to the plugin or the pull request you tried so we have a work log that oh okay we know that it should work at least on that plugin. Yes, for sure. Cool. So you are volunteering to help, you know, reporting on this one. No, you mean just making sure that it not forget to do this. Okay. Okay, I'm removing the milestone. We won't have much effort to spend on this one. So that Stefan can focus on on infras here first, but yeah, if you can already report the work done by Bruno on this one that will help. So when we will take it in one or two weeks, we should be able to continue. Great. Is that good for everyone? Oh, Mark is there. Hello, Mark. Let's finish the issue and then we can, we can ask questions to Mark. I don't see any new issues here. So, I need to write was the issue that I had raised. Oh no it wasn't there. I'm sorry. Nothing. Excuse me it was a documentation thing different. No problem. So I got a few issues to write. I'm going to try to the new charts and use I am 64 for it's. So, get Jenkins I use is like what I call the legacy mirror bits charts and updates enter work that are made it shows that the new mirror bits parent charts which is a composition of sub charts is now the way to go. We invested time and effort on the new charts. And that new chart allows a Stefan demonstrated this week to run separately. They're single and I am 64 while mirror bits remains on Intel until we have an official RM support for it. So we need to migrate get Jenkins to that new short. That operation can have impact on get Jenkins I also it has to be announced. And the migration need to be done carefully, even though the risk of losing that is not a problem, because we have a ready space and the ready space can be is updated on each mirror bit scan. So we run a scan we wait 10 minutes and it's okay. And the data, the persistent data is on Azure volume, which is, which won't lost any data. So there is a low risk, but it can have an effect on get Jenkins IO starting to serve request slowly because it has to go to the fallback. Another issue to be written off course issue. We had something earlier today. Public gates, not resize Stefan. I will need your help to, I will try to call right this with you. So again, the goal is now that we have migrated a lot of workloads to RM 64. We need to study the sizing of the machines and the scale horizontal scaling to have the best pricing as for today given the current workloads. The first step is to is to use smaller Intel to lower the price first and then smaller in for notes. Check for RM 64 notes size and amount changes. The goal to see if we are able to have the promises of RM 64 saying hey five to 10% decrease of the cost. I don't have any other new issues. Just a reminder, if you haven't seen the message for the weekly release packages and while we're released Kevin did the work on the changelogs which will be done soon if it's not already. However, I've retained the Docker image tag creation, because we need to first do a few operation on the windows LTS images before taking the risk of building the new weekly and accidentally override the tag. Thank you. We will report on the channel. So yeah, it's just delayed a few hours, but we should work on it right now. The date for the next weekly is wrong. It's 28 November. Good catch. That's unfixed. Thank you. Do you have new issues or thing you want to mention? So, so one that's worth mentioning here, not an issue for us, but you should be aware. I received a request from Steve Chin of JFrog asking hey could the Jenkins project please publish an attribution on the download page so www.jankens.io slash downloads that notes that we use JFrog to store our binary artifacts. I've submitted a ticket to the Jenkins.io site to do that and explain to Steve that downloads that Jenkins or the www.jankens.io slash downloads actually is does not use. Oh, you had it. You already got this. Okay, so I'm wasting time telling you something you've already processed. No problem credits, credits due to RV. We'll already let us know on this one. Great. Yeah, so, so the long, the long discussion I was about to do is unnecessary. Excellent. Yes, we will I think we should do it. I think unless there are objections from others, I'll plan to go ahead and be the one to implement it with some description of our mirroring infrastructure and attribution for each of our mirror providers as well. Absolutely good idea. So, Tsinghua, OSU, OSL, Yamagata, Aachen, Belmet, Xmission, etc. They're all we're deeply grateful to all of them. And Cloudflare. Yes, and and when Cloudflare is implemented we'll put Cloudflare there as well and we should put Cloudflare on updates.jankens.io. Digital Ocean because it's hosting archives.jankens.io since two months ago. Yes. Oh, I had forgotten that one. Thank you. Okay, very good. Right. Today's meeting. And so, at Digital Ocean. Right, because they host archives.jankens.io. Yes, thank you. I had forgotten that. Thanks very much. No problem. I'd like to state to ask a review from everyone from the team. I believe everyone from the team should take time on the pull request that Mark will have to check that it's running properly and we don't forget any sponsor. Well, and I think I want to do a separate page that talks about our mirroring system, right, because I think it's a nice piece of technology that we're using with links to mirror bits with a highlight of how grateful we are for their involvement as a project and ongoing development. Yeah, it's attribution on that page and then links elsewhere to a longer description about this is why we mirror and it's crucial that we do so. Thanks. I don't have anything else. Mark, just to confirm the statistics issue, no progress to report. I am not sending any message or changes. No, no progress to report. And I consider it low risk. Therefore, it will probably be a while before I make progress. So you're welcome to drop it from the from the specific milestone if you need to. Okay. Thanks. About the VPN, I assume you did not have time to check. I did not have time to spend on either any member of the team. Right. That would also be dropped from the any milestone because I did my need for it turned out to not be as crucial because the weekly the LTS release was so successful. Okay, I believe we should still ensure that you have VPN access. We should, we should close that issue with success. I agree. Okay, I don't have any other topic. Is that the case for you? Do you have something else to add folks? Nothing else from me. Okay, so I'm going to stop screen sharing. I'm going to stop recording. So goodbye for people watching us.