 Hello everyone, welcome to the Jenkins Infrastructure Team Meeting. Today we are on the 12th of September 2023 around the table. Today we have myself, Demi and DuPortal, Mark, wait, might or might not be able to join us. Stéphane Merl is there, Bruno Vartan and Ashutash Saxena, is that pronounced properly? Yeah. Cool, so welcome. Let's get started with the usual announcements. First of all, the Jenkins Weekly 2.4, no it's 3.1, is released. So the packages, Docker image are out. So last release elements as usual will be done by either the documentation SIG, mostly Kevin and Mark, a bit later, will be done later today. That means the Jenkins Infra Team is ready to deliver that new version to Infra-CI and Weekly-CI. Mainly, Stéphane Merl, you are ready to go. Usually it's Wednesday morning on the European time zone. But yeah, you have the go, you can proceed. CI, Weekly-CI. That's all for the weekly release. That's the first release without us mirroring the Apache Central Maven repository. So that means that's really going in the right direction. Since we removed them, we didn't have any major thing. We only released plugins. So next step will be the LTS release that should go fine and security plugin release. Do you have other announcements? Nope. Okay. Yeah, maybe. Oh, that would be the major event. Sorry. No, I was thinking of the world. DevOps world? DevOps world. Yep, anonymous calendar. So upcoming calendar, we should expect the version 2.432 of Jenkins next week. It will be in September 19. That will be the same day as your team meeting. Don't remember for the next LTS. So let me check last week's notes. Yes, might cheat it. 26 for 26, maybe. No, not 23. Oh, the next LTS release. Yeah, I'm wrong about the number. It's so the weekly release today was 2.423. So you are correct. Next week will be 24. And the next LTS release of Jenkins will happen the 20 September 2023. You can remove RC. Yes. Absolutely. RC was done last week. And that version will be 2.440.2. Do we have announced Jenkins advisories? Unless you have something about the weekly or LTS release. Nope. Okay. So we had Jenkins plugin security advisory last week, last Wednesday, during our past milestone, which went very well. There wasn't any plugin that we are using, except job configuration history on one of our private controller. But that's not a concern and has been updated. So nothing to say about the release, the security releases. Last elements, next major events. DevOps world. DevOps world happened in New York this week, this third day, and it's Friday. Then Chicago 27 September, and it will be in Santa Clara in October. I don't know for the Asian and European steps, if they happen or not anyway. So we'll wait for more to give us more news next week. I think London is already planned. And I think it's January or February next year. Cool. Thanks. So if you want to meet some community members, that's one of the opportunity. And there will be a session. Sorry to interrupt. There will be a session after each day of conference, which is called let's talk about Jenkins. So it's not ask me anything. It's just just talk about Jenkins and one community member will be there to discuss Jenkins with all the participants. Let's talk about Jenkins. Nice. Thanks. I believe the first time has been confirmed to happen. Yes. Three and four of February, I think. February 2024. Okay, that's all for the upcoming major events. So before we jump on the task, the infrastructure task, it's for you Ashutosh. It's your turn. So can you explain us your requirements under this recording? I will take notes. If you need me, I can lead you and let you share your screen or you can guide me on that screen share. You drive. Yes. I think you have been frozen. We can't hear you. Is it only me? I can't hear him. Ashutosh, I think you have been kicked. Yeah. You don't seem to be muted, but we can hear you. Okay. Better. You're back. Right now we are using GitHub Actions for uploading Docker images and my personal Docker of account for hosting the images, which should not be trusted by users. So we need official Jenkins account and Docker Hub for images and for testing the images and if they're working or not, we'll need to shift from GitHub Actions to Jenkins CI. Jenkins.CI. So today, you are already using GitHub Registry to store the Docker images. Is my understanding correct? Yes, but they are hosted on my personal repository right now. Oh, I was speaking about GHCR.io, which is a container image registry like the Docker Hub that can be used publicly or privately like Docker Hub, but it's inside GitHub, which has the advantage to, if you use GitHub Action, you immediately can start working on building the image on GitHub Action and then immediately store it and distribute it through that registry as a first step, of course. Right now, I don't think we're using that. We are just loading the images directly to my own. Okay. Okay. I believe it will be safer for you and for the project to start with GHCR.io. The reason is that anyone who has the maintainer permission on your GitHub repository can also deploy an image based on the build from GitHub Action. So not only it will be safer as a first step, again, it's a first step. Not saying we don't have to move to check in something, but right now it's a first step for you because that will allow you to use a Docker image that can be shared with others, not only you. It's technically possible with your current setup, but that will be easier for handling the project to avoid staying on a one-person project. And also that should unblock you until we make a decision on where to store the image for that project and do we need to store it as an official Jenkins asset? Does it make sense for everyone? Yes. So now about the official Docker Hub organization that are storing images for the Jenkins project. We have two main organizations today. We have Jenkins. For instance, you have the official Jenkins controller image, which is, for instance, that's TAG. That's the Jenkins organization and Jenkins infra. But Jenkins CI infra, sorry. CI infra, yes. So that organization has some images such as, let's say, this one, for instance. These are two examples. Each organization are mapped to a corresponding GitHub organization. Maps one to GitHub organization. We have Jenkins CI for the first and Jenkins dash infra. I will add the link on the notes for richer notes later. So what is the difference between those two images and how do we decide when the artifacts that we produce are consumed directly by Jenkins users, such as the official Jenkins image, official for agents controller, a plug-in HPI file. Then we decide to host them on the Jenkins Docker Hub organization with the code on the Jenkins CI GitHub organization. When the artifact itself is not directly consumed by the Jenkins users, then it means it's mostly used on the Jenkins infra code organization on GitHub. And if it produces a Docker image, it will be stored on the Jenkins CI infra corresponding Docker Hub organization. The example to make it easier to understand is the website Jenkins.io. The source of Jenkins.io is on Jenkins infra slash Jenkins.io, and it produces a Docker image, Jenkins CI infra slash Jenkins.io. That image doesn't exist on search for it. We have something else, but that's the idea. That website is consumed by Jenkins and user, but that the service and itself are not the artifact. We don't have the need to share or to provide to the community the official image. We are an open project, so the source and the image are available to everyone if they want to contribute, if the community needs to take it over. However, in terms of operating the service, that's the reason why the Jenkins official website is not under Jenkins organization, but under Jenkins infra organization. Is that general explanation okay for everyone or do you have concerns or question about that explanation? Okay. Good explanation. Thanks. So that means now we have to decide what will be the destination of the images from the GSOC project for Ashutosh. This will determine where do we store the source code and where do we build and release based on the trust level. The question is, will the Docker images produced for that project be consumed directly by Jenkins users or will be indirectly consumed? To me, it's directly consumed by newcomers, so new Jenkins users. Yes, this mostly would be directly consumed. Most of them. Okay. So that means that anyone going on Jenkins I.O. will have a tutorial and so they will follow the tutorial and we expect images in the name Jenkins slash something, column something to be present there. Is my understanding correct and is that assertion looks good for you? Yes. Okay. So that means we want, so that's the first step for you. We will need an exhaustive list of the images you want. So the tags, we don't care about the tags. I mean whether we have a 1.0 or latest or whatever, it's a trap. We don't want the latest tag. We should not produce any latest tag. We should only produce version artifacts. But we need an exhaustive list. So I guess we will have something like Jenkins, I don't know, tutorial, controller, whatever. That's the kind of information that we need an exhaustive list because we will need to then check with the person, the way we are administrating that part to create the repositories, allow permission and start the process for deploying these artifacts. Also, we will need a list of, for each of these images, what will be the expected name of the repository? Is it a mono repository producing a collection of images? Is it one repository per docker image? Something else. Is my question clear, Shutosh? Did you hear it correctly? Yes. I got it. So it looks like you have sound issue. I didn't hear you answer to my question. I got it. Can you hear me? Yes. Can we hear you? So what is the plan for the source code? Do you want to have one GitHub repository that will deploy every images? Do we have one repository per image? Right now, I think the tutorials part should be a separate repository that contains images for tutorials and another one for the main docker installation for Jenkins. Bruno, what do you think we should do? You need to justify the technical reason why we need multiple repository over one repository, though. I think what we have for the time being on your docker hub repo is one repo with different images. And I think we could use the same thing. Just one repo should do. And we have various images and sometimes various tags. But I think mono repo would be enough. Everything is technically possible, but must be specified prior to creating the assets. Mainly, the next question, once we know the source and the destination, then the third element is that you have to specify. So this free question needs to be written. We cannot just have the question here. So here it's more brainstorming with me giving tips. But here you need to specify the lifecycle of releases, which means you need to say when do we need to trigger a new build and deploy a new version of one image? Whether it's a mono repo or multiple repository, what are the conditions leading to a new version with a new tag? And then once that image is deployed to the docker hub, what would be the conditions triggering the whole let'supdatejenkins.io website? Does it make sense, Ashutosh, for you? Or do you want more? Yeah, the thing is for the time being, we are overwriting the image whenever a change is made. So I know that's not good practice at all. You know, it's like using latest, but even worse. We have Dependabot working. We may install Update CLI one of these days. And I think we may have release drafter, but I don't know if we should rely on... You are defining the how. I'm asking at your level question, folks. What would be the condition to trigger a new release? Is it a new dependency that we track is available? Or is it once a week? Or is it when a human decides to make a release? Or is it something else? It's more that kind of question because the rest is only technical details that I have no doubt that Bruno Ashutosh and other will solve. Yeah, so whenever a human being decides it's time to make a new version because we made some modifications to the Dockerfiles, for example, but also when a dependency is updated, like, for example, a new LTS, then we have to update. Okay. Depend... New dependency is available. Okay. And then same question, but we don't need an answer now because it doesn't... It's a separated process. So it's... I don't want to block Ashutosh from that part, but you have to think about that. Ashutosh, you are producing images. What is the value of these images? It's only if someone consumed them. So you have the GSOC part and I am a voice of the Jenkins project that say, hey, that's cool. You already have the value as the GSOC part, I hope, but that's a discussion that I don't have any voice on. However, handing over your project to the Jenkins community, that could be you working for the community, that could be someone else. It's not about who and not about how, but more about what again. And in that case, what I'm thinking about is what will be the conditions and you have to think about that to be sure that once you have a new image or the change, what will be the process to deliver that to the end users because pushing an image to the Docker hub is not enough. Because right now for the GSOC, you chose to override using the latest image pattern, but that won't be the case for Jenkins save. And we must not. Does it make sense? Yes, it makes sense. Cool. So it's a new constraint. It's not something that you had to resolve until now, so that the context changed. So it's not that you did bad or did wrong. That's absolutely not. It's just a context change to think about a different context, to make the handover from a GSOC project to something widely used. I let Bruno guide you on the steps, what would be required and what is not a requirement, so a nice two-up thing, because I don't have the scope of the GSOC fully on my mind, so I delegate to Bruno and Jean-Marc on that part. But if we want that work to be pushed on the Jenkins project, these are requirements. Okay. I think we have all elements to answer you. You must probably, the destination will be asking the GitHub Jenkins CI organization administrator to create a mono repository that you, Bruno, Jean-Marc, at least will be admin of. And then we will have to ask the Jenkins docker image administrator to create the repositories and have the proper token if needed to allow building these images, which leads us to the infrastructure parts now. So as an administrator of the Jenkins infrastructure parts, but I don't have administrative permission on Jenkins, I can only work on the docker part for you. So my proposal is that now we are going to open a Jenkins-infra-ldesk issue. So Jenkins-infra-ldesk is a GitHub repository owned by the Jenkins infrastructure team used for everything that has a foot on the infrastructure or docker hub or GitHub permissions. So we have a centralized location where to track these kind of requests. So that means, Ashutosh, we'll have to summarize with the help of Bruno and Jean-Marc everything we say today and you will have to open an help desk issue that will be the action of it. Because then from there, we, the Jenkins infrastructure team, will be able to contact and ping the different actors and have a centralized follow. Is that clear? Yes, it's clear. You will have a CI that can be available on ci.jenkins.io, the public Jenkins. This one will be easy, a Jenkins file. We action from Jenkins-infra-team to track the job for builds. So we will create the job somewhere on ci.jenkins.io. We'll give you the link on the help desk issue. You add the Jenkins file on a pull request and automatically ci.jenkins.io controller will track and give you checks on each pull request and each commit on the principle branch. But that's only ci. The cd part, the deployment, will be on a private part that you won't see. I'm still not sure which controller needs to take care of ci. We have two controllers that are able to push to the Jenkins Docker Hub. We have trusted ci.jenkins.io, which role is to manage the update center and the official Jenkins images. And we have infrared ci.jenkins.io, the controller for the Jenkins infrastructure test, which pushes all the Jenkins ci.jenkins.io images. But a few Jenkins images, just such as the acceptance test harness. At first sight, I believe that we should use infra ci. My proposal is without thinking in details. It's just because that one is easier to operate. We can in an easier way get information because trusted ci, as its name implies, is a really, really closed controller. And I believe the amount of dependencies and things that we can put on the images here start to be way above what we expect to run on trusted ci.jenkins.io. And it's my proposal to have it on infra ci, where we already have infrastructure level images, which is literally the kind of images you are building, except the infrastructure is only for the tutorials, not for the public infrastructure. Does it make sense, or are there remarks, feedbacks, or counter offers on that one? My proposal is here to operate unless security concerns about the image content. We can revisit the choice of the city, of course. The main takeaway for you, Ashutosh, here is that you will have separated ci and city. That's way different than what you are building with GitHub Action today. Today, you can control when I have a tag, then deploy to the core hub, or when the build happened on master branch, then deploy to the core hub. You have control over this one. You will lose part of that control as soon as we will start the official push because we want to run that on a trusted environment. However, you won't lose ability to run integration tests or any kind of test and build on ci.jenkins.io, which is publicly available. So now that will be you to open Jenkins Infra Desk and we will cover these steps in the Infra Desk. Is that okay for everyone? Be aware that we might have other feedbacks that could absolutely say the opposite of what we just said now, asynchronously on that written issue. That's why we need written communication because overall communication is for brainstorming and asking and answering questions. But that one will serve as an official specification that we can personally use and come back to overtime. So that should answer your question. Is there any other question on that topic? Thank you. No problem. If you forget anything and you want to come back or have any questions further, please ask the question. Start a rough issue with just a bit of context and ask your question there on the threads. No problem. We can answer them. That's not a problem. This can leave. You don't. You are not expected to produce something written once and never touch them. Stefan can confirm that we quite often edit our issues contents. The important part is that we have the question and the discussion persisted. So the future Ashutosh in eight months when he will have to come back saying, why did I make that book a file? You will have a written answer that you can think about. And the community will benefit from that. That's my good. Sometimes you even reopen an issue that you have closed. So no big deal. So everyone does that. That's not a problem. That happened. Is that good for everyone? Cool. Thanks Ashutosh for the work you did on this. That's a great project. I hope you learned a lot and have a bit of a net list. And don't forget to subscribe to our podcast. Don't hesitate. You are free to stay here if you want to continue the meeting, but no one will be mad at you if you stop now and do something else productive. No problem. Your choice. I want to stay for once. No problem. So hello, Oleg. Yeah. Am I audible? Yes. Just in case. So I just wanted to hop on the call after the meeting we had with the CDF. A bit to get you up to speed about what we discussed. Can we just wait for the first pass on the board before discussing that on a recorded session? Because it's recorded? I'm not going to share details on the recorded session. I just said why I joined. If you have a few minutes after the recording, it's all I would need. Yes, no problem. I will have some, so no worries. Okay. So welcome, Oleg. So let's start on the tasks we were about to finish during the last milestone. I'm going to take them on the order of my screen. I will take notes at the same time. Bruno, Stefan, I see that you have closed the issue and changed from nightly bills to weekly ABA bills. Can you give us summary of what you did and the outcome? My job consisted in opening an issue. Stefan, your turn. You're such a liar. We did the upgrade. It was not as easy as we thought but it's done in Pakori Beach and Jenkins-Sanfra and Kubernetes management. That's your lie. That matched all of those changes. So right now we're using the nightly bills of the early ABA of JDK 21. Yeah, so it's early access. I think it's from the second week of August, something like that. We haven't seen any newer version of early access at JDK 21 built from Timurine but frankly we'll get the first official version I think 7 days from now. So that's pretty normal but Stefan made all of this work so we will be able to update to a normal early access version pretty easily I think. What do you think? Yeah, I'm confident. And that's a personal joke. Once the new version will be released you might need to remove the Dash EA suffix Yeah, Dash EA sometimes even EA-beta. We'll see. That's okay. Nothing search everywhere and replace all can't mess up. There is Dash EA also to remove. That will be funny. I trust you will think about that and mention it during the weekly team when it will happen. We have JDK 21 track even an EA version. Cool. Nice job folks. Thanks. Anything else to add on that topic or questions? Stefan, you renewed the VPN serial Yup. Anything else about okay. I didn't forget about the event in the calendar so we have now some trigger that some warning that will trigger two and three weeks before. Cool. So renewed for six months as usual. Why six months and not one year or unlimited? It's a certificate revocation list. So we need this one. We need to control finding this one. Yeah, I know. Ideally that should be renewed every month but that start to be quite the work. So six months is an intermediate that has been discussed years with the security team. We can always challenge that more than six months is out of question. Calendar added. Thanks Stefan for updating the doc. Oh yes, I had it a way to get the new exact exploration that which is not exploration by the way. Next update, thanks. And we renewed the GPG keys for Stefan and Erwe because you changed machine created new GPG keys. So we encoded and we also removed the outdated key removed Olivier or black we removed the outdated. So Olivier if you hear us, you are not able to renew the VPN CRL anymore. It's been two years but yeah now because your GPG key is outdated. So that was the only solution we had to be able to re-encode. If Olivier want access again, he can ask us no problem we can re-encode with a new GPG key. That's all. Thanks Stefan. Is there any question clarification on that topic addition? Okay, so next topic exploration of the digital ocean personal access token. So that one is on May. So these tokens are updated every three months. These are the tokens that allow to create and delete every pieces of resources on the digital ocean cloud architecture which is mostly 30% of the cloud agent of CI Jenkins IU and their caching proxy and soon that will be the home of the new updates by default. So you understand that this infrastructure needs to be secured and regular rotation of the tokens is mandatory. I forgot to add the calendar events last time I renewed this in June. So of course that broke the builds for a few days last weekend so I opened that issue I did on the milestone and changed them. But that time I've created the calendar event so beers on me folks next time. Outside this it's like the CRL nothing to had it's a process we run every three months nothing changes work as usual so unless you have a question or need clarification. Nope, okay. By the way I've closed it after checking that every digital ocean related jobs were working as usual and were fixed. Thanks to helping contributor to make their plugin builds work on windows that was a gradual thing about deletion. The fix was really simple but it had to be done so many thanks for that. The contributor confirmed they were able to proceed. Is there any question on this one? Okay. Just a slight remark building something related to Jenkins on Windows is a struggle in itself so thank you for helping a very brave contributor. Thanks. Now walk in progress. What are the tasks that we weren't able to finish? Some are cross milestones some we didn't have time some are new so let's cover them one after the other and let's continue. First one no action required I thought I removed it from the milestone so I might have added to the milestone here so let me remove it immediately so there is a plugin that will be removed from the plugin Jenkins because it embeds the proprietary dependencies and then there is a long discussion about what will be the solution. There is no action expected for the Jenkins infra team that's why I'm removing it from any milestone. We don't have to track this one. It's more for information. I'm going to take them in my order of priority. The most prior today's artifactory bond with reduction project. We are officially not mirroring Apache Central Maven since last Friday. Hi. Mark asked for a g-frog and received the logs until Thursday or Friday. So now we are tracking logs. The idea is that in two weeks we will start to see a positive impact. Why two weeks? Because we have cleared the whole cache system so now we have to refill that cache which involved downloading a lot of dependencies. Only once but we have to download them. By the way there is an issue on the Maven HPI test plugin and I need help. I understood the problem so I partially understand it and I need help to drive this to completion. The integration test of that plugin are broken since we started to fine tune settings XML. I need help because it involves a lot of all Jenkins decision that were made in 2015 with Maven whatever version. The initial idea that Basil also pushed on was to remove the old plugin in charge of trying to proxy every dependency so that each integration test scenario doesn't download the dependency each time. However, that plugin does not resolve properly the mirrors and everything. So it was failing on CI Jenkins but not on contributor individual laptops. My problem is whatever I tried when playing with settings XML use for this test. I cannot reproduce on my machine the error I see on CI Jenkins even with the same Maven setup. So there is something I'm missing maybe obvious but I need help from let's say specialists to give this to completion. If I can't find anything on my own I will have directly Basil Daniel the usual every Jenkins contributor I could find from the past seven years that was just a note anyone interested you can look on the issue. It's not blocking artifactory because the thing is that we cannot afford going back or GFrog will be really mad at that problem but we can't afford not having integration test working for the Maven HPI plugin so we need to solve that second issue as soon as possible. Is there any question for this one? Okay. I'm putting now I am 64 so Stefan can describe his work while I take notes. You want to pose in fact. I try to create an exhaustive list and in fact I missed a lot so I have to work again on that exhaustive list which is not exhaustive but the system shared pipeline library that built our images is now fully functional even with the legacy part for windows and the new one even when we specify path and specific arguments so I switch to migrate a few more to IRM 64 for the public K8S cluster and we it's another issue I merged two issues in my brain because we also work on avoiding IRM on the same node. I did prepare everything for wiki and for account app so account app should soon be ready to IRM 64 Okay, cool. Thanks. So that mean we have the list then let's process I need to merge but my proposal is that we start with what is in your list so you don't need to spend more time on other images is that okay for you and we proceed and that mean we can also hand over tasks or move if there are some that are easy we can move them to new contributors if they're interested Great. Is that okay for you? Yes but we need to add the fact that we need the CLI for IRM to make sure that we got the condition to make sure we have the IRM right now it's not written anywhere I did in the issue not absolutely exhaustive but a good start warning about update CLI must be aware of the IRM 64 images and check both IRM and do you agree Stefan that it's an opportunity for us for each of the images to push them no no no for each of these images to properly check and fix the life cycle meaning we should not use latest anymore on any of the images and instead use our automatic tagging deployment system yes to pin them yes I have to switch images with an SH that we build to send their automatic deployment instead of latest or manual or weekly updates and they say friends don't let friends use latest so we were using kind of latest that's something we checked with Stefan because we had to update some images with packages regularly because you don't want a CV so first problem is that some of our images held up for not naming it hasn't been changed in two years which is catastrophic from my point of view and the process should have been update CLI or whatever tool say oh this week what is the SH of the latest version you can surface to the operators to the production operators that oh there is a new SH there is a change so you have to find a change in that case it was a SH for numerous reasons including no solution were able to to get the source of reference for these images because we don't have tags so the pattern for building these images is the same as what Ashutosh did for the GSOC we were building a main branch and we were overriding the latest tag and then we had that detection system that say oh there is a new SH that's not bad or wrong that's just how it used to be which now with the RM64 problem we need to switch to tags because when we deploy a tags version we defer to the Kubernetes cluster the choice of oh I'm running an RM64 I want to manifest for the RM64 for that tag the image otherwise if we don't use SHH checksum if you want to stay strict that mean we need to update all of our Elm chart to say oh the Elm chart must know based on the node selector oh I'm going to run an RM64 here is the checksum for the RM64 or Intel or then my checksum is there which mean updates I need to keep track of both SHH not a problem it's just we have to know which strategy and today we are enough and we have enough money to sustain a version based with tags because that would be a way for us to avoid dormant images like LDAP does make sense yes I think we also need to spend a few minutes few few hours to improve the chart pipeline library to tag and push in one pass and not having main tag but we need to speak about that yep speed up the docker image library to create tag to create push tags at the same time for both SHH and docker instead of running additional builds so Stefan can I ask you for the upcoming milestone to open an issue describing the expectation here you don't have to find a solution on the code you just have to describe the high-level workflow and I propose the following reason that will decrease by half the number of builds we do on infra-ci for docker images or at least 25% but in any case it will be less credit spend on Azure yes, agreed issue to create right thank you why is to decrease build costs from infra.ci is there any question about RM64 did I miss something on my notes that you said Stefan nope, can we speak about the anti-fint here right after this one because it's matching we are let's go with high availability of the services on the public cluster thank you so what did you do on this one not much I just prepared a request to make sure that any kind of application that have more than one a replica account got this replica account on multiple different nodes and of course I started with the RM64 because they are the one that are most more probability to get that because there is a few nodes and that means that Kubernetes tends to spawn all the replica on the same node cool and I was able to review your pull request and validate on a local cluster that it works and the next step is to deploy and see how it behaves on real life with a.k.a. auto-skiller because that one cannot be tested except in production so we might bring down the wiki for a few minutes but better doing it in a controlled way so that means you need to decide an operation time open a status check in SAIO and then we can merge and see that one that shouldn't take long I was about to do that right away after the meeting so I will have to wait if you trust that it won't break the system no problem otherwise there isn't no problem which means then we need a list that's the new but that should be almost the same list as the one you did for the RM64 a list of tasks to apply this then we can create issues and mark them as a newbie and then that means you will be able to drive the newbies if you have new contributor that want to try opening pull request and contributing you can drive them give them feedbacks we have Octoberfest coming while you can spend your brain on something else more valuable than drinking the beer for instance is that clear perfect no more question okay next step blah blah blah I'm going to report for Hervé about updates Genkin Sayo migrated to another cloud so problem statement update Genkin Sayo run on AWS on whole virtual machine which is not even idea available that's a really important service that's the most important service of the world infrastructure as I would say the goal is to use mirror redirector system so Hervé was able to build an HTTP redirector which is running on Azure and now is started to parallel tracks starting a mirror on digital ocean using the same chart as we have on Azure cluster so we use the same bricks that will be the default mirror that will be hosted in Frankfurt then in parallel is working with Cloudflare to start a sponsor chip so they will host mirrors of this one in the form of S3 buckets the goal is to have one on US East and one in Azure event so the proposed timeline will be having a brown out before end of September to try last week of September unless we discover something complicated to move the update tank inside a domain name for one hour to the new one on Azure and see the result of course we will test with controller before that and we will ask from security and usual contributors but the whole idea is that we have redirect so we can continue using redirects that will remove from 5 to 7K of bandwidth billing from my AWS worst case if everything migrate to digital ocean so digital ocean primary mirror is being built worst case billing scenario that should be around 700 bucks per month which is 10 times less than in AWS so even the worst case that would be easier for us if we cannot sustain both CI GenKinsayo builds and update GenKinsayo on digital ocean we will move CI GenKinsayo somewhere else or find new new sponsors and so airway has started contacted digital ocean or is going to to increase or move the the sponsorship here the goal is to leverage the impact of moving that service to Azure on Azure itself loadflare answered and is evaluating sponsorship for US and Azure worst case we have a box for we can sustain one year of mirror for one year on US as secondary fallback we have a clean deployment of LM charts so the next step for airway will be to continue installing the mirror on digital ocean and then deploy the update center Gison index on all the mirrors at the same time so that will be done by trusted CI GenKinsayo every 15 minutes so that will be the next big step is there any question or clarification on that the most important topic one, two, three, okay so now, now, now I got a few SSL certificate expires in 25 days so Mark thanks Mark our best monitoring system ever detect that the 5th of October the certificate for this free domain will expire so after a quick research when we moved on migrated AKS clusters a change was propagated that means these domains are the three of them are pointing to our cluster which redirect them to fast leak for WW GenKinsayo CDNIs the version of the website the thing is we still need to have TLS connection before the redirection happening and the culprit is there is a problem with the certificate renewal inside our AKS cluster I try to give a summary here but we need to remove one certificate let's include setup from Fastly and remove one ingress domain from our AKS cluster and then renew the certificate that should solve the problem once for all the main reason is because we have one ingress with four or five hostname which mean one certificate with suggestions alternative names and one of the domain names WW GenKinsayo is failing which make the world certificate renewal to fail and why is it failing because WW GenKinsayo is managed by Fastly themselves let's encrypt include it I think every and I missed that cross domains due to how fast it works so that should be the culprit so we are working on this one for the upcoming milestone I'm taking it by default any question? next one is we had unexpected slow times on some bomb steps so basically the bomb is a way to track GenKins bugs and there is that long running bugs when you have a lot of concurrent agents running a lot of concurrent pipeline steps that make it slow down somehow it's really hard to reproduce and to track but when you have 300 or 400 parallelized steps which is our case for the bomb steps some SH steps and now as Mark discovers some stash and unstash steps as well takes minutes when they should take seconds so for instance seven minutes to run a curl head on one of the ACP the SH process takes less than one second but then it takes seven minutes for GenKins to retrieve the state, close the connection and do whatever they do internally so due to that JSC helped us and we enabled on some of our controllers a new startup flag which changed the way the pipeline cps system, the durable task when you restart controller pipelines can resume after the restart the way that mechanism works it changes its behavior I don't know the low level details or I don't want to share them now it's public but we are trying that new mode now that should have a great impact so now we are going to watch and measure the impact on the bomb bits is there any question on this one okay so now I'm going to track this with JSC this one has been removed Gira login page on the left status mark and high we didn't have time on this one Stefan, status on matomo yes there is two different parts there is the MySQL instance that I started to create as code infrastructure as code so we did merge the network part and now we are I have a request in draft that need some discussion and review that should provide MySQL managed instance to use as for the rest the image I've done nothing yet on it okay so new subnet draft for the terraform manage MySQL instance so we need to discuss is that correct yes normal work yet on the docker image no you you already started to clean right I I think I did start something two or three weeks ago but not this week but normal work yet okay yeah I think something like I am 64 that's what I started cool next certain email from SPF I didn't have time to look will you have time to look on this one Stefan honestly or do we move it to backlog you mean by myself the problem will probably be to understand where those mail are coming and where is sending them and will circle but that's the problem that's just a question do you feel you will have time given I'm off and there is most probably he'll for one or a few days I will try I will try okay that's something I like my let me have some time to spend on we'll try to find some things just a word about Jenkins Infrapecary Mage from our new contributor was not able to visit us this week he started to work on playwright artifacts because we discover on the CI Jenkins IO agent whether container or virtual machines but our Linux we had unwanted artifacts mainly package, json and dot modules and that wasn't a blocker he found that it's because of the sanity check because we check that given the dependencies we installed during the provisioning then we should be able to install playwright test and run a command by installing that test framework and the associated command it will embed the tools and its dependencies creating these leftovers so we started a discussion do we want to have that tool as a tool and we need to track it instead of using latest or do you want to only keep removing them as part of the sanity check so Stefan I will let you continue the discussion from there since you initially implemented a playwright I try to extract information from my memory and put them on issue but I need you to help the contributor if you can't we will brainstorm with him next week if you don't have time Aida I didn't even remember that but yes I will try to move my brain as I said on the message recommendation is to only clean up artifacts after the sanity check and I believe they started to work on Jenkins CLA documentation as Mark gave them but that's more non-infrastructure topic so thanks for that work non-infrastopic so just mention because we have someone else helping us so that's very nice and that's all for me sorry I kept you really long do you have other topics you want to have okay I haven't seen a new triage issue so that mean we can close so quick reminder I'm off Thursday and Friday shouldn't have an impact but yeah at least let's work done but I trust Stefan will do a lot he already did and we'll see if he's going better any one last word before I close the recording no okay I'm stopping screen share thanks for everyone I'm stopping recording see you next week for people watching the records