 Hello everyone, welcome to the Jenkins Weekly Infrastructure team meeting. Today we are the 13th of June 2023. Around the table we have myself, Damian Duportal, Herve Lemur won't be able to join. He's on the train and it appears that the data cellular network is not really working as expected. We have Marc Waite, Stephen Merle, Bruno Varten and Kevin. Hello folks. Let's get started with the announcements. The weekly 2.410 was really successfully at least on the technical bits. We have packages, we have release an artifact, we have Docker image. And I assume the last bit need to be done later around the changelog and the last item. The changelog is done, published and visible. I haven't checked the containers yet. Okay, cool. Artifacts, packages. So I checked the Docker image. Thanks for creating the tag mark. Change log bits published. We are okay. So, Stefan, you are ready to roll for updating our images and deploy that new version to infra-ci. Just a point more. Last week, I don't know if you remember when you created the release on the container image for that previous release. The trusted CI saw the tag. Because it's checking every five minutes. But it did not trigger a build. The reason is because we have had it set up by default that say don't build tags older than three days. But you created the tag the same day, right? And that's the really sensitive, no, not sensitive, right? That's the complex setting that the date, the timestamp of a given tag by default is the timestamp of the commit pointed by the tag. And when you created the tag, no tags, no pull requests will merge on the Docker image repository since at least three days. So, we had that issue one or two years ago with the CD process for the plugins. And we had to fix that by creating an annotated tag and then creating a GitHub release using that already existing tag. Okay. So, instead of using the GitHub UI to create the tag, I need to create an annotated tag in a separate, separate clone and then push that tag. Exactly. From your machine, that's okay from wherever you want. The reason is because when you publish a GitHub release, if the point attack does not exist, then it will create it. But the GitHub API does not provide annotation because the annotation means adding metadata that can be signed. That's why it's not possible for the release UI. But there is no problem to create the release and instead of writing the tags, you can point to an existing tag that you created earlier. So, how do we resolve the mistake I've made? Do I need to delete the tag and recreate it as an annotated tag or? No problem. If you create a tag that is not annotated, the only check is once you have created the tag, for now, let's just check that in the next five minutes, there is a build that triggers untrusted CI. If it doesn't, like last week, you have to start the build manually by clicking build on the discover tag on Jenkins. And that's okay. That's the only, that's the way to say it to Jenkins. Okay. You've seen the tag. You don't want to trigger it automatically. Please trigger it manually. It will build a tag and push as usual and that's okay. Ah, okay. So, so the fact that if I do an, and I don't, I don't currently have access to trusted, so it would have to be someone who does have access. But the technique then is run the build interactively launch the build interactively on trusted.ci. And it will detect that tag, even though it's an not an annotated tag, it will see that it is a new tag and decide to build based on the fact that it's new or I'm not sure how it okay. All right. Create a new annotated tag and create and publish a release associated with this tag. Or create the release, which I'm trying to write this done, which creates the non annotated tag. And then check on trusted.ci. If the build triggers automatically otherwise started. Okay. I'm mentioning that because right now we just have to ensure that someone kicked the build. So last week I had to do it when I saw the problem and I forgot to explain that because it was after the team meeting. This week we had commits in the past days that will merge usually the we have depend about on Sunday so we merge pull request Sunday Monday so that's okay for the weekly release. Yeah, that's important to have this in mind if the build doesn't start automatically minutes, the perceived timestamp by Jenkins is older than three days. And in that case you can start it manually. Right now we know if it doesn't appear here, what to check. And the goal is still automating or that world part so when the release process will have to create the tag we will just have to ensure that the. It will first create an annotated tag and then create a GitHub release with a GH command line that associated to that tag. That's what the CD process is doing since one year. So it's two or three line of shell. Is that clear for everyone is there any no question on that point. Yes, cool. I don't have any other announcement on my side do you have folks. Cool. Okay, let's continue. So next weekly 2.411 should be released next week the 20 of June. I've already forgot about the next LTS. Let me copy past from last week. No need to copy paste I can call it out to you Damian 2.401.2 releasing 28 June. Thanks, and release candidate releases tomorrow. Okay, please candidate but no action from our side for the release candidate right just that's a call for others to begin for all of us, those of us who are users of those to begin testing. I'm opening the Jenkins adversaries. No emails in the 16 of May. So nothing. No major event for me as far as I can tell. Same for you. Okay, so it gets started on the test that we're able to finish. One that was closed just after I generated the note so that we get started from this one. And then for opening an issue with a lot of information that were really useful for us to quickly debug and identify the problem. Once we made the head up trusted CI was having some hiccups. The main reason was because one of the security group we applied on the new trusted machine was pointing to the former held up IP that was outside our system. Now that it's fully managed we had to fix that so that's the it's automatically updated when the IP change which was the case with the migration to the new cluster. Due to that we had a few builds but not all builds that was a bit random. I don't know what is the discriminant but some people were stuck since few hours or days. And Jenkins.io update was stuck since at least 12 hours. So thanks Kevin thanks for your pointer we were able to fix that problem we triggered a build and it was automatically updated on a list of pages we had. And since some pages weren't updated we had to wait for the fast lead the cache we waited for the operation today run by Irvi because we were also migrating today Jenkins.io to the new cluster. And in order to validate the full migration we had to fully decache the whole website and fast lead after the migration. The result is that the change that you are expecting is now visible on production. And I understand that it's okay for you to close the issue after my request. Yep, I checked out the site and everything that I had brought up in the first place everything appears to be rendering properly now so thank you very much for taking care of that so quickly. Cool. If only all the issues could be as complete and as efficient as this one I will be so happy. Thanks Kevin. Any question. Okay, so next item is when a user require the mirrored repository. Right now we are trying to not add repository mirrors for tiny project because we are trying to limit the bandwidth on Artifactory. So we added an exception on short term to unblock this user so they can still benefit from the artifact caching proxy, it's enabled and used. But when the one or two projects are building on C agent in saio and are trying to find artifact on the jetpack external repository, then it's not using ACP anymore. So the user, I'm not sure if the user where it was able to find but there was every was able to propose them the changes so they can reinable ACP. So now that's on their side. So the issues consider closed, at least for unblocking the users. Instal renovate and Jenkins info repository. There's been a repository where they helped other team members to use renovate for dating software dependencies. I don't remember the repository but yeah that everything was okay because we saw the renovate dashboard issue being opened by the and now they have a bunch of dependency updates. So that works. Thanks. Thanks Alex for pointing us that we had a data but a public dashboard which is on status junk in saio. That's why it's a public one. We have a lot of private ones. So on status Jenkins. There is I think it's services or monitoring. No, might be monitoring and this dashboard shows metric endpoint for us, such as the latest Jenkins core package available and that one was empty. We had the typo and the configuration file. So thanks for saying thanks to that issue. Everybody was able to fix that, that problem. Any question. Next one. After last, the previous milestone we had to migrate to pet Jenkins IO virtual machine out from OSU or sell to a new machine in Azure virtual machine because the puppet enterprise edition we have is not supporting the latest 2222. So we require the open to 20 machine. Also, for security matter, that's better to have security and credentials on area that we manage. Following that big change, I forgot something we have a web book notification on the Jenkins repository which host to the puppet code. The goal of that web book is that when we merge something on the master branch, it send an event to the puppet machine to tell it, hey, you need to pull the latest changes from the repository. So that has been fixed and now working well. So when you merge your pull requests, the puppet agent are now in the upcoming minutes getting the latest version of the code, which wasn't the case before. Any question on this one. Trusted CI out from AWS to Azure. So the, the heavy part was already done two milestone ago. These were the latest cleanup phases. Now Trusted CI is now using optimized agents. We don't need to add SSDs to the machine. They use local disk, they are ephemeral, they use the same spot instances as what we have on CI Jenkins IO for cost reasons and performances are better. The build time for the Docker image has increased of 50%. So we have better CPUs, the machine have the same size. It's just the CPU generation that changed. So a decreased 50%. Yeah, sorry, not increased. And all the network security group were closed as we saw earlier a bit, a bit too much network security because we weren't able to reach it up today. But that is it shows that Trusted is as improving them of security and accesses. It's far from perfect, but still better than before. The Trusted CI virtual machine are still shut down on AWS. We might want to remove these machines soon, but the issue here is closed. We are now handling everything on Azure for that project. So cleanup of AWS resources to do. Also, we introduced a new change and I wanted to, to have a round table of advices here. In order to reach the SSH question to go to Trusted CI, we've had it and allow list of public IP. That's an additional security layer. We did that initially to protect the virtual machine while we were experimenting until we had proper network security group and firewall. We decided to keep that and see if it was not a blocker because we don't have a lot of person that should access that instance. Is there anything that could be blocking at first sight for you? Do you see that taking an account that the process will be if you need to access it? You need to open a pull request on the Azure repository to have your IP in a list of Trusted admins. And once that request has been merged, then you can start accessing the virtual machine, otherwise you are locked out. That has been a bit annoying for me, but it just annoyance and not slower or blocker. Since I was traveling, I had to change regularly my public IP, but I'm traveling. So that's an additional layer of security if someone steal my machine, they cannot add Trusted unless there is the full pull request change. I was also thinking maybe studying how to, instead of that, only allow connection through the private VPN. That means you first need an access to the VPN, whether you are on the world and once you have connected the VPN, should be able to reach Trusted only through there. That one require a bit more reward because we need to create a network connection between private network and Trusted network, because they are separately virtual nets. But that will allow less maintenance and less pain because if you have your VPN, that's enough. So that's my point of view, but maybe that's more initial work, but then less maintenance each time someone need to access Trusted. So please, I propose a vote just to get a site of what would be the preferred option, at least for you folks. Can you raise your hand if you say we keep the current model, so allowing an exhaustive list of public IP. Okay, two hands. Can you raise your hand for allowing from the private VPN 123. Okay, can you raise your hand if you haven't raised your hand yet. Okay, so that mean right now, maybe continuing as it and see if it's a burden, if it is we have a fallback. Is that did I understood correctly the result of the votes for me that's exactly what I had in mind because we don't really know that rabbit hole that we are going through for the VPN. Looks looks easy agree, but continue as we are most of us are not most of us are not remote all the time. Right. If one of the if a team member were continuously remote or always having their IP address change. We might revisit the decision that I think, unless unless we get in that condition statically assigned IP is is good enough for now. Okay, so for everyone here, if you were used to access trusted agent in CEO, you need to browse to the runbooks and look that the updated runbook that shows you the brand news SSH configuration to replace for one because everything is done for your SSH your home dot SSH slash config file. So you need to check the runbook mention and points to the list of allowed IP. So, if you see any improvement on the runbook don't hesitate but the information is there, but you have to update your config in order to access the virtual machine. Okay. Alright so that and you say that's in the runbook. Good. Okay. We have a there is a DNS DNS record for bonds machine from outside and then you can use internal private DNS from the bounds to the secondary machines. So the setup should take in account using this as much as possible to let us the ability to to change the IP if needed. Any question. Okay, so we should see the impact of this on on the AWS billing of June. I will try to check that billing weekly, because maybe we can start already see things, but I haven't done it yet. Making public the plug in its course project for GSOC. So that has been done. Thanks for everyone working on that part to help our new GSOC users. We rotated the P 80 the personal access token used by infrasci to manage the digital and infrastructure. We do it every 19 days. That's a usual process. It's an odd process to automate. But yeah, so we just have a calendar dated and everything is looking at it. Any question. Okay, let's check now the work in progress. First issue remote repository for repo care of slabs. That was an whole issue not prioritize but it has been moved. Following request from car maintenance, some kind of maintenance that really needed this one. So I need help on this one. I did the mirror, but I can find the lower URL because the URL that may even can reach and download artifact from with the current configuration of the stapler project. That URL answer HTTP 404 when it's artifactory trying to download the artifacts. So I guess there is a kind of go cage on the on the next instance that us that repository. If you're close to the URL by yourself on your web browser, you can have a nice JavaScript user view interactive or an HTML view. But both views does not provide a main meta that has so I'm stuck and I don't know what to do. And particularly G frog artifactory documentation recommends to not use mirror repository and instead run the double you get spider of the remote cache one time and upload this as a local repository. And eventually automate that part. So you would only download what you really require on a remote repository and avoid outbound but with. Of course, we say no to other repositories. And that one is the same except is for sensitive project. So I'm not really sure what would be the direction I might have missed something obvious but I wouldn't say no to another pair of I would know a bit artifactory and may even. So I might ask the people who raise the issue. Yeah, I bet that seems reasonable to me so what I think I heard you say is that the artifactory request to mirror the remote repository is being rejected. Because they're they're refusing to deliver the content to something that identifies itself as artifactory. And because of that, we then can't create a mirror of that thing. So we have to create our own private copy of it, and then maintain that private copy, rather than letting artifactory do the mirroring maintenance of it. That's my understanding yes. Okay. Interesting. So the initial reason was to ensure the quality of service so I don't see that as a bug so we'll also get back to the, the person will raise the issue and will raise the priority of the issue two days ago. So I'm going to show them to show them the problem and see if they have other blockage that might not, that could not be clear on the issue, because my understanding is that they should just be able to point to the system and we have to ensure that that repository is excluded from ACP from to allow the repository to case I'm sure everybody implemented that so there should be no blocker for infrastructure point of. Check with the persons raised the issue. My guess is, I haven't searched in details we could also contact the repository administrator and ask them, ask them if it's possible or if it's wanted policy maybe it's an error, or maybe I just use the wrong URL or misunderstood something. Any question for this one. Okay, so I'm going to move that issue to the upcoming milestone because we need to at least state what is the, what is the status and see what we can do about that. Issue not toleration and taint for not pool IRM Stefan, you've opened the issue it's in triage. So I'm going to let you present the problem and we'll discuss if we need to work on that now put it on back load or close it. Following the creation of the not pool for IRM 64 that we introduced for public KS we would like to avoid application to to spawn on it if they are not compatible with our M 64. We will be dealt by Kubernetes but as we cannot be sure 100% we need to add some taint to make sure that not compatible application will not be spawn on it. It's not easy because the the not pool exists already and we need to make sure that the application that are running on it are able to to migrate on the other pool. Before we erase this one and create a new one with the taint, and then make sure that it's working correctly for the application to spawn. My right. Absolutely. I've added. We had the issue. I've updated one of the application that wasn't sticking the Elm charts to the name of the not pool. So what's what did Kubernetes try to do is so oh I got that mission here. And T affinity or blockers so I can schedule on that there is no constraint forbidding me from trying to start the machine. The thing is, that application has an image that we build and that we only build with the Intel Docker image for that case, which is a different case than the other images that Stefan already tried, which are running on iron. But in the case of that application, it wasn't. So it was failing trying to deploy the new version. So the good thing is that there was no service interruption because Kubernetes does not remove immediately the old version it starts to spin the new version and when the new version of a pod it's is started and healthy. Then it added to the load balancer and start removing the old one so no interruption, but our deployment process was broken due to that. So we did a quick fix by pinning the application to the Intel not pool. That was a quick fix. Now, as Stefan said, if you want to create a secondary not pool, if you want to update the not pool we want to change the disk you want to add things. These operations delete the not pool and create a new one. So that means everything running on a not pool will be stopped and removed. The dot not pool will be deleted. And the new not pool will be created. And during that time, Kubernetes will try to constantly reschedule the pods removed from not pool. This will create service interruption. That means we are too constrained for scheduling. So we need to relax the constraints so that instead of that we would say, oh, I need a new not pool with a new set of toleration. We migrate the workload. We ask Kubernetes to migrate the workload on the new not pool. And then we delete the old one. In order to do that, we need to think carefully of the selection of the node using the node selectors. And what Stefan described, the taint and toleration, which are things to avoid running an Intel application on IRM or a Linux container on Windows host or that kind of anti affinity, I would say. So that's exactly what Stefan described, and that will be an improvement. So the question now for you, Stefanie, is given the current state of priority, do you think you can work on that on the upcoming week? I have no clue. Okay, so my proposal is that we move that issue on the backlog. We don't put pressure on absolutely doing that. And if we find time, we can pair on that topic. Is that okay for you? Yes, thank you. Okay. So for me, it goes to the backlog by default, but it will remove the triage. That's Azure Kubernetes. The goal of the toleration here is that if we have, if we use N chart with images without IRM, we really want to protect ourselves from such a problem. It's not blocker, but that's important. Thanks, Stefan. Okay, so backlog or bonus. The big one, migration of hot public gates cluster to the new public gates cluster. All the services have been successfully migrated to the new cluster. The last one being Jenkins.io earlier today. So all services are now migrated. So we don't have any more services running on the overlap network, except for CI Jenkins.io and the form of a PN that allows accessing CI for SSH. Second thing, we have a healthy and properly configured production cluster that allows IRM as we saw. And third, we should be able to start again, working on IPv6 access to some of our service, including get Jenkins.io, because that should be working on that cluster. Now the next step are they started to collect the cleanup process. And cleaned up everything. For instance, we still have the former mirror bits namespace running on the previous cluster. Because we were waiting 24 hours, particularly for the mirrors for the DNS records to be propagated everywhere in the world, because we still saw incoming requests yesterday. So now we will do a step by step process. Almost all services has been removed from the former cluster to be sure we don't accidentally have requests and we haven't seen any error yet. Mirror bit is really the last one. We have stopped one hour ago, stopped managing that old cluster. So now the next step will be removing the namespaces, waiting one hour before deleting and releasing all the resources of the former cluster. And then we will have a set of tiny, a myriad of minor tasks about cleanup, the former reference to that cluster. So really nice job, Herve, the LDAP migration, the mirror and Jenkins.io were sensitive in terms of infrastructure and the amount of requests and everything went really fine. That was a great example of team building. We had issues due to the tentative of the LDAP last week that has been improved and the final migration was without any request lost. So nice job on that part. We only have cleanup tasks for that. So great job. And I'm adding this one by default on the upcoming milestone despite the survey being off because we should be able to get some tasks. Herve is working tomorrow, so we will take time for handover. If Herve wants to take care of the cleanup, then we will postpone that tasks to the in two milestones when it will be back. We can start working on IPv6. Any questions, comments, any comments? Okay. Next one, install and configure Datadoc plugin on CI Jenkins.io. So that one accidentally led to five minutes of CI Jenkins.io unavailable. So that happened. We have learned from the process. The main point to have in mind for everyone, when you want to remove a plugin from a production controller, start by checking the G-Cask configuration. Otherwise, the restart of the controller will lead to an HTTP 500 error because G-Cask will try to configure an item that is not provided by any plugin since you just removed it from the controller. Let's turn it on for everyone. So better rollbacking the configuration first and then removing the plugin. That's the proper order. That could be a great warning to your blog post Bruno, by the way. Just an admonition saying, hey, don't forget to remove G-Cask configuration before deleting that plugin. Otherwise, your controller won't restart. Or add a note to your blog post, create a separate test environment where you can run, spin up temporarily, test it quiet, and then shut it down. Yes. From my experience, that only covers 60% of the cases. For instance, when you have a production LDAP or SSO system, that's really hard to go until the moment where Jenkins load the production-like G-Cask setup. But sometimes the A4 is not worth it. So, but yeah, that could be a good point to propose by default for most of the cases. Because people who have read this kind of, or the kind of parameters described are maybe HK. So yeah, no good point, Mark. Thanks for the pointer as well. Which means test your change before and don't forget to check carefully your G-Cask setup. So rollback setup and begin. I haven't had time to synchronize with the survey. I would like to do that to take that issue and at least set up the basics for sending metrics and enabling. And I will then hand over to Ervay when he'll be back on what to do with this metrics, if he's okay with that. If he wants to take care of the whole thing to increase his understanding of the network problem here, then in that case, I will let him do. But yeah, so by default, that means that he might use that okay. Hand over with Ervay or postpone. Because now that everything is rolled back and the plugin removed, there is no problem on CH and can say you. Any question? Okay, so let's continue. Artifactory bandwidth reduction option. So Mark, it's your turn. I have not, I have not done my action item Damien and I had that action item already from last week. I have the action item to summarize the results of our brown out our reduced our intentionally reduced functionality period. What Damien I did was we disabled Damien did the work we disabled the J get repository at the top level, making it private. We then ran some tests to see, would it still be visible, even though the root level mirror was private, would it still be visible under our public definition. And the answer came back, nope, it's not public as soon as as soon as you make the root thing private. All of the pointers to that root thing that are hidden inside the public repository the public virtual repository are also private, and therefore we were locked out we couldn't download j get at that point. So the, the desired. Gee, this would be an easy way to do things turns out it's not an easy way to do things will need more discussion with JFrog to see if there are other techniques that can help us without having to enable password protection of of everything, or password protection of any things we don't publish. Or we don't generate. Did that did that cover enough of a description Damien and did I make any mistakes that you need to correct. Now that's, that's also my understanding of what we did what we understand and what we need now. Is that clear for you Bruno, Stephen. Yes. I'll, and I'll write a summary of this and send it by email to to JFrog because we need we need some help from them in terms of the approach that's we've we've got. Well, we'll do, we also need to do some data analysis on the most recent data they've provided a new week's worth of data, and we'll do some analysis in hopes of finding other things where we could, we could reduce bandwidth use. I have an additional things that I haven't shared with you yet mark but given that and the direction it's going and given the work that are we did for migrating held up on the new cluster. I feel like I should prioritize a game working on a available held up instance with replication on the new cluster because now we have a fixed network on that new cluster. So the issue I had with the replication should not be present there so is there any objection if I start eventually we can pair on that, Stefan, on ISO and charts that propose to have a replicated held up so the goal will be to install the new instance of held up as we really discovered we have a back efficient backup recover mechanism of our current held up which mean creating a brand new instance with brand new IP. Since we did it recently we are quite at ease with that process. We should then be able to get a restore and see how the replication work on that test instance that might needs trying with a beta dot held up Jenkin say you're temporarily that I hope we won't read the state and probably but the goal would be to say what is the behavior when we start draining the underlying machine on one of the two instance, and if you build up is still there that will also allow us to scale horizontally if we start seeing a lot of request on these held up machines. Right. Any objection if we spend some time on that topic. I think we must. We need to make sure that we to check if we can have also read the only notes on the held up to handle the pressure of the of the parties is an is a let's say implementation detail it will depend on how is the replication process working that we need to assess. Okay. I don't want to spoil everything yet but that's an implementation detail. The goal for us is to be sure that the problem here that we need to study we want to have an held up instance that will first be able to handle the additional load if we enable authentication. That's only the technical part I don't speak about user experience here which is another problem. Just if we had to do that, how the Jenkins info is going to handle that overload. We know we can scale vertically held up, but we would also want to be able to scale horizontally to spread the requests. The second thing is when we need to operate the Kubernetes cluster at least once a month that's what that's what our average. We don't want user running a bit request being in error because authentication is done for one or two minutes. That was acceptable until now but if we enable that on artifactory for all requests then that will be a problem. And the need for highly available that just to be sure we can have an instance we support one instance being done well maintenance or surprise maintenance. Okay. I don't know if committed with artifactory about the problem. The UI not being able to generate the settings XML with encrypted password for you for divorce. I have not that's still my action item that's another one on my list. We, we didn't even discuss it with them in our in our status meeting because we were too busy with other topics in the status meeting. Okay. Is there anything we can do to help. Do you want to end over part of this task to help you mark. No, let me let me just take it. I'll just get it done. Okay. That's that's when I don't want to I don't want to change our interactions with them. No problem. So the question or points or elements on the artifactory bond with reduction topic. Okay, so that one we keep it on the upcoming milestone because that's a priority topic for us of course. Next important topic open 220 to offer upgrade campaign. So we upgraded search CI during the past milestone, as I said last week, one or two virtual machine per week is already enough for that upgrade. I forget for me will be to run. We have two machines running on AWS to tiny machines. I think it's no writing is already a cluster. So we have usage and stats Jenkins. That's two minor services regarding telemetry. So I propose that we upgrade these two machines to 222 during the upcoming milestone. Is that okay. Remaining AWS. And in parallel I'm still trying to work on CI Jenkins I owe a new contour of virtual machine that will run on Ubuntu 22. So that one is later. It's this one in here. Is that okay for you do you have other question about that I will ask. Let me remove the other task. Which task. Sorry, the one you are on because you moved it. No, no, no, don't move it. That's another is just a reference. Okay. So we have that request from giving we didn't add the time to ask to do the initial assessment. I had the message earlier on either the pull request or the issue. The goal is just to synchronize with gave in to do the initial deployment of a matter more instance on our infrastructure. So I guess we should be able to start working on that during the upcoming milestone that it looks good for you, Stefan. I assume we can work on parent that on this one or I will take it by default. Yeah, the main the main job is to spawn my SQL service. Exactly. So I might need your help on splitting tasks so you take some elements I take some, or we work together on this one is that okay for you. Yes. The next task is from the assessment I did earlier today we have all the information we need from giving so it's our it's our job now to get started and do feedback if we need all in required for missions. Any question. The next one Kubernetes one done 25. I will want to upgrade the digital ocean cluster during the upcoming milestone. I haven't seen anything wrong on the changelog that should impact these two clusters. So I would just do a double check. And I would plan to run that operation. Ideally Friday morning. Is there any objection on that timing. I don't, I don't require help, but anyone interesting to walk over on this one is welcome to join. Next one. That's an issue that I think we should be able to close. Damien on Kubernetes I've got I guess I've got an open question so one that 25 reaches end of life in October of of 2023 so we'll roll over to one that 26 next quarter. Or one that 27. Okay, exactly. We have to roll on 125 before end of July. Got it. In any case. And yes 126 should be done during the summer. Thank you. One dot 26 will be a tiny release one dot 25 is a huge one. It's a very free release is that you have huge steps. Thank you. So next issue we had a user having issues with changing Jenkins file and permission on their repository. We haven't heard back from them. I'm a bit annoyed because I feel bad closing the issue but we don't have feedbacks and I'm not sure what could be done I'm currently checking if they merged the pull request they asked me help on. I'm not looking at the proper. So, yeah, I guess. Yeah, they haven't done anything. So, if it's, is it okay for everyone if I keep that issue until last milestone but if we don't hear from them next Tuesday then we will close the issue because there is nothing else we can do. The request is now green with the change that applies the recommended setup for plugins. The build is okay so they should be there and for the problem about the not seeing not being seen as a trusted user. They have to use and follow the procedure for being plug in maintenance. They have direct admin access to the repository so they should they have everything on their mind on their on the hands so I don't know what we can do more for help there. Any question. Point feedback. Okay. So, just a reporting about the new CI Jenkins a UVM instance type. So right now, I'm fighting with the new inbound agents for Azure VM agents that team did. So, it's really hard to set up on instances such as trusted CI I thought it could be easy to test the world scripting of that part because you need to write shell script that will take care of creating the service that connects back to Jenkins. It's not that much, except that it tend to connect to a public URL and with trusted we have a specific set of that makes it hard. So, my initial assessment of it will be easy to test on trusted CI is not a good assessment so I'm going to start testing with a new testing instance like Stefan used to do on CI Jenkins I will try to work on this one. The reason I'm doing this is because we need to first move the agents to the new network. And we need inbound agent for that because otherwise CI Jenkins I won't be allowed to SSH to that submit. So I'm going to start with a new proper setup on the proper network with the proper storage account will be set up. Then I should be able to good strap the new machine and start the test. I did a full test a full restoring tests so that take four hours to back up the full Jenkins home and send kids to the machine and I have an instance with no internet access that starts and I see everything green. I expect on CI Jenkins so that should not create any problem. The thing I wasn't able to test was to spin up real life builds, because that will happen only once we will have stopped the other. And we will have off course issues with blacklisted IP which are not configured as code and written somewhere when reaching other clouds. We might have, yeah, we will have to announce that change and check it carefully during 2012 or so not. But then a lot of bombings are running for instance or not if we need a lot of merge of the code. My plan is not to do that for the upcoming milestone but in two weeks, I will prepare all the work for that this week though. I'm an inbound agent, of course, so that's why I don't want to be too optimistic here. Any question? Point you want me to clarify from that topic. So the input agent, someone is working on it to help you out right now is changing the code of the way it's dealing on Azure. Technically it works, it's just that the proper setup to have something working is not easy to get compared to Kubernetes plugin. You have to find the correct Jenkins URL, the correct URL, the correct parameters and you have to write the shell script properly and adapt it to our system where Jenkins is not admin. So I need to, I think that will be worth sharing that with you because you wrote the part with the Datadoc setup and in the end you removed the pseudo permission from the Jenkins user and I need to insert my code on that script. Okay, and the cloud in this part. Yeah, exactly. Okay. So this morning I successfully was trusted, I successfully started an inbound agent, which script is sleep 10,000 seconds. I was able to SSH to the agent login as Jenkins and run every manual steps with success that created an agent that built Jenkins IO. The thing is that the parameters are used for spinning up the agent where we cannot automate them because I use a direct TCP connection and I don't try to use a gene LP URL. So I use a specific set of flags for the agent process because I need information from the controller. In that case that was manual so I was able to retrieve the information directly on the ephemeral agent, but it's not easy to run. Okay. And that is caused by the specific network setup we have for trusted CI. So I've already opened the issues and I will report back but now the goal is to switch that to see agent. Because it makes sense is that clear. Yes. Thank you. Thanks for that question. And so the last issue artifact caching proxy being unreliable. That will be that also blocked by inbound agent. So here I am. Now can we check the new issues unless you have questions on the current working process. No, okay, so let's check for the new issues to see if we have tried each. Empty the space and a net or the duration and paint we already covered it and I forgot to remove the triage. Proposal for application in pubic. Migrate where m 64. Oh, Stephen. That's yours right. And I need to date it now, because now everything have been migrated to pubic K eight S. So we may have more candidates. feel like we can add it to the milestone and we work together to select one of these applications and migrate it and stop there. It's already selected. That would be reports. Reports? Yes, because it's in the engineering. It is already ARM64 compatible. Do you say compatible? Okay, so yeah. Is that the correct wording, Marc? Compatible with ARM64 CPU? Is that compatible correct? That's correct. I should trust my English way more. Okay, so I've pointed my finger at you. You are a volunteer for that, as I say, and you are working on this. Thank you. So is this a place where we could add checkboxes to each of those little items and therefore get a checklist here that at least I've been using Damian's technique recently. I'm going to put a checklist on this and I'll put a link to the pull request for each checklist and check the box when I hit the link. I will do that. You're right. But only for the second candidate. Yes, that part. Yes, the first one is all the candidates, all the all the possibility, but the candidates are the only one that we can really think of using on ARM. And maybe, and most probably, we will run Matomo on ARM64, by the way. Yeah, why not? Yes, there is an official image. They provide an official image. I checked for you, yes. Exactly. And the only blocker was the MySQL image they used that wasn't behaving properly, but we don't want to run MySQL on the Kubernetes. So no, that would be a separate services. Yes. Okay. So that one is... But maybe the taint will have to come first. No need to. In any case, we can have both. Adding the taint is a, let's say, it removes the burden of specifying the value. I don't see that triage issue recently opened. So for me, we have covered all the topic and we have a new milestone. Is there anything else you want to discuss about the priority of the meeting, or should I stop recording? My actual task is not in there, but I think it's... I think we can stop recording. I have some topics after recording, but I think we can stop recording. So for everyone watching the recording, see you next week. Bye-bye.