 Hello everyone, welcome to the Jenkins Infrastructure Weekly team meeting. We are the 28th of November, today around the virtual table we have Stefan Merle and Kevin Martins. I hope RRV and Mark will join us. Let's get started with the weekly release 2.3434. As far as I can tell, packages are out. We have the Docker image to tag build GitHub release. Is there anything on the changelog, Kevin? Yeah, it's set to auto-merge right now. Mark and I just finished up the changes that were needed. Deploy soon. I haven't seen something else on the new weekly release. I haven't checked that the changelogs are proposed. We continue. Do you have other announcements? Because I don't either. Next week we will have 2.45 as usual. I already forgot about the next LTS, but I believe I can copy pass from last week. It will be middle of December, is that correct? Oh no, we don't know. I always forget where to find this information. If it's okay for you. Let me try. It's the 13th of December. David? 13th? Okay. Thanks. Yeah. 2.426.2. Okay. What is your source of truth? This is on calendar. Yeah, the chicken's event calendar. Okay, cool. Do we have security? Advisories. As for the next major event, there will be DevOps World in London. I think it's the 5th or 6th December, and we have the first one, February 4, 2024, in Brussels. Anything else? Yep. Yes, maybe we should talk about the two evenings for Jenkins now. I'm not sure about the official plan yet, but good point. Stay tuned. We will have news about the first evenings. Let's go on the work we had to do. Okay, let's get started with the work that was finished. Stefan, you can get started on the RM64 migration. Yes. I did close the issue because we did most of it. We finished with weekly.ci.jankins.io that is now working on RM64. That was a bit of a challenge for this one because we had to migrate the volume. Most of our public K8S workload is now in RM64. We just kept two services on purpose because for now, either it's too early or we don't have the RM64 building, and we should have it directly from the providers, like for mirror bits, for example, and Matomo, we still have that crazy problem of communication between the RM64.net pool and the MySQL instance. We choose for now to leave them on Intel. Everything else has been moved. Even Keycloak, I believe we might need a new issue. That will be part of the next milestone, if it's okay for you. Yes, yes, but the remaining services that could have been forgotten, that might only Keycloak, but is that okay for you to check? Yes. Finally, now RM, because I believe we also have LDAP, Mirror, and you said Matomo. Stay on x86. My SQL issue with RM64.net pool. Stay. Open a new issue. Next, milestone. They are not in the list. That's why. Services. LDAP. Keycloak. Okay. The main challenge, let's migrate them to ZDRS while moving to RM64. Overcast is close to nothing. Keycloak is there. LDAP is there. And decrease x86 node size. We have an issue for that, that I've created during this milestone. Pool. I think it's on the issues here. Okay. I've added the reference. Okay. Is there something else on the RM64? We've got two issues for the next steps. Through that. Infra.ci on RM64. No, Damien. Sorry. I have too much in my brain. And the issue is the 24. Okay. So four next issues. We'll discuss. So we have to open a new issue for this one. And we will decide if we need to work on it or not. And I believe, Stefan. The nodes pool size is a joint operation with you on high. And you wanted to start working on the Infra.ci steps. Is that okay for you? Yes, it's okay. We'll double check later. Yeah. Okay. Next issue finished. There were Gerard requests that has been done by a team, I believe, or Alex. Same. We had issue regarding permission, but that has been taken over. Usually they end up on our milestones, but they are done by Jenkins administrators and repository permission of later people. Oh, Stefan, I just realized, Erwe told us he had a medical appointment. I just realized that too. I just forgot. Let me send in a message. Okay. So Erwe was able to find and fix the issue about plug-in site build failing on Infra.ci. So what happened is that there were word builds. And I believe he was able to fix it. I don't remember the exact fixed. It was inside Gatsby. And it was related to a specific environment variable that let Gatsby and its plug-in to have a word behavior. So it is fixed. So the work on plug-in site was able to continue. And as Erwe mentioned, we should be able to merge to Jenkins file to avoid having different behaviors. Thank you to the viewers on CIG and Infra.ci in the future. Thanks, Erwe, for that work. Now, work in progress. First of all, we had an issue from a user. Same issue. They told us they were blocked by BelNet.b. But in that case, they weren't. At least on the BelNet mirror administrator, they confirmed that they didn't add any deny list with the IPs we mentioned on their system. So I've requested the user to send us a trace for to see where are their requests blocked. I haven't had an answer yet. So for the next milestone, I propose that we ping them. And if we don't have any answer, we enable the mirror again. And if it failed for them, they have to answer. Is there any objection on this? Okay. Without a response from Requestor, we'll enable the mirror again. We had a request from Oleg. Same. He didn't confirm he wanted to be removed from Jenkins Infra to have less notifications and to be moved to the alumni group. So I kicked him out and re-invited him. I haven't seen any result from him. So I will try to ping him personally. Done. Waiting for confirmation from Oleg. So one last milestone and then we close without further information. Any question on this? Get Jenkins IOMigrate from MirrorBit to MirrorBit parents. No work done. So no time to lose on this one. I plan to start working on this next week. So eventually for the upcoming milestone, worst case in two milestones, there is also issue to open with cost-related to this service because that service is using Azure bucket storage with costs between 1.5 to 2K per month. And that will be easier at first sight that need to be discussed to instead having a centralized persistent volume shared by multiple ILA available instances to have one local storage for each and we update it in parallel. That might need a bit of rework on some scripts on the PKG virtual machine during the core releases that will add one or two minute penalty when trying to synchronize the releases. But that will have at least the benefits of not paying that much. That will match with what we did for the newer updates. Yes, absolutely. The code is in place for doing that so that should be okay. And we should be able to parallelize to multiple locations. Storage is too much due to Azure file storage request. Yes, outside from this, that work still needs to be done. So we will be able to use the same chart for the new update center and that service. Do you think that we can work with Mirabit to have the IRM version directly? That's unrelated to the point here. Agree. There is no IRM 64. But yes, the answer is yes, you can. I talked earlier, they already contacted them, but I'm not sure. We can send pull requests to propose no binaries. If you're interested, you can go on. Just think reserve it to be sure you don't repeat the same work as what you used to do. Yes. Next issue, pull size. So since we moved a lot of services to IRM 64, now we checked a few elements about the size of the node pool. This collection of machines running on our Kubernetes cluster. We already started the cleanup, and as Stefan and I checked, it might be interesting for us to keep the small instances because right now we have on the normal walk around free instances to pack the services, which is a bit less budget than having two medium because medium are twice the size and twice the price. So better to have three than the equivalent of four because two medium is four small. So we should just add a message confirming and explaining in a written manner what I said. But we should be able to decrease the size of the Intel machines. I was waiting for Stefan to close the first batch of IRM 64 issue. So now, Stefan, that we have migrated weekly CI, we should be able to check the usage in terms of CPU memory usage on these nodes and see what we can do if we can decrease to small. I believe we should because same as the IRM 64 sizing. So we should be able to start working on this one. Finally, I saw a message in Azure recommendation about the fact that the system pool hosting the DNS server for the cluster and those elements is only having one node pool. We should have at least two nodes, so we are unsure high availability for storage service, load balancer service, DNS service, virtual network services. So we have to work on this for the upcoming milestone, of course, that should be quite easy. Is there any question for tasks one done, one to be reported and two to do for the next milestone? Stefan, we had the infra CI Jenkins IRM 64 node pool. I think the name slightly changed since let me update it. Oh, no. Is that only about the node pool or is the scope larger than that? I think it's larger than that. I understand now that was not clear enough. No problem. Is that okay for you? We had a comment to update on the scope. I propose we right now we change the title. What will be the scope for you infra CI on IRM 64? Yeah, it's migrating to IRM 64. Is that agents? That's both. In fact, we need to create a node pool on IRM 64 for the agents and we need to move infra to IRM 64. Either we do both in the same issue or we do two separate issue. For me, sticking to one issue is okay, but that means you will have to update the summary. Is that okay for you? In my brain, that was that. No problem. Then we agree on the goal, but I'm asking you to update the issue content. Is that okay for you? Yes. Issue body to update with your full plan. Let's start. Okay. Mark, welcome. Just in time for VPN infrastructure. Were you able to retry or is it still stale? It's still stale. I'll need some help and getting that help is going to require that I schedule a meeting. And complications because where the computer sits, etc. Sorry. No problem. Just wanted to have a check. Okay. Let me mark here. Perfect. Sponsorship as a subscription. Hey, that's my turn. With the help of Stefan, because we are trying to pair as much as possible, we were able to successfully start CIGENK inside your virtual machine agents, ephemeral ones inside a new subscription. Right now, I'm not able to find a way to pay or to know if the credits are automatically filling. So it's my credit card, so that will be the surprise. But the reason is because the report say, hey, you have consumed exactly $0. But the area, the section where I have the credits mentioned the usage of virtual networks and resources we created. And I dig the bits. And in fact, it's because virtual machines started for less than two minutes are not built. And since the only test we run was spin up a machine successfully, then run an MVN version command and then stop it. I believe we are not charged for that, which is good and bad news at the same time. So that means we will have to need to schedule a bit more workloads. If it's okay for everyone, I will prefer waiting for the first of December for adding more workload, unless CIGENK inside you already start picking that new agent. I haven't checked. It's just because it's my credit card, so just in case I will rather wait for Christmas for paying Azure, that will let me fool my book. Anyway, Merry Christmas, Microsoft. We are really grateful from them for giving us these credits. And that also validates the work we did by creating a symmetric virtual network, and we paired them with the existing network that allow a controller in the current subscription to spin up agent on the new virtual network. And they can contact without requiring a public IP and routing through the Internet. Which means now the next steps, I need to add a comment here explaining I will do the same for Infra-CI, Trusted-CI, Cert-CI, for all of our controllers. So they should be able to run virtual machine or ACI container for CIGENK inside. The next step is ACI, because that costs for us way more than the usual Azure virtual machine for CIGENK inside you. Stefan, can you explain to everyone the only limitation we saw based on Microsoft answer when we tried to spin up machines? Oh yes, because that new subscription cannot have spot instances for now. It's not allowed by Microsoft, that's just not technical, that's more of an administrative allowance. Will you say that? So we need to wait for them to allow us to have spot instance in that subscription. So for now, Damian had to prepare and change the card in Puppet to be able to disable spot instance only on that subscription, but allow them on the other subscriptions. Damian, do you think that limitation is an intentional limitation from Microsoft? Because we're using donated credits? No. No, no, it is not, okay. I think it's just because it's Black Friday and soon it will be Christmas, so they don't give them away so easily they need them. I see, okay. I need to answer them later on my email, but that's the exact thing, I think the same as Stefan. So we will retry later. The proposal is that we still start to migrate our workloads and we see the cost of only this February workloads. And then we will decide along with Microsoft in the upcoming weeks. Is that okay for everyone? Oh, we limit to two minutes because like that it's free. Good point. I hope you have optimized your bills. Let's talk about the bomb now. Oops. Ours. Oh, no. Next steps. ACI for CI, Jen Kinsayo. Trusted cert. Infra.ci. VM agents. That's all for that topic. Is there any question? Demian, is that sponsorship something that we should write like a blog post for announcing? Or is this something that already exists that we don't need to announce like that? Good question. I think we certainly should announce it, but I'm not sure on the timing, etc. So yes. Yeah, no, at least in my mind we should. Demian, sorry for my answering on even though the question was asked to you. No, I think we should know that we are sure. We can start preparing something, but I will give you a go-no-go once I will start the credits flowing in the billing for that new subscription. Is that okay for everyone? Because if not, you have to write the post, but on Demian Du Portal, because it will be his credit card. Exactly. Right. And it might also be a good thing for us if we get about the same time confirmation that Digilotion will renew the sponsorship. We then have two blog posts announcing Microsoft. Thank you. Digilotion. Thank you. And that gives us ammunition that gives us arguments that we can use to try to persuade AWS. AWS, you should also get involved. And it's about this time that CDF will submit their big proposal to AWS for all the donations that AWS should do. The first credits to be used for billing. Yeah, good point. Does it answer your question, Kevin? Yeah, no, it's perfect. Thank you so much. Thanks, Mark. I don't have anything else to add on the subscription. Do you have other questions? No, okay. Digital Ocean, no work done. I'm a bit late on the email. I found the previous email. I just have to write it. So expect something before end of week. That's my top priority right now. Except putting stuff inside boxes to do it before end of week. So every is on the medical appointment. You won't be able to report. But summary contributors.chainkins.io website is up. I'm not sure if it has content. But everything is in place for that. I think it's contributors. Yes, we have a website who is fastly in front. It's like the other. The backend is hosted inside our cluster. So now it's a matter of deploying the content of the source code generated by InfraCI inside the production website. And it did change the Jenkins file this morning to build it from Infra. So I believe tomorrow we will have a first version if we have one on the source code. Great work on that. Great work Kevin as well. Because I believe you worked with survey side by side on preparing the topic. Designing everything. Well, I, Chris Stern especially has a ton of work that he's put in that they've put into this that they've been working with airbay on a lot of the infrastructure backend stuff. So just huge, huge, huge thanks to Chris for all their help. And everyone else of course Christina Pizzagale of other people. I reviewed the concept behind the contributor spotlight and the retain Jenkins retain top Jenkins contributors with the Jenkins board on Monday and all the board members that were present agreed wholeheartedly. Yes, this is the right thing to do. Thanks very much. We were very grateful Kevin for your work and for the Infra teams work for Christina Pizzagale and her design of the site and for Chris Stern's implementation of the site. Excellent, really a great great effort all around. And on the infraside that has been an opportunity for us to discover that most of our buckets or other buckets don't have network security rule. And sometimes for good reasons, because we had to to use blobby expert or easy copy tools to copy data on it. But now that we have automated a lot of things, it's time for us to add rules on this one. That's a topic I mentioned earlier, Mark just before you joined about get junk in Sayo, which shows mirror bits in production since one to eventually three years. And we realized with both serve and Stefan by checking the other costs. That's the storage behind get junk in Sayo the storage system costs costs us almost 2k per month on the other bill. I said one for for Mark. So now you said one to two. Yeah, we are almost at we depend on the month is not because of the storage itself. The storing data cost 30 bucks per month. But it cost us before of mainly a right operations and uncached operations on the object storage system. So we are billed because of a really, really large amount. We think without being able to confirm that the cost come from the three minutes update of the update center script, which copy from PKG to that bucket every three minutes, a set of plug-in when it's updated. And the second thing we discover it's not costing us a lot in its less than 10 bucks per month, but that's half of the request on that system or an authenticated request or authentication request, which mean we need to put some network security rule to remove half of the request on the storage system here in any case. Mark, if you are okay, I think I will take a session with you to guide you on the cost exploration. Same exercise as I did with Stefan and Irvi if you're okay. Yes. Do you think that should be a good thing? I believe it should be a good thing for Mark to have the same knowledge level of knowledge for us. Yes, hoping that this time it will not go to a 503 year old, but yes. It's very interesting to pinpoint to the writing operations. So, and is it possible that AZ copy or AZ copy or whatever it is we're using to do the synchronization is doing unnecessary rights that it's got? I mean, that that could be, but that's more than 65% of the cost, just the operations. So, for the websites such as Jenkins, we haven't checked in details. We are sure that the cost I mentioned is only in the case of mirror bits. We have a first set of short terms steps to put everything on read only to be sure that we still see this cost and they come from the source and the network adding network rules as well to be sure we don't have someone trying to write content from another machine that we forget. But also the cost of using a normal block device, a premium standard SSD and the reason initially while why we have that centralized persistent is because each mirror bits instance and Apache server need to read this data and it's highly available. So we were in need of a system that in which we could read on many different machines. However, with the recent update center walk using mirror bits, we discovered that it's not the problem to copy in parallel to a short amount of locations. So that mean we could use mirror bits with local storage with local persistent volume on the data. We will pay instead of 30 bucks per month, we should pay 200 bucks for the storage. That's more or less the storage for normal SSD of that size. That's something we check quickly with Stefan, but 200 bucks is 10 times less than 2k per month. And there is no additional or hidden cost. So then the additional cost will be us changing the script and pick a G that runs every five minutes on every plugin updates to copy in parallel with her sync or whatever system on both volumes. I need to open an issue, but here the I mentioned the network rules because the network rules and the read only volume would help a lot to decrease the amount of request here. Is there any question? None for me. Thanks. That's a great discoverer. Absolutely. Stefan. Yes. Your turn. The sanity check system on the Packer image to have written an automated contract with the developers of CIGEN Kinsayo about what tools and behavior can they expect from the agents? Can you give us a report? I did the Linux Ubuntu part and now I'm working on the Windows part without any success for now. It's taking forever each time. But I managed to have one Windows machine waiting for me right now to play with it and understand why it's not working. The next step will be to factorize the common test between the Ubuntu and Windows and like that we will be able to have a three set of check one for Linux one for Windows and one common. And of course we need to update all the update manifest to match those files and update those files with the new tools versions. Perfect. Is there any question to Stefan? Great work. So let's continue. That's kind of background task. You run a build, you wait one hour doing something else and you iterate. And you cry because it's not working. Yes. That's my everyday life. Next subject. Redirect Chinese pages to English pages. What's the status on that Kevin and Marc? Kevin's made progress, Marc hasn't. And we still need help from Damien at the end of the day. Right. So first step though is let's get Marc to make progress so that Kevin and Marc are synchronized. Then we will come to you Damien for a separate discussion. I had to do this embarrassing discussion in the governance meeting where I noted that Kevin's made progress and I haven't. So the shame factor is kicking in there. I'll see what I can do to make some progress. If I understand correctly, you've already got a Kubernetes cluster configured and you were able to do some initial steps. And I haven't done that yet. Yep. I got Ctl, Minikube, a couple other things set up and going on my machine. It's more just the configuration and interaction between like the home charts and cn.jengens.io part was the mystery for me. Okay. Oh, yeah. Okay. I think I have an idea. Alas, I won't have that much time until next week. And I don't think that I will have time until next week. So certainly us scheduling time this week would only be an exercise in trying to make me squeeze more time into my day than exists. And keep some hours for night. Yes. So that means if it's okay for you, Kevin, I propose that we delay. We don't add to this milestone, but on the next one. So that means next Tuesday we start to bring back the subjects and that will let you time to either experiment on your own or spend your time on other tasks. That sounds perfect. Great. Thank you very much, David. Availability for next milestone. Let's delay off one week. I am sorry, it's just I have a day of Friday and I'm moving stuff. So yeah, I will prefer having time next week that will be perfect for me. Yeah, no worries, David. You take care of yourself. We can come back to this when you have time and you're ready. No worries at all. Thanks. Okay, we'll invert back subjects just to be sure. Update genkins.io to another cloud. So the pull requests on update center two is open. So now it's a matter of time for someone from the GenSec team to have the time to spend on reviewing our pull requests. For that part we are blocked now because we don't want to merge that even if we have the power to do so. And if we have multiple review from each other, eventually from our team or someone will also merge power on that repository. We want Daniel or Vadek to have their own eyes since they build the system and they have all the knowledge required because we don't want to risk breaking the update center updates. Yes. So I've contacted the GenSec team. They have a lot, a lot of things and Christmas is approaching. So right now we consider eventually they will have time to do this before end of year, but most probably January. So that mean, yeah, that one. We can take the time needed in that case and that's whatever they started to work since yesterday. We have a GEP to work on this. Now that we have validated functionally speaking the new system. Now we have two main system. So Erwe will start a new GEP that shows the prototype, the pull request and explain why do we do that. The idea is to have the whole team to contribute, but Erwe proposes to at least now start leading. Then we might endeavor or not depend on availability and motivation of everyone. And for I propose, Stefan, as we discussed with Erwe, for that milestone, you don't have to put your hands dirty on that topic. You keep working on RM64. And the next milestone we delay you should start working on the performance testing. Is that okay for you, Stefan? Yes, I'm sorry, but I never say performance testing. It's for me, it's a stress test. Yep. Sorry. I heard that naming with person walking with Tondem and VAC 80. So you're not that old. Trust me. Thank you. Thank you. The milestone after. That's all for the data center. Is there any question on the topic here? Nope. Okay. Network security rules. Just one thing to check, Stefan, can I let you help me? We will check if we have network security rules on the Azure buckets on this one. And we have to check that we had them and we don't break the service. And we can we can play around before because it's not in production yet. Exactly. And then this is why I would like to have the RM64 for mirror bit too. Yes, but I know. We clearly don't have the availability now for this year at least on forking it and building it ourselves. We can discuss that next year, but I believe the maintainer have taken. We have new maintenance on mirror bits and they plan maintenance release be in December. So that means RM64 should come soon in January expect. Stefan last to pick scale with sponsorship. Can you report? The link we have in the issue to apply to an open source project is dead. There is no more open source project in in scale away. So I was not able to apply. And I have no news from my my friends inside to know where to send email and where to get a contact in scale away for an official apply application. You know appliance. No. Apply. Apply. So nothing new in there. That's new, but that's not good news. But there's nothing you can do about it. If we can't, we won't spend that much time in that case. I propose that we. We keep that topic for the milestone. So we just wait for news. I might know people inside scale with another department. So let's try different areas. And I propose we know. Yes, I think so. And if we don't see, we don't spend that much time and find others. Yes. Any questions so far on scale way? Okay. So next milestone, I will create it after our meeting. I'm late of two and soon three meetings to upload recording. I plan to do it once I will have a decent internet connection and a bit of time. Maybe maybe I can help. There, I'm not sure because there is a part that require a YouTube administrator administration access on the Jenkins. So eventually Mark, but no, it's not time consuming. It's just, I forgot. Okay. Well, but so there are, there are at least I can do that. And I think Kevin might have permissions to do that. So we could deflect it off of you and have that kind of bookkeeping. I just barely did it. So if you don't mind Damien, I'm happy to take the action to, to do the upload. That way we keep you focused on infra tasks and this annoying task of updating the meeting notes is, is somewhere else. No, no worries. If it's okay for you, we'll keep the task because we have a procedure written, but we realize soon that we need a special set of permissions. No worries. That one is not time consuming. It's just, I forgot. Okay. Thanks for the proposal. The thing is next time I know I will have some overload, most probably around first them, I might need the help. But the thing is that you also will have some overload around first them. So that's all for me on the current topics. If it's okay, let's look at the new topics that could have come recently. I've opened an issue from a user. Add it here. New topic create. So a user was asking us on the mailing list about a, they need the public IPs of all the mirrors in a textual form. So they could automate or at least have a source of truth about the public IP that their network infrastructure could reach to download plugin for Jenkins. That's an excellent idea. And thanks Mark for answering and giving the nice idea of parsing the results of mirror stats. So I've written a proposal. We should be able to write job on infrasci that will run a shell script, which at least will parse the result of that page and extract the list of mirror from here and write them on takes the file and the JSON file that could be machine parsable. Because we only have a few tiny list of domain names, which are configured directly in mirror bits. It's not automation as code. So that's why we need a way to parse that page and extract it. But then that will be really nice because we could also in the future update that script to add other public IP such as updates. Jenkins.io, for instance, or all of our public IP, a bit like there is an API on GitHub that I think it's meta or something that provide all the public IP for the GitHub action outbound IP, etc. So the first step will be mirror bits. The user was interested in helping. So that could be interesting to guide them on, hey, if you're able to write a shell script, we can create the empty shell of the repository and then the shell script could be tested on infrasci. What do you think, folks? Don't you think that something exists already in mirror bits to provide those IPs? It doesn't already checked numerous times. That's a good feature request. I feel like you are ready to open feature request on mirror bits. Actually, I think this is one of those same things. People should not use proxies on their corporate networks and people should not rely on IP addresses of things outside their corporate networks. So I'm not sure that the mirror bits people should actually publish this list, but we have people who need it. I will rephrase it differently. There is still area for improvement on mirror bits with the ability to configure it. Yes. Given a given text file, it's ingested inside the system and it's so to configure because right now we have to do it manually and that input could be already available publicly because it's just a set of domain names. But I don't get exactly what you said because Mark, they really need to open the firewall to allow connection to those IP outside of their network. Actually, I disagree. I think they don't. I think they should stop doing that and allow their people to access the network. Yes. A lot of people to access anywhere. If it's people, yes. I think blocking your employees from access to the internet, if you're going to do that, you should probably just cut the wire completely and admit you are there. If it's that serious air gap, but then you accept the business impact of air gap. Exactly. I don't know what you mean then. Okay. So is there anyone okay to help the user? Is there anyone motivated with the time to guide our user to set up that system? So you're asking that, so my assumption is they could parse that file as easily as we can. Am I misunderstanding something? Oh, my proposal was that if they want to help us once they have done the script, we can run the script in a repository of Jenkins Infra and deploy these, or these files inside reports, which is public. Got it. I see what you're saying. So it is that if they choose to automate, if they automate the parsing of the HTML page, we would be willing to, and it would have to be us who update a publication location and a systematic refresh of that data. Got it. Exactly. And the idea, the work for us to do right now will be to create, to specify and plan right what I'm saying in a written manner with something like create a new repository with a Jenkins file that we add on InfraCI and an empty, I don't know, run.sh script run by Infra and that script is responsible to generate JSON and a txt file. We start with this and then they can open the request to run a shell script with the shell script that worked for them. We control the output and the content of the script and we start merging. We're sure we have CI and then we move on. Okay, now it's done. We spend time on deploying to reports Jenkins IO and boom, benefits. Nice. Are you interested, Stephen? I was waiting for you to push me under the best, but yes, yes, of course. The idea is to have contributors even if it's a one-time contributor. The idea is to, people can take the, let's say the ownership on a request they made and they help us improve at the same time. So it's a win-win for everyone. That would be my first repository that I create in InfraCI. I don't mind helping you as we say that we try to pair as much as possible nowadays. So no worries on, you won't be alone on this one. I'm not, I'm not worried. Do we have other new issues? I don't see new issues here. Mark, Stephen, Kevin, do you have other topics you want to add or to discuss? I need to look at the governance meeting notes because I think there was something, but now I don't remember what it was. So just a minute while I look. Wasn't it related to the BOM agent's issue I had to create? Good question. Let me look. I should be able to see it. So there was, oh no. Well, so yes, there was a, create an issue to switch the agent implementation. That was an action item that you had that I didn't find the answer to, but that wasn't my question. It was more, what was it? No, I don't, oh, as your credit's donation and you've already reported on that, that was the topic. And by the way, the, okay, I need to write a separate issue that is somewhat related to the sponsorship, but if it's okay for everyone, I will create a new issue about the, the goal, Stephen and Kevin is to try temporarily to use the new subscription for spinning up virtual machines instead of container for running the BOM. The goal is to be sure that do we have the same slow behavior when we have a big high number of jobs running SH step in parallel on CI agent in SAO? Do we have the same contention behavior when we use Azure virtual machine plugin with virtual machine instead of the Kubernetes plugin with Kubernetes on Amazon? A way of pinpointing that bug in fact. Exactly. Where it's coming from. Exactly. That's the idea. Then the next step is that if we see a difference, we have to rule out the AWS slash Azure and since CI agent in SAO already run inside Azure, the next step will be, oh, let's spin up a NAKS cluster and try to use Kubernetes plugin like today for the BOM with container but running inside Azure. And we see if the behavior is the same and if we can rule out the next one. Exactly. So these are the next steps. I don't have anything else. That's smart. Do you folks? Nope. Okay. Nothing else for me. Cool. So then I believe we can stop sharing, stop recording, and see each other next week.