 Hi, welcome everybody in the infrastructure team, weekly team meeting. We have all around the table, Hervé Le Meur, Mark Wait, Renaud Verrachten, Kevin Martin, and myself, Stefan Merle. We may have Damien joining soon if he can find power and internet connection. So for the announcements, right now, we got the weekly. I'm not sure it's the good one, four, three, seven. Is that the current one? Yes, that's correct. And it is now visible on at least one mirror. I've confirmed it. And so I'm not sure that the three mirrors now. Three mirrors. I'm not sure the tag for the, sorry, the word is coming. The image, the docker image is not done yet, I think. It will come later. Building issue, we, sorry, it took you, sorry. Go ahead. Damien started last week to add some budget numbers. I don't have ASU on AWS. I just have the current digital someone. So I've got not current, but he reported to me recently that we were under budget in November, successful and on track to stay under budget in December. And that we've begun increasing consumption of the Azure donation. So we're not yet at full consumption of the Azure donation, but we are increasing our consumption. We did work on that this morning by trying to use more of the new Azure sponsorship with the packer images built. We worked on that this morning. So it will be more and more on that subscription. Yes. And for Digital Ocean, we saw that we can, we can keep going until January, but not too far in January. So we need to make sure that Digital Ocean is renewing early in January. We'll see that later. But I've got confirmation from Oliver today that he's still renewing it first week of January. Perfect. OK, so let's move on for the next announcement. As it's the end of the year, next week we will all be on holidays between Christmas and New Year. So we will not have any in-front meeting next week. I'm not sure you will have a weekly next week. What do you think, Mark? I propose we cancel it. Any objections from others to cancel it? I'm talking about the Jenkins weekly. Will it be? Oh, no, we will. We would definitely have a Jenkins weekly. I'm too low. At least I propose we keep that because I'm too lazy to turn off the clock. Too risky. You're right. So you will have. I have no interest in turning off the clock but generates those releases. OK, so next week you will have the 2.438. Yes, 26 of December. It's a very good day because it's my sense day. I'm not sure you just use a sense. The next LTS is still the 24th of January, 2.426.3. Let's see if there is any announcement for the advisory. Nope. No, no, no, no announcement. OK, so nothing to say about the security release. And next major event is the forced M and we can confirm that we will be in Brick cell most of us from the 2 to 3 and 4 of February to attend the forced M and the contributors submit the 2 of the 2nd of February. We need to book everyone, planes, train, hotel and everything. It's OK for you. Do you have anything on the appointment calendar to say? To add? Cool. So let's go with what we have done. And starting by someone archive a repository that has result plug-in. I'm not quite sure what this is about. So I can talk to that one, Stefan, if it helps. I can take over if you want. Oh, cool. Or even better, Damien's here. No, no, no, we'll definitely continue, continue. Yeah, you are doing this so good. Oh, I'm sorry, I have a poor failure. Sorry. What we actually have is a truth failure there, right? Can you chair, Damien? Oh, we lose Damien. See? Continue, continue, Stefan. He doesn't want to. OK, so can you talk about this one? I can. The archive of database? So Valentin Delay detected that we've got a plug-in that is designed to access a service that no longer exists and has not existed for many years. And there is no point in having a plug-in that is delivering service to an entity that no longer exists. So he proposed to archive the plug-in. He submitted the pull request to the update center to archive or to remove it from distribution and to mark it as archived. We've done that repeatedly with other services that disappeared. When a service disappears, the Jenkins integration plug-in that connects to that service no longer makes any sense. Fabricator, I think, was one of those. There have been several that went through that, where a service ceases to exist, and Jenkins integration to it is also no longer relevant. So next one, Plug-in C doesn't build an intro due to missing Docker agent. Damien, do you want to talk about this one? This one is for Erwey, I believe, since he took care of this. Sorry. So I started 10 days ago merging Jenkins file for plug-in site. And for that, I also needed AfroRCI to look at the correct Jenkins file. This pull request was merged, but I didn't finish the merging of the Jenkins file. So the plug-in site job on AfroRCI was failing, as it was looking at Jenkins file using Docker instruction. I've since then merged two Jenkins files, and some plug-in site is now currently running on both instance from a unique file. Yeah, so that was just because in the between. Good work. Thank you. So with the TI to the latest version, that's the usual update that we're dealing with. Right access to a new user, CTH. This one, I have no clue. So let's check. That was dealt with by not my fault, was sent an invitation. OK. So thank you. Agents of Ability are failing. I can talk about this one. Yes. So what happens is that since that agent is not managed as code, because we don't have a Puppet 6 package for that CPU architecture, we have to manage it manually today. And well, since we changed the tool installation version of Maven on CI Jenkins IO a few weeks ago to use Maven 3.9.6 instead of .5, that started to make the script failing. And as you can see on the screenshot here, we enforced the path to the 3.9.5 or 6. So it was searching for a 3.9.6 on that agent. So thanks, Mark, for taking care of this by removing the failing check for that agent. I've rolled back the change, installed the new Maven version. So now we have two paths from here. The first one will be avoid alerts and failures by ensuring that when we automatically update Maven tools version on the controller, we exclude that one. The alternative being we keep it that way until we find a way to automate that machine installation. And in that case, when we see a failure, that means we have to operate and updates the agent to avoid the agent rotting somewhere. I don't have a specific or strong opinion. What do you think about keeping it as it and updating it when it fails? Or do you want to avoid failure? Update it when it fails works fine for me. The machine is relatively low use for us. And so the risk is not high. And spending effort on other automation teams more valuable to me than spending it on System 390. And I'm assuming that when we will upgrade Puppet to Puppet 7, that would be easier, doesn't it? I don't know if Puppet 7 or 8 have a native package as well. Theoretically, we could compile the Puppet agent binary and build it. That's not that complicated, but we never spend the time for this. Other alternative is to define a tool location which will be different that will detect if it's on that specific platform that takes care of downloading Maven. That could be an alternative. And we stop installing Maven as it, leaving only the GDK to Maven to install there. That's one feature of the tools that we could have there. But no more action right now. Yeah. OK. Thank you. Do you want to take over? Yes. OK. Can I let you screen share? It's not a problem. OK, so the next one is removed by line agent build history plugin. So we did an experiment with that new plugin, but it appears that that plugin has performance issues. And moreover, we installed it for showing it. But on weekly CI, we don't have any agents which defeats the purpose of showing off that plugin. So we removed it from the image, and we can discuss that topic more in details when it will come back. But right now, performance issue and no agents is a reason to remove it. So thanks, folks, for taking care of this one. Next one. No question on this one? No? OK. Next one, SSL certificate for repo Jenkins CI expires 20 December tomorrow. So we have a three month valid certificates emitted by Let's Encrypt Authority. At the same time, Kosuke started and completed the generation of a one year valid GoDaddy certificate, which means he contacted directly GFrog to install it. So at any moment, the certificate on repo Jenkins will be updated to expire only in one year. I've added the calendar event for the Let's Encrypt by default four weeks, three weeks, and two weeks before the next expiration. When this event will happen, we will decide either where we are able and successful on having GoDaddy certificate. And we just update the calendar. Otherwise, we install a new Let's Encrypt certificate as proposed by RV. That's not that much a problem if we do it in advance. Is it clear? Do you have any question or things to add on this one? No? So next, but last topic, Mark, it's for you. Yeah, so we had a spam outage, a spam incident that's described by Uli Hoffner in that message. December 6, 2023, a spammer modified 1,000 plus Jira bug reports. And some of the modifications were damaging enough that we reviewed with the board and with Jenkins officers a proposal to rollback. And we did. We rolled back to December 6, 2023. The rollback means we lost submissions from people that were made between December 6, 2023 and yesterday's rollback. I have captured CSV representations of some of those things. And Daniel Beck captured others in the security project. But we should consider that things between December 6 and December 18 are potentially lost from issues.jankins.io that was accepted as much better than the alternative of attempting to repair the damage that was done so badly. Some of those issues had their type changed or their connection to a parent changed. And by changing the connection to a parent, we could no longer reconstruct them. And that's just too painful. So we rolled back. And the permission that allowed the damage to be done has been removed from everyone but a relatively small set of users. Any questions there? That's clear. Great, thank you. So now moving to the closed as not planned. So we have a few elements. A plugin was not showing, but it was brand new. It wasn't released and it took time. So thanks for people for helping the contributor. It's now on the plugin site, so no action required. CD release for the Google Earth plugin has been fixed with the help of James Nord. As far as I remember, there were issues on the CD process set up on that one. And dependencies. Yeah, there's still an open question there with regard to git configuration. But I think he succeeded in releasing it. Absolutely. That's a, James was able to fix the unlines that were blocking the build, the main build on CIG and Kinsaio, which in turn was blocking the release. We have a subsequent issue on the git configuration set up. We had a question in the related to IPv6 support so today repo jenkincaio.org and the whole GFrog platform as a service is not exposed using IPv6. So that's the answer. And I've asked the support though, how do they manage their engine users because they cannot use IPv4? They've been requested to open a request for enhancement on their portal. So that's something I need to do soon. But right now, the answer is we cannot support IPv6 only for repo jenkincaio. And we cannot afford hosting and building a gateway for that. That will be too much bandwidth and the cost will be unacceptable for the project. Unless we find a network spot so we can do this for us, of course. Any question? So thanks for this contributor. Now moving to the work in progress, I'm gonna take them on the order on the notes. A new issue opened by Baddiel yesterday, a plugin failed to build since we operated on the G-center removal on the repo jenkincaio. So as indicated on the issue, I did that in pair with Stefan this morning. We reproduced the error from a scratch environment and then we relaxed the constraint on the filter. So the G-center orphan is a new repository inside the public that replaced the former G-center which was used by too many people to download non-genkins artifacts. So right now we are applying filters that select which artifact doesn't match the pattern. If the artifact match the pattern, then it's mirrored and we can continue working. If it doesn't, then it's not downloaded and we receive 404. In that case, we were receiving 404. So we had to use the right level of pattern selection and we verified it was working on the local setup by starting from scratch again. So now I'm just waiting for Baziel confirmation that it's okay. So that issue is almost plausible. Marc, do you mind to move the remove G-center on the SSNATI prepositories as the next issue? So yes, no problem. I think. Seven. Yes, no problem. So you want me to talk to it? No. Yes, and then we will have some add-ons because we had a great idea that's worth sharing here and I need to report the changes done earlier today. So can I let you start? Yeah, so the Artifactory Bandwidth Reduction Project implemented production last Friday and that implementation in production looked good in our initial assessment. We found a few surprises even in the initial assessment. One was, oh dear, I forget. Well, some tools that were surprising us, we've adapted to them on Friday and then, oh, great configuration. That was what it was. Thanks to Baziel Crow for that. Then Damien, you let you talk to the next topic. So while working on this today, outside the first bullet of my last comment is the former issue. With Stefan, we also discovered that G-center and Atlation were still available not through the public repositories that we use for Jenkins build, but still publicly available, which can be a source, a source of mayhem. The reason is because G-center and Atlation repository, which were the entry point of the mirrors, has been removed from the anonymous access. However, the G-center cache and Atlation cache, which effectively are G-center and Atlation, that's all Gfrog works internally, this one were still on any remote and anything. So by removing them, both of them from the both permission schemes, now you need to authenticate in order to check the content of these mirrors, which is still needed for exercise like we did this morning to search for artifact that are no longer mirrored, but that cannot be used easily if you want to download the artifact for free without authentication. And finally, feedback from Herve, that issue from this morning and the line that we are, we still have cached artifact on CI Jenkins IO because the plugin is still building on CI Jenkins IO, while on fresh new system it doesn't. So the proposal will be to clean up the ICP cache now that we have persisted the new settings for last Friday. Okay, with tell me the benefit, that just sounds like an opportunity to consume much more bandwidth as we repopulate the cache. I think, well, I'm prone to say, let's not destroy the cache and repopulate just because I don't want to consume the bandwidth, but I think the argument there is, oh, then that's delaying our discovery of problems. Is that? Exactly, the thing is cleaning up the cache allows us to control when we will start to see plugins that might break like this one inadvertently. Problem is the most used plugin have been tested, but some dormant plugin might have issues and we will discover them on the worst moment during a security release or that kind of problem. I'm thinking a lot given what you said, and I have the following proposal for your folks. Since the goal is to make GFrog happy with the bandwidth, we should avoid until the end of the current month to have a decache that will consume a bit of bandwidth. I don't think it will be that much, but still avoid that risk. So they should be content with the December results. Half of the month will be with cache and without G-center. And we plan in January, once everyone will be back from Christmas holidays, we have a planned ACP cleanup for beginning of January in that case. So we control then we will all be available to fix or help contributor. What do you think? That sounds fine to me, although I'm also okay. I think everybody's logic is sound to say, maybe we should spend the bandwidth now. If we're available now to adapt to it, I'm okay with destroying the cache now. I think either is fine and I think I had not considered the let's find things sooner is much better than finding things later. And it probably means we ought to not just flush the cache, we ought to flush the cache and possibly launch a build of every plugin that has not been built in the last six months. Now, those will discover problems that are unrelated to caching, right? There are a number of plugins like that where it no longer builds and it no longer builds, but we haven't seen it yet because no one's tried to build it. Exactly. That sounds good. I think ACP cache flush immediately is fine and if we want to wait until January, that's also fine. I think it's a good idea. I retract my objection. I propose January as a separate topic. So then we can focus on less tasks for that milestone given the approaching holidays. Is that okay? Yes, that sounds great. I've just edited the note a bit because I believe Erwe might have to go early and since I was late, I just want to free him because Kido was on stuff. I forgot about this because I was late, my bad. Erwe, update center, that should be quick. Nothing since last week, except that working on another LBS issue, I'm reviewing how we are authenticated for the fracture used by the update center. So I will update the pull request to be reviewed by the tank integrity with a command to authenticate. Replacing the SAS token. I say sorry. SAS2SP, to do review by the Gensok team in any case. And as Stefan reminded the team yesterday, we have a performance stress test to run. These are the updates on the step. Did I forgot something? No, there is still some thinking on small proposal reduction, but it's working progress and no real, yeah, nothing done since last week. Nice job, thanks. I see big cleanup in January. Is there anything to add, something not clear? Nope, okay. Next topic, still you, Erwe, I believe this one should be quick. So the goal is to replace Blobix for by the AZ copy command. I believe it's the same status. Yeah, I listed every file share used by different services in Trinklin-Sanfra. And on this plan, I just need to put in place service principle to interact with by share instead of using a SAS token. As Tim, Jacob, and Juan, they are more difficult to rotate and to rework than other authentication methods. Cool. You wrote a finely grained plan, which is very nice because A1 can take over or you can continue after a long holidays with a fresh mind, so great job. Next topic is digitalization renewal. I believe you were speaking about this earlier before I, John. Yeah, so I've confirmed, Oliver, that we could wait until the beginning of January to renew the sponsorship as we still have some credits for this year. He confirmed to me that he will indeed renew the sponsorship first week of January. So we should be good. Cool. So I believe since you will be in holidays, is that okay if I take care of this one January? So you won't have to worry about this. Is that okay for you, Arvet? I believe these are the last elements you had, Arvet. Did you add other subjects? I don't see others. Okay, so you're free to play Santa Claus for children. Not today, but yeah, so. Oh, it's okay. It's a secret. Yes, but I stopped believing in Santa at least since last month. I know it's John Mark Mason. Anyway, next topic, G-Git cloning, not converting on lines and windows. That topic has been opened by James Nord. Arvet, you're free to go if you need to, no worries. Yeah, I'll stay a little bit. So that has to be checked, but however he pointed out that he saw different behaviors between the two countries. So different behaviors between the windows virtual machines and the windows containers, if you can scroll on the bottom, which mean the setup, the thing is I don't know windows enough so I need to help from a window specialist because I don't know that setup. Where does it come from? Is it a setup on the windows-based level or is it on the Git installation that G-Git could reuse even if it should not because G-Git runs inside the GBM and not from the Git binary installation? Mark, stop me if I'm wrong, but I believe if you don't have Git installed but you use G-Git, if you have the GBM with the controller that allows to run Git operation without the Git installation. Correct, although as far as I know we have Git installed on all of our agents. But I think, Damian, this is one that it's probably best just to assign it to me because I don't know that there's enough gain for anyone else in the team to go learn the details of how Git does its configuration and installation. There are a number of possible misunderstandings for James that might resolve this and just needs a little more research but I don't think it's a good use of your time for us to put you on this when I'm okay if you just assign it to me and let me take it. Okay. It just, it's an ongoing topic in the world of text files and how we handle end of line conventions and automated formatting, et cetera. Okay, thanks for this, which mean I've assigned you to this one. Assigned to Mark, ongoing Git, it's already assigned to Mark. I just did it. Oh, I edited the title because I was bugged by the typo on Git. Okay, Mark, but don't hesitate because there is clearly something behaving differently depending on the below operating system. Right. Well, and it seems that there's something different a difference of behavior on ci.jankins.io whether container or non-container. So there are enough differences to make this report very interesting to us. Absolutely. Next one, symbolic link for latest for Windows table. So that issue won't be moved to the next milestone because in fact it's blocked by the AZ copy migration. We need AZ copy to dereference sim link when copying to Azure file storage. So I'm adding a comment on the issue and that one until it's blocked will be back on the backlog. Any objection? Okay. Next issue. Is email not received? I propose that we, I'm gonna close that issue. The person never answered. It's been almost two weeks. So unless someone object, this one is going to the closed list for the next milestone. The problem was the gray listing. The email was sent but gray listed by the receiver and that's something without information or contact. There is nothing we can do about this. Mark, can I let you speak for the next one? Oh, this one, okay. I put this one, this is the modified, whoops. Nope, am I on the right one? Issues modified by spammer. Oh, sorry. No, I was thinking about the next. Okay. Issues are modified by a spammer. I believe this one is fixed. Yes, at least I just recorded it as fixed, but I did it during this meeting. Oh, okay, perfect. So let me move it on the closed. It's actually already there, Damian. Oh, cool. Thanks. So I'm gonna do the same. I did it as JIRA database restored after spam attack, but it's there. Cool. So now you can speak about the DNS domain registration then. Yeah, so there we found, so we're reminded that Tyler Croy kindly donates the Jenkins.io domain name and the Jenkins-ci.org domain name, and it's hosted and he makes the payment every two or three years when it comes up for payment. And he makes that payment roughly 30 days before it expires. And so our alert systems warn us 45 and 60 days before that it's about to expire. And we then watch carefully to be sure that he still renews it. And if he were to choose to not renew it, we would contact him and get his help. Right now we've got one open topic still coming. So he renewed one of them. The other one we expect will renew in the same timely fashion. And in our next meeting in January, we can double check. So I've added a reminder on the team calendar for January just to be sure that with the Christmas we don't garbage connect too much our mind or we can with no worries. Right, we're now down to 38 days before it expires. And so in about 10 days, we'll be within that time range and see if Tyler does the renewal for us. Perfect, thanks Mark. Thanks Tyler for sponsoring the project. Stephen, I believe you both, no one on the team was able to walk on the leftover migration to where I'm 64, is that correct? Yes it is, yes. No walk down to be continued. Okay, because you spent your time on other major issues, so no worries. I'm keeping it there because we can split the burden on that area. This one I did a little, but I forgot if it's two weeks ago or this week. You did things with XQ? Yes, I used the new tool XQ that extract those domain and put them in a file and export them in the reports.junkins.io. And now I need to do more work to get the IPs and build a JSON file and need to work around. But the skeleton is up and running. I'm not sure it's detailed enough, but work in progress. It's the mirror. Next step, XQ installation. Oh, it's done. It's not released and not deployed. That's released. Requested. And so just a word because I pushed a bit Stephen on that area. For that issue, we are going to publish the XT file for now. The XT file, we had to be careful because once we start publishing this on report.junkins.io publicly, that will start to be automated. That's why I propose that we close the issue once the XT file is okay. However, we will need to continue working to produce a JSON file that can be first like the GitHub API, which means anyone outside can automate retrieving the Jenkins Infra IP and we can add more IPs. But this constrain us on thinking a bit about the model we want to provide inside the JSON file. We might think about versioning like an API. So that would mean we would have a version one IPs. And if we want to do a breaking challenge, we should be able to have a V2, V3, et cetera. But that mean careful planning. So that's why right now the XT file is the first step. And once it's available, we have done the whole continuous deployment process, then we can iterate. Any question? So just a word for Mark and Bruno and eventually Kevin. That could be a source of a newcomer issue because updating the script to extract more information using update CLI, shell, whatever mechanism could be really easy and could be easy to contribute and test locally. And that could be nice contribution to the Jenkins Infra project. We just need to set the foundation for the first extraction and then anything on our repository could be retrieved, especially the public IP. They can be exported somewhere and then reused. Just a note, but most probably in January. Any question? Next topic, migrate, change the Elm chart used for get Jenkins.io from the legacy mirror of it to the new one used and built by Airway for the new update center. No work done on this one. It's still to be done soon. Now that we have almost finished the subscription setup, I've started working on this. This one is sensitive. If no one objects, I might try to do it between Christmas and the New Year's Eve because that will be early days. Because that's what the early days. Yes, but that one is very sensitive and I would prefer having less person impacted during the operation. And that's quite easy. I've almost done all the work two months ago. It's just I had to cancel the operation due to other issues. So I would prefer having it on a calm moment with no changes, no one walking, no one consuming. So I mean, I can break it during five minutes without any major issue. Okay. And in exchange, I will finish one hour earlier one of these days during working days. Agreed. Is it a deal? Yes. Schedule between Christmas and New Year's Eve. Christmas. Okay, any question? One as a reminder, one of the benefits of that change of M chart is that we will have a read-only file system on mirror bits and HTTP for get-gen-kin-sio which will allow us to fine-tune and detect where are the rights that cost us a lot of money on Azure. That next step on January. Great. Not pool-sized tuning. That issue is almost there by using smaller virtual machine. We saw that we were a bit, let's say, hungry in resource allocation. So the last miles will be trying to change the new node pool which has five machine trying to shrink it to three machines because 99% of the resource usage for both TPU and memory is clearly above the limits we have set. So specifically for that use case we will change the default rule of thumb that we use on Kubernetes where request and limit are the same. That's a rule of thumb and the good get started. Now what we want is packing more services because we know that these services usually use less resources than what we were expecting. But that has been the reason. It's not, yeah. Exactly. We checked on Datadog, that's why I know that 99% tide is okay. We still need to keep the limits as they are today but the resource allocation that Kubernetes use to decide whether to pack or not the services on a single machine, that one can be reduced. So it's just a few pull requests that should happen today or tomorrow or last case. Is there any question on this one? Okay. To be done. Resource decrease, but no limit change. Next issue in Fracier and RM64, that's yours, Stefan. Yes, let me remember what I did last. Yes, I did set the new node pool in the IRAM on the intro. I did set up a new agent on the Mpracii using that IRAM pool with the RM64 within the label name. I did try it twice. And now I'm working on the Packer image for the Olin one to make sure that we use as much as we can of that new IRAM node pool and to replace the Elm one, the Docker Elm, I forgot the name of that agent. I'll file it up. To converge to use as much as we can the Olin one and to use the IRAM node pool. To be migrated to Packer image or in one. And the next, I believe our Terraform project. Yes, we need all the tools first. That's the problem now. It's working on the Packer image to add all the tools that we need for all the agents. As much as possible. And the hidden benefit is less updates, less updates CLI, less builds and less cost due to that. We will only update dependencies only once on the Olin one image. I believe you started with DOCTL. Yes, and XQ. XQ done. It's not published. We'll say Damian. Is there any question things to clarify on that topic? No, okay. Next topic, sponsorships. Okay, so all controllers are now using the new subscription for Azure Virtual Machine Agents. When they are fmroll, of course, not for permanent agents. That's the for all controllers are using the new subscription for VMs. Exception of release CI, which does not, doesn't use any VM at all, which does not use Azure VMs. We did an operation earlier today. Thanks, Stefan for catching that there were some overlap on the network I created for that. So we shifted the network earlier today with success. Network overlap, fixed. I did a mistake when planning the site there. Next step and work in progress. Packer builds in this subscription. And I think after that, we should be able to complete until we find another workloads to be run. But I propose that we can start closing it. As a reminder, no spot instance, but we can run, we can try running the bum with VM here. Because we are, I mean, the quota is insane. I was able to increase the virtual machine CPU quota to more than 4,000. So we should be able to spin up more than 500 agents at the same time without reaching the quota. That would be a lot of money, but all pay with the subscription. I've checked the subscription. We have reached the $600 consumed. And for December, we are at 536. Reminder that it costs more in the subscription and that new sponsor than on the previous one because we don't have spot instances. So the credit will be consumed way more, in a way more, we will have to use way more credit. Until we are able to. That's more confident because they will not kill the spot instance. And at that period of the year, that will kill a lot of spot instance. Absolutely. So Damian, this is for Azure. Yes, absolutely. Okay, thank you. So it is that we can't use spot instances with the donated credits. And for now. It's only because right now they don't have enough spot available instances on the region where we use it. Ah, okay. So it's not a general limitation of the donation. It's rather something specific for this region. Oh, thanks. And when we ask it was the week before Thanksgiving, so most probably they won't have until January. Right, the holiday period is certainly high demand. Great, thank you. But yep, good point. It wasn't really clear. Is there any question on this one? Stefan, what's the status on the ghost test that allows us for Packer to test both Windows and Linux? That was not easy, but we did manage to go ahead and now we got a ghost for Windows running quite well and all the data CLI updated to update the correct file. So we are now at the place where we can factorize and have a common one and I mean pinpoint the ghost to deal with either Linux, Windows or both of them. That would be useful for the step that we're doing right now for the all-in one with all the tools. So it's working better and better, but that will have to wait for the new subscription usage. Great work because you were able to find issues that we fixed. So the next release of the all-in-one image should have a fixed and updated version such as YQ that hasn't been updated since two months for instance. Great work. Any question? Okay, next one, Chinese website redirection. So I'm sorry, folks. I, each time I try to say, which time I say, hey, Kevin, I will take time with you. Then something happened and I'm not available to work and to help so I'm really sorry for this one. I hope we should be able to do something but I believe that will be in January. I don't mind trying this week, but honestly. Yeah, so Damien, Kevin and I had a conversation and I felt like we shouldn't take any of your time until both he and I have completed our activities to be prepared for that. This is not urgent enough to justify us taking time over other things, particularly when I in particular have more actions that I need to be taking before we're ready to meet with you. Let's get me to do my part. And then we'll, after January, sometime in January, we'll hopefully meet with you and get this off our list. Okay, thanks for the work. Is that okay if we delay this one to January? Yes. January. And I propose we close the scale with sponsorships since you didn't receive any answer. Stefan, we could. I did send a message saying that I will close the issue as I don't have heard anything from him. Okay. So, sir, let's close and search other sponsorships. I'm pretty sure we think of us as soon as you will learn that those kind of sponsorships are coming back because I was on his back a lot. Thank you. No problem. Thanks for taking care of that. I propose that January 2024 will be either they come back to us and that's really good or we try with OVH or Etzner. Okay, for everyone. That was the last issue. Mark, do you mind? Yeah, yeah. Do you mind on your... Oh, it says, Stefan, go to the L desk and let's see. New ones. If we have new issues or untreated issue. Okay, so we have the backend extension, indexer build is broken and infrastructure. Oh, I missed this one. And for me that is a duplicate of the larger thing that is that the backend extension indexer needs a complete re-implementation or work to find some ugly band aid, some alternate solution until we can get to a re-implementation. Okay, I'm gonna close it as a duplicate then. Great. Thanks, Mark. I don't see other triage or new issues. So people, do you have specific topics to speak on the infrastructure? No, okay. So I propose that we stop the recording and so for people watching us, see you in two weeks. Bye-bye. And for the other, then we can start saying things after the end of the recording. I'm not the one able to close the recording. I think I can stop it.