 Hi, welcome, everyone, in the Jenkins Infra weekly team meeting. We are today the 30th of January, 2024. With us today, we got Bruno Verraten, Kevin Mortens, and myself, Stefan Merle. Whoops. Here it is. Let's write that. We may have Erwe Le Meur joining us later, but he's in another meeting right now. As for the announcement, the weekly 2.443 is out. And if I'm not mistaking, yes, the docker image also is out. So everything is perfect. Announcement, I'm sorry, I'm going too fast. Do I have something to add? Just want to let you know, Stefan, the change blog is also merged for 2.443. So that should be available as well. Thank you so much. So UCM, I'm going too fast. So do not hesitate to slow me down. For the announcement, we will all meet together soon at the fours them in Brexit. It's not even next week. It's this week. We meet on Friday now. Friday, it's the Jenkins. I forgot the exact name of the Friday meeting. Jenkins, something. On the previous summit. Yes. Thank you Bruno. You almost got it. Yeah, almost. So the next meeting, the next weekly meeting will be cancelled the 6th of February because we will not work a lot this week. We will meet together, which is work, but not coding one. And the next weekly team meeting will be on the 13th of February. Is that correct? I think so. Yes. So next weekly, we'll still be next week, 6th of February, with the version 2.444. And for the next LTS, it's still the 21st of February, with the 2.440.1, sorry. Do we have any security advisory? No, we had one last week, but nothing as for the security advisory. And the next major event for them, next weekend. But the next one will be Collex 13 and 14 of March, 2024. In Los Angeles, I think. I think. So let's check what we've done this week. Oh, that one. Oh, RV is not there yet. Nope. This one is brand new. We had a question from a user to get information about the nodes that we're using for the mirror bit, meaning gagingkins.io. And in fact, we managed to get a JSON representation of the information we have. The output is like this one. Very useful for machine to machine. And here it is. It's available in report.gengkins.io slash infrastructure. It's versionized. So this one is the V1. And you can get all the information on the mirrors used. Oh, it's empty. Good. So RV, we're not done yet. There is a problem somewhere because it's all empty. Shouldn't be empty. Wasn't empty this morning. Yeah, it's empty. Re-opening. I will put the screen capture after that. Sorry, so it's not done, but almost there. And we will have a lot of very useful information in that JSON. We plan to add more and more information publicly available on that area in reports. When creating an account, I mismatch email. Oh, do you have any questions on the JSON report? No. OK. When creating an account, I mismatch email. Closing. No answer from request, though. So that was closed by Damien yesterday. OK. Oh, this one was a huge one. We had a big port exhaustion problem on the network on AKS cluster. We managed to cancel all the SNAT problems. And as you can see, at the end, there is no more SNAT on the network. So how we did that first was reducing the time of the port staying open and then adding more IPs to deal the outgoing network. So now everything is spread. And we are under any limit of SNAT connection count. So it's a very nice job. Thank you, Damien. Oh, this one is the file storage. We had information that in order to lower the price that we were paying for file storage, we could simply use a premium storage class. We were planning to move everything on SSD drive and change all the configuration. And in fact, just changing the kind of storage was able to lower the price because all the operation, the writing operation, are included in the price. The main problem was that it needed a migration because we cannot upgrade storage to the premium one. We had to move everything and migrate by hand all the data, which we did last week. And as you can see on that graphic, the number of operation dropped completely and the transaction now on the premium storage. That one was the older storage. And on the premium storage, we see that we're using it. But it's included in the price. So the price should stay in those blue area, meaning that we will save like, if I remember correctly, it's more than $1,000 per month. $1,000 per month, maybe even more. I forgot the price exactly how much we will save, but we will save some money. I don't think it's that much, but at least we'll save, which is a very good saving. Thank you again, Daniel. And if I remember correctly, the tip for the premium use is coming from Tim, I think. Not quite sure. Not quite sure. But thank you all. Then, oh yeah, we had an issue with the version of Docker. That we did upgrade it. And the new version of Docker is now in all our agents. So solving incompatibility problems for CI. Again, a huge one. We did migrate CI Jenkins.io from the normal subscription paid by CloudBiz to no pay CDF, sorry, paid by the CDF to the Sponsored subscription, paid directly by sponsoring from Azure. And that was a very nice migration handled mostly by Damia. And it went rather well. So as you can see, the cost on the CI Jenkins.io is now at zero in the normal subscription. But in fact, it's been moved within the Sponsored one. And Damia made some calculation. And we should be around those objectives that we had to reduce the price in the current subscription. Do you have any questions? I'm sorry, I forget to ask you all the time if you have questions. Do not hesitate to cut me. Thank you. So oh, this one is an easy one. We did agree to the last LTS, old version using the LTS. Let's see what we still have open. So this one, we spoke about that, and it's back to open. Here is an issue trying to migrate BlobExphere to azcopy. I don't know. I thought that Hervé would be here to speak about that. But he's not. What have you done? Nope, there is nothing more than last week. So probably hasn't had time to work on that. He was dealing with the service principle to have something very secure. I think it's still on the road. This one, it's a big one. It's very sensitive. The update Jenkins.io is used a lot. So we are waiting for an overview and a check from the security team to make sure that everything is under control. It's just migrating the weights, working, and spreading the load between multiple nodes with mirror bits. Yes, this one is a problem on the database. There is a wrong line that's breaking the database. Damian is trying to find lines. He found some by dichotomy, and he did update some. And if I remember correctly, still errors. Concession research of Oracle, we would have to wait after first time. So it's undergoing. That one is funny. So work in progress. Expecting delay. Oh, that one was digital ocean related. I think that's for now. We've been unplugging the ACP on digital ocean until we understand a little bit more what's happening. So no more digital ocean in the building of those builds. Work in progress, too. This one is a funny one. This one is only for Kubernetes with node pool using spot instances as node and using the autoscaler and starting the autoscaler at zero, which is the configuration we were using for IRM and Intel for two node pools, infrasci and IRM small node pool. Sometimes the only node got killed because it's a spot instances. And then the autoscaler is not able to spawn a new node even if we try to spawn agent on it. If I manually put one as a minimal node in the node pool autoscaler, it then trigger correctly and grow up to the maximum. I did open a ticket and issue with Microsoft and I had a visual with them. They told me that we should be studying the scalar at one, which means that we have to pay at least one node all the time. And it's not really great. But then we looked into the prices and using spot instances for the full month is kind of cheap. It's less than $15 per month. So I did upgrade to one for the node pool in IRM 64. So we shouldn't see any more of the problem. But we did so it on the other one, the Intel 86.1. I let it to zero for now because it's really not usual. I think it happened twice in six or seven months. And putting one as a minimal for that other node pool means to spend something like $60 per month, I think, if I'm not mistaking something like that. So for now, it's only the IRM 64 on that pool. So we have to keep in mind that if agents are not spawning on the infra CI and not IRM, we need to make sure that we put that minimum to one. And not only I haven't done that because of the price, but also because we are moving most of the building on IRM. So we shouldn't use much of the Intel anymore. And that may lead us to find something we have forgotten to move on IRM. So that's a little help to see if we forget something. You have any question on that? OK, then infra CI IRM 64. This one now, I think it's the last image that we need to move on IRM 64 and the all in one to remind you, the aim is to move all the Docker image that we're using to the all in one that we built with Packer. Like that, we have only one area where we need to upgrade and keep everything up to date. And it's the same content used for VM and Docker images. So it's very useful for us. That last image name is Builder. And the only problem with this one, it's used with infra CI and CI Jenkins IO. So for now, I'm working on the builds on infra CI. I bumped into a problem with Bundle and NPM that found. I discovered that the ghost test that we were using on the Packer image was giving us false negative. And I found that ghost was not really able to handle two level of tests with a hand and a not. So I opened an issue within ghost. And I changed some path in the Packer image. I'm waiting for the new tag to be spread everywhere to check if I'm right or not. But it's going further. And that migration on AM64 bring us to save some cash. That's the same issue that we had for the migration on CI. And we see that all our effort are bringing us some saving every month. We are correctly working on the saving money on Azure because of that. The migration to IRM is saving money, not much but some, is providing us both compatibility with Intel and IRM, ARM, and also it helps us to clean up everything and to update all the image we're using. So it's a really very interesting job. Last but not least, yes, we updated all those storage accounts. And now we have to keep an eye on them and to make sure that we follow their growth and everything. So AirVay opened two pull requests to make sure that we're tracking correctly the usage and the full fineness of those storages. Almost there. I need to review these pull requests after that meeting. So here it is for that milestone. We already have the next milestone, not for next week because we will not have that meeting. But the week after and Damia already moved around the issue that we still need to work on. Those migration needed us to migrate the data. We did that for a few stuff already. But LDAP, for example, will be a big one. And it was started because of this net exhaustion problem. But it's on its way. This one I forgot Java 21 memory error. I don't have any new event on that one. So we'll see. Or this one we will see during the fourth day because we see all of us it's to revoke a certification in the OpenVPN image. So we will probably be able to handle that all together. This one we should do that after first day is to clean up somewhere we're calling the data. But we really need to make sure that everything works fine and to be all ready to handle any problem that may occur if we merge that. But it looks like it's easy. But we really need to be careful. This one I think it will have to wait. It's a way to provide a versionized Jenkins IO for docs. Docs Jenkins IO. No progress. So yeah, that was not a priority. We'll see after first then. In this one I have no clue for God. This one we are. I don't know. Do you remember this one? Below two in this page. Then I'll take him around. To slow download. So that may come from the nodes, the mirrors. So work in progress. I'm sorry I forgot this one. OK, so those are the ones already planned for the 13th of February. I'm sorry, Evans followed the usual notes. Do you have any question? I was too fast. I'm sorry. I hope it's still understandable. Not sure. So if it's OK for you, I'm done. I will move all the milestone this one to the 13th of February. And for everyone online, I will see you in two weeks for the next weekly intra-team meeting. Bye-bye, guys.