 Yeah, now it's on. So welcome for this new infrastructure meeting So yeah, the last week has been quite busy So the first topic that we that I would like to cover is automating Jenkins releases So thanks everybody for the help on this last week So basically we did a great achievement. It was a two years long project to switch from KK basement to the Jenkins infrastructure in order to release a Jenkins version So yeah, it was at least it was a bit painful, but I mean, I guess it just mean that we have to do it more often So we had the first three days last Thursday. We did a second one yesterday, which was Which was way better So right now we are still focusing on weekly releases We now have to start preparing the stable than the security one I think for the next three days we have to better work on their communication Especially regarding the G there the key that changed for them. They've been on repositories and for the Red Hat and serious repository But yeah, I don't know if you have anything that you want to bring on this topic right now I don't I don't think it's important to cover all the different issues because we have we had a lot and we fixed a lot thing that we did not plan but Yeah I'm still looking for the red hat public key So just if you can send it to me separately, no need to do it now, Olivier I just need to be sure it's your night on ISE. Oh, you did good. Okay, so I can grab it from there then. Thanks, Tim Does that mean I mean is it not available on the website? That key has been updated then the the old key is gone replaced by the new key Yeah, you can't install the old versions because the old keys go on Yeah, that worries me Okay, so have we broken LTS by that? Yeah, so that I think that's a good point basically what happened here is that we changed we changed the key for the new releases and if we want to use the old keys they are still available but in the different repositories like they be unstable red at stable or whatever and I think it's Important to have a specific backup. So that I mean you raise a good point here Yeah, we need we need to be able to yeah, we need to be aware. We need to be able to provide all keys Yeah, so you'll still be able to get it mark if you get it from the LTS folder But we should copy it to another copy it into the main folder, but just with a different name Great. Yeah, maybe maybe we can use let's say the The GPG key name directly in the fine name so we know which key is I mean for what it's used So, yeah Has joined us Who joined us Regarding regarding the stable and the security Version the main thing that need to be changed is obviously where we are pushing the artifact. So for the stable For a stable we just have to fetch Jenkins from a specific branch What I'm still not not sure yet is Do we just use the same Jenkins instance like really that's the other Jenkins that I own Do we use a different job for stable for security for weekly? Yeah, those need to be those need to be Defined at the moment another option would also to have to define a specific branch for the different release line So we do not provide that option by parameter That's rough to think before switching to the stable and to the security Release line, but for now we still rely on cuz okay for releasing those version That's gonna be painful because of the keys changing Yeah, so especially for the especially for the stable Especially for stable and I'm not sure mark. Do you have an idea when will be the next table? release Yeah, Wednesday On Wednesday, so we will isn't different to 222.2 do out tomorrow. I Thought that we were I'll have to double check Olivier But see I don't understand why that would be an issue because the way it's The the old key is still in the stable directory structure and because the old key is still in the stable directory structure Kosuke's builds for stable will still behave exactly as they should Yes, that's that's so it won't be it won't be an issue for this week Okay, that's what once once we decide to switch to the new release process for a stable release Then we will have to to maybe either modify the Jenkins file I mean we have to bring some specific configuration for them for the stable Okay, but yeah, I think I think that we I'm not sure what would be the best way to communicate I don't know if we have to write a blog post But I mean we have to be better at communicating the fight that we are changing the key for the stable release Yeah, so certainly it will need to go into the when we do the stable release change We'll put it into the upgrade guide for that version of the stable release. We should blog it agreed wholeheartedly We should probably tweet about it Every other channel and then we should also put on our armor for the the inflamed users who submit bug reports having read none of them Okay, I mean the big thing would just be to have to get it up on the package touching inside our website as well So they a lot of users were confused that it wasn't up on that because the website publishing wasn't working, right? Yeah, but yeah, I think Twitter plus the website would be the main avenues plus cover off other ones Okay I Think I don't think you have anything that you have to discuss about the automating Jenkins releases do we Well, so you had mentioned LTS first and then security I was prone to go the other direction But I guess one is not without the other right if we do a security release It has to be both LTS and weekly and so I Think I mean I think the reason why I first mentioned LTS is because it seems easier to me The so but yeah, you you're totally right that we need to first be able to trigger a security release And then switch to the LTS the reason why So the requirements ask by Daniel for the security releases So you want to be able to trigger release and push that release on a specific maven repository And that a specific maven repository change for each specific security issue So that's only the security team and the people who reported the security issue can work on this So this is from requirements by Daniel And so yeah, we need to be able to provide a dynamic way to to specify which maven repository we want to Where we want to push on the release and that's why yeah, I don't think it's rocket science. It's just like The process is not the same But we can't really release a weekly security fix until it's the Until that's done. I assume because we can't really change the key back Yep So yeah So otherwise the next to pee that I just want to briefly mention is that The issue I don't know if you I didn't have the time to Re-spent a lot of time and see other jinkies that are you but I guess we still have the issues with Amazon instances Being disconnected. I don't know if we have any ideas how to solve this now I think we had talked about It's possibly because they are so Disconnected from the master But I think we had a discussion about that on IRC at one point I was talking about moving master to AWS at some point Somebody talking Yes You can hear you just fine. I suspect you just got your family in the background. Oh, sorry. Yeah, we do There are seven eight of us nine of us here. So exactly you just got a family. It's okay So, I don't know if that's something we want to look at here I can Yes, it's just it's just the background noise Alex has a family behind him. Oh That's weird because I can't hear him at all. Oh Keep going Alex So we at some point, I don't know if it's our plan to move completely away from Azure or Or what but I don't know if that would solve the problem moving the master over AWS or not That could that could that could solve it and I think it'd be Either either either either we deploy your communities cluster on Amazon and then we can easily really play see how the jink is that I Oh, so team Jacob already work on a short so we could deploy CI the jink is that I other can communities So that would be I Know other people are seeing similar issues I hang out in the Gitter channel for the EC to plug in and other people are commenting that they have the same issues and I haven't seen or Seen any resolution from anyone on on this issue. So it seems like it's a fairly common problem So I don't know If the EC to plug in people are working on it or not Yeah, so that that's consistent with what Oleg Dinashev had suggested is he'd had similar unreliable Experiences with the EC to plug in independent of the network network distance between the Azure cloud and AWS cloud I Am looking at the ECS stuff I don't know. I don't think that can replace all of our agents just because there are some that require Or it would be Docker and Docker which is has its own pitfalls and stuff like that, but it may be able at least give us The normal plug-in builds a little bit more stable and things like that. So that's something I've been looking into We could possibly do Docker and Docker on a dedicated CI cluster anyway Yeah, I've just seen some pitfalls with Docker and Docker and you can't do that with windows very easily You might be able to on AWS a little better, but on Azure they have issues running Docker and Docker on Windows So that doesn't necessarily give us Windows builds Now sorry about that No, I mean the sound was weird for us as well I'm just wondering if we are not the only one to have that specific issues with ST2 Maybe what would be I mean, maybe we could try to identify how we can improve the plugin I mean, who would be the best person to contact for this? I Can ask in the getter channel and see I mean we can contact the maintainers who are listed in either the repository update their Area for that plug-in or directly on get hub and see if it didn't work with them all but I believe that Hmm. I think Ryan is one of them one of the Jenkins core maintainers. I Tried to check on get hub, but yeah hubs down Oh, okay It's the end of the world as we know it did you just say github is down mm-hmm It happened that would happen two weeks ago as well Okay, so yeah to me to me to me the best work I mean one of the best would be to first identify if we can visit again And then if it's too painful to do that because it's also it's also a sign that we are sending to the community like We don't want to use this to plug-in because it's not really a reliable arm, which is Not being I mean, yeah, if we can find you can fix it So we can if we can collect some logs and put them on a ticket and the EC2 plug-in and See if we can get some advice from them or our areas where we fix it I'll be using the EC2 plug-in for dynamic agents for Jenkins. Yes. Yes. I'm Seattle Jenkins. I ask Since I mean since it's since this is in Kubernetes Could we just maybe think about using the Kubernetes plug-in to launch dynamic agents within the cluster? So I see I've got it poor Yeah, so so basically Seattle Jenkins.io is not running in communities right now It's running on a Burmese and an Azure Fetra machine Team team prepare a shot to deploy Seattle Jenkins.io on Kubernetes So this is something that we could work on If we do this so basically what I would like to do is to deploy that communities cluster on the Amazon account So we could have a second cluster just dedicated Seattle Jenkins.io Just just in case everybody knows that the Jenkins control point doesn't have to reside in the inside Kubernetes We want a different cluster for it anyway because it's the CI Okay, because proper public k8s currently has a whole bunch of secret information on there And we wouldn't really want the CI cluster Okay, let's go near it. I assume No And once we've got that cluster we might as well just put CI Jenkins.io on it so it can be maintained properly Currently, it's just manually updated by Mark and Daniel mostly Okay, I think we cover all the topic regarding Seattle Jenkins.io on Amazon. The next one was regarding archive Jenkins.io.org So several months ago, we had to move that specific machine. So that machine was running on a rack space and Since the rack space is sponsoring us again. I don't think this this is a big priority So what I'm just what I would just do here We just leave that machine there and I will just deploy an additional mirror on communities because I already did most of the work there So we just deploy archives at Azure the Jenkins.io. That's it. And I would just close that ticket Another work that we started Working on was having agent for us through and it's a 390 and ppc64. I think the main blocker is still to To configure those agent and to add those to Seattle Jenkins.io, right? Correct. Yes, correct. They're they're running just fine for me in my test test environment We just have to do the work to get them configured in the automation so that they'll be Maintained correctly in the CI Jenkins.io as agents. Now. I understood from From Alex that we also have arm 64 capability now on CI Jenkins.io Alex is did I understand correctly? Yeah, they're easy to agents so they would have the same issues that we're having with the other easy to agents But yes, we do have arm 64 Agents though Great. So if I want to if I want to broaden the get plant plug-in testing and the get plug-in testing Platform wise that's a that's a good excuse for me to target. Thanks. Thanks very much And it's I think it's labeled with arm 64 as the label and I think that's that's only label on there right now right Next topic is regarding fastly so the now we now have a fastly account It's configured working for plugins of Jenkins.io. We made the configuration for Jenkins So first for plugins of Jenkins.io. It was really simple to put in place We also enable IPv6 for plugins of Jenkins.io. It was quite simple The only thing we had to do was just to specify a specific C name that support IPv4 and IPv6 So yeah, that's the first website that we support on IPv6 Regarding Jenkins.io. We already made the configuration so it's almost ready The only thing is if we also want to have IPv6 for Jenkins.io, we have we cannot use AppExamine So we have to switch to WUWUWU that Jenkins.io. I know that there are some configuration Directly on the hand charts. I'm not sure if they are also redirect directly in the website like in sites Jenkins.io is something that I have to verify but So yeah, unless unless you have any good reason to not switch to WUWUWU to the subdomain This is something that I would like to do in the coming days For now, we will just use the Fastly account for those two websites. So I would like to identify How much we are using in the budget that we can use per month Before before I mean before putting more Website there What's the budget that we got in the end? We have 1000 per month And so This the serious temptation to put a piece of mirrors or of updates is not viable in the thousand dollars I mean I would do something that I would like to do maybe just a piece I would just like to see how much we are spending in one month some plugins of Jenkins.io and Jenkins.io Another another option would also be to pay the extra care credits But if we want to do that we have to reduce our bid on Azure So it's like we have we have a but we have a budget that we can we have a budget that we can spend Under the CDF parts and so right now we still have the target to reduce our Azure Build and so depending on how good we do how how and how do and how good we are there. Yeah, maybe we can spend some extra money on Fastly We are still also in the process to simplify the building of the infrastructure project, so We are starting discussing with Dan Lopez from the CDF to see how we can Directly expense send a invoice to the CDF. So basically what's happening for Azure accounts right now That's KK who is paying the bill and then has to be reimbursed for Fastly I've been my credit card to the to the account. So we still have I mean we still have like multiple accounts where we have our own information and we would like to have everything linked to To the CDF. So it's still a working progress for now. I Just had to put that on hold for the automated release Another yes, I apologize. I've got to drop off. I've got to be in this next call. Thanks very much So just and then I would just skip with the last last point So regarding the sponsoring page do something that need to be done So a lot of things happen in the sponsoring area over the last few months And so do something that I have to update on Jenkins that I use I would just probably put small icons instead of just the name Something that we like to work on this week And I propose to stop the meeting here because yeah, we are already really late Before the next meetup. So except if I mean, do you have anything that you want to? I'm good We could if I could if you've got a minute or two sometime just to fix incrementals. Yeah To be fixed Yeah, I saw I saw but I was too busy with the automated release last week and I didn't want to to switch the context so Yep, I keep that in my head. Thanks for your time. Thank you. Have a good day. Have a good day