 I have everybody welcome for this new Jenkins infrastructure meeting. So the first topic that I want to cover is this weekend is the first time which is major open source event for the Jenkins community will be there. And I'm planning to do some demo if people are interested. Basically what I really want to highlight here is don't break things before this weekend because I don't want to have to deal with infrastructure issues over the weekend. So if you if you if you're afraid to change something just hold on until Monday and then we'll have plenty time to fix every issues related to infra so don't don't don't change major stuff. And if you want to demo something or learn something feel free to ask and I can prepare that for these weekends. We also have. I just invite you to look at the first them websites for any content related to that. And one last thing. I think it's worthwhile to highlight as well. The first them as a specific chat room. So everybody can just join create an account and talk. So if you don't know what you'll do on Saturday and Sunday, feel free to just install that application and your smartphone and participate with us. If you are available. At least to answer questions. So that was the first thing that I want to cover any question. Then let's continue. The next topic is about root less genealogy agents so something that we would not say discover but that we've that we decided that we want to change last. We are building specific genealogy agent occur images agents for that we use and see I don't think is that I also those agents include no Python Ruby. I mean those are pretty generic images. They are usually running as roots and so basically what we want is we want to have images running with Jenkins user using the UID 1000. And so Kara in Damian are working on that. So that's something that they realized during the process that we were shipping a genealogy agent alpine the current image which is which does nothing I mean just redundant with the default genealogy, the default inbound agents. So the plan is to deprecate that specific image. You can communicate that on the mailing list but basically, if you are using it. That's the right time to stop using it and to use inbound agents with, you know, would probably delete that image in the coming weeks. If you are relying on those genealogy agents. We also invite you to double check that you if you need the root user or not. I mean, the root user is still there. It just by default, it won't be the default user. So that's it. Any question anything to add. The next topic is about Jenkins infrastructure. So what I want to highlight here so we made we made a few refactoring to the one of the tool that we use on that git repository which is updates you like. And basically, it broke the pipeline since Friday so we have many ongoing change that need to happen. And we were waiting for the release to happen today, which is done now so we'll probably try to enable the job again, but yeah, there are many PR so feel free to review them and then otherwise will apply them in the coming days. And again, we'll try to pay attention to not introduce major changes. So if you did take something that will be a major change, feel free to put the label on holds with a few with a comment basically saying what's why you suspect something dangerous there. It's always useful. I think all is said. The next topic which is so Damian started looking at deploying a Kubernetes cluster on our Amazon account so the idea would be to use that communities cluster just for Jenkins agents from Seattle Jenkins that I use so Damian wrote the Terraform code to deploy that you built a specific Docker image containing the Terraform version that we need needs. So there is already a git repository. If I remember correctly, it's a Jenkins infrastructure slash AWS like the same pattern that Jenkins infrastructure slash Azure. So feel free to make any comments there. The plan here is just to use it to replace the Azure container instances. So we will just provision, we will just configure specific agents on Seattle Jenkins that we just provision puts on that cluster. So more testing in the coming week. And the last topic that I put I mean for me to the agenda was a few, few things regarding sovereign so we had issues. We had issues with sovereign so sovereign is one of the mirror. And that we are using in the infrastructure since many years now. They had hardware issues during the months of December. They put that mirror on maintenance. They solved hardware issue one week ago. And we had a second thing with that specific mirror, which is, they were using an IP from the United States, even though the cluster the server was running in Netherlands. So we asked the mirror maintainer to use a different IP. So, so mirror beats could detect the right location for the server, but apparently took more than a day to update the Dennis records. So for example, a disabled server, you know, I documented the procedure to disable that mirror again in the future. So normally someone else would be able to disable it, it's more quickly. We still have to put in place monitoring for every mirrors. So if someone is interested to contribute to that, I can show how it's not not really complicated. And so yeah, so normally everything should be back now I have to double check the states of the Dennis records. And if it's working correctly, we'll probably just put it back to the pool. So, so we had some very high visibility failures due to the server on server. I'm really worried about putting it back online. I'm my trust level in surveillance ability to manage that thing has decreased. It's a terrible thing to say, but, but I'm worried about them. So, is it is it online now, Olivier, or is it as I mean, last time I checked that was the server was online. So I, they don't have, they don't have any. For me it was working correctly. So when I did the test last Friday it was working as expected. What's what Garrett highlighted over the weekend was the Dennis record that points. So the Dennis record was flapping between the old IPs and the new IP. So sometimes you were target, you were going to the new IP sometimes to the old IP which obviously can generate some connection issues. So my, the way I would proceed. I mean, there is no reason now that there is no reason that it's done again so I would put that mirror to me to monitoring so we could identify the response time. And so if we detect any high response time are issues which we can just easily disable it just one comment run. We have people on different time zones. So I think it's just a matter of, can we identify mirror issues soon enough so we can disable it. I think it was the TTL on the zone record was set too high. So it's just not propagating fast enough says some not all, not all DNS sort of relays correctly there to the TTL anyway. Yeah, the way I checked I think the TTL was one hour, but again, not every DNS, the DNS server under TTL correctly and from my experience, each time you change the DNS records, it's always, it always lead to a house and especially considering the amount of checking instance relying on that mirror. So, I mean, the, the incorrect geo IP already made it complicated because it meant major Jenkins users on the eastern coast of the United States were going to that location. And we had one very, very high visibility person, Martin Dianju, who, who he's based in in French Canada, you know, so Quebec area, and he had major failures because Sarbarian was unreliable and, and that's a free that's an awkward thing to have somebody that's that if I get contributor broken by infrastructure that it's fairly safe to just disable their server as a mirror for the short term. So just just just so that you know I have to double check because right now I'm running some tennis queries and it's the flapping. It is okay so there's a risk that it's because if it's flap and I those DNS queries. I don't think we're monitoring those right that's a facet of operations that I've never monitored before is is is are we getting DNS flapping the thing the tool that Gareth highlighted is something I've never seen before it looks like a brilliant tool. But what surprised me is that the TTL is one hour. Yeah, so I have to send an email to that mirror maintainer again. Yes, I also have to be sure that the TTL is honored by all the client DNS and relay in your stack. In particular, if we have alpine image in the, in the equation, this could be troublesome inside Kubernetes cluster. So here I will vote for removing the mirror for now. And then, if we have to put it back just to be sure that we first implement a way of putting it outside of the load balancer or whatever we use. I don't know or at least have a way to measure if it's okay. So monitor the DNS request and also the IP which is answered every hours. And if we have a mechanism to alert us or remove at least automatically the server from the pool. We won't. I mean, we could implement that, but we don't have the time to work on that. So. Okay, so because I agree with Mark that the risk areas is quite high. So better to. The resistive there. The resistive there. I'm testing with the with the Google DNS and still flapping. So I have to send an email to see what's going on there. Maybe so, but if it's flapping, doesn't that lobby that we should disable it and it's already it's already disabled anyway. Okay. All right. So it's, you're just not going to enable it. Right. That's perfect because that says, just like your argument was earlier that we should. Things that we worry about, we should not implement before. This is one for me. We should not enable surveillance mirror until after foster when we can look at it more depth. Yeah, I'm double checking that now and it's definitely disabled for now. Perfect. That covered my worry then so long as so long as we're not not relying on that server between now and foster I'm happy. That's fine. And also want something I don't know if you missed that information you don't need a cuban it is access to disable mirror beats. So you need a mirror beats client with the right configuration. So if you can manage to have. So basically the configuration is stored in a ready status base. So either you access a ready status base and you manipulate the release database which sounds like awful. But if you have the configuration file you can just run mirror beats command pointing to the right configuration, and then you can provide comments like least mirrors edit mirrors and blah, blah, blah, and because of the outage we had back in November package that origin the jinkies that you also have also has to get the configuration in the slash a pity directory. So, that that could be an option last resort. We are, yeah, we are still on time. Next topic, which is a record for startup by Mark. Yes, so I, I was being asked by Oracle. Hey, have you registered for Oracle for startups and so I just went ahead and registered with my credit card. So Olivia you and I had talked about, could we please register with another card. I haven't done that. I also haven't done any experiments yet with Oracle for startups to see what it would mean. After FOS them. My first thought of an experiment is to use their Phoenix data center as a mirror, but that may be a too dangerous a thing that would give us a mirror that's closer to the US West Coast, and, and give us a chance to experiment with infrastructure. I'm open to others though we could also try different things it's right now the thought is, they're willing to give us a 75% discount on infrastructure and I'm interested to see if it could help us or not. My proposal is to wait, maybe we can plan a session together, and we just deploy mirrors that that would be a simple service to deploy on that to not rely on specific cloud vendor and because there is already a ham shot for that. Excellent. Well, and that that sounds really good because I've understood that Oracle's bandwidth charges are less than other vendors. And so that that makes it a little more attractive. If they're bandwidth charges are lower, that could be a win. And if you want if you Maybe not in fact I was suggesting maybe we can also deploy the same region than the Azure cluster. But the problem is because we have a mirror running on the communities cluster we have Azure that mirror that Azure that Jenkins that you but it would be difficult to identify the traffic specific to the Azure that mirror. This is all the other services running on that cluster. But yeah, anyway, I propose that we plan a session next week. But once, once you start once you create the accounts. We should use it because usually they don't sponsor for years basically. Right. Yeah. And that's all I had on that topic. So, so basically the way sponsoring work is when you create an account that usually say they offer you sponsoring for one year to your three years. And the problem is in this case two years the problem is if you create the accounts and don't work on that for six months. The day you work on the account you just lose six months of sponsoring. So that's why I'm saying if you enable it should start using it. And then, and the last topic to the agenda which is the Jenkins release the latest Jenkins weekly release is done. So the good thing is get ready or change did not break the release process. I was, I was sincerely worried I thought oh my last night was code reading. Oh but what if this. Nope, it worked flawlessly. At least I'll check list but well done Garrett. I saw you were very brave to merge it. I'm monitoring the channel all day just just in case. So, so just for the people who don't know what we are talking here. So I did the first version. I did a first Python script to detect what's the latest version. It was working for weekly releases, but it was not working for stable releases that's the first thing. And that is that's Python scripts need to be used from different location. And because it was a Python script, it put a strong requirement and having a Python environment. And so basically what Garrett did was to rewrite the script that I wrote initially in calling so it's smaller. It's a small binary that we can use. So we just download the binary and then we can use it. And also it supports the stable release version, which means that for stable release we could simplify the number by one of steps that we have to do for a new release. So, yeah, that's that's a nice improvements. And obviously you can use it outside the Jenkins project because it just just a way to know the latest version so I think it every person monitoring Jenkins version could use it. Yep. And the other topic that you want to bring here before we close the meeting. I mean, it does not make sense to take 30 minutes if we cover what we have to cover in 20. Thanks for your time and see you later. Bye bye.