 Thanks, everyone. Welcome to the Jenkins infrastructure meeting. It's the 25th of August Let's take a look at the agenda so propose a topic to talk schedules briefly Then Tim talked to mirrors upgrade Update center improvement. I'll give a brief status report on JIRA upgrade We have a sort of a hot topic on docker terms of service that needs further discussion Status report on Oracle cloud Release status reports, and I think that was it any other topics you want to add if there's time perhaps The docker images and they use off version specific update sites Good Daniel. I miss that you're here. Okay docker Docker so where how about we put that before oracle cloud and after docker terms of services So docker image use Versions specific update sites Yes If you access updates of Jenkins.io, you get The plugins that are compatible with your versions of Jenkins and the mechanisms the docker images currently are using Is one I would like to retire And it basically just needs a pull request merged as far as I can tell Okay, got it. All right So topic is added. We'll let you let you talk to it. That's great. Thank you Any other topics we need to add to the agenda? Okay, then let's talk mirrors upgrade Tim and Daniel Yeah so basically we spot The mirroring service from mirror brain to mirror birds So then so the main reason that Olivia hadn't done this previously was He wasn't able to keep the Plug-in releases and sync because if you try and sync the whole folder it takes About 20 to 30 minutes But so Daniel built some functionality into updates into which emits only plugins that have been up released in the last three hours So we're able to do a delta sync which is Normally between one and ten plugins and it takes something like 10 seconds Rather than 20 to 30 minutes And so with that we were able to change it over There's been a few hiccups and So we have issues with There was a HTTP HTTPS issue which Daniel fixed in the HT access files, I think Well, the biggest issue initially was that nothing was getting married Everything was going directly through to storage. So you see some the bandwidth just went through the roof on Friday And we eventually tracked that down to there was a rejects on what should get mirrored and HPI wasn't in there And that was in the chart and fixed that eventually And after we did that then we noticed that mirroring was slow for the initial release and The archives download site is performance is just so bad So that archives was a fullback which had everything mirrored on it But it's just performance on it's absolutely rubbish So we've And it was kind of okay because it was only used for old releases so basically archives users are fullback when something's not marriage and I think we mirror something like the last year of releases So there are some so some older Jenkins versions aren't married and I guess the conclusion was people who are updating often can deal with slow speeds the problem became though that's When plugins are initially released Mirrored mirrored bits takes a while to kick in and so they were getting really poor speeds. So I've ended up at least for the short term putting in a Change in the fullback service to be as your which has impacts on our bandwidth But it's only for well for plugins It's only for the first just over an hour that takes it takes between an hour to an hour and a half for Mirrored bits to notice that it's on the mirrors I'm just gonna try and look at improving that if I can But it also does mean that we are serving traffic for the old Jenkins versions as well I think that's where most of the bandwidth is going at the moment It's it's not huge. So I think we just have to monitor it. So we've made a couple of fixes We noticed that Latest plug-in. So there's a URL Something like slash download slash plugins slash plug-in slash latest Which was getting redirected to get dot jings.io directly and the sim link updates not getting updated And so Daniel fixed that and the update send it to redirect to a specific version instead and Was and then we noticed the next day that there's the same issue for wars And so Daniel fixed that as well. Thanks for your help and updating to hear Daniel and So that should stop all of that traffic So we're keeping an eye on some I think yesterday afternoon ish Was the last fixed week. Well, you say evening European time. I think was the last fix that went on And I was having a look at the logs this morning And there are only two plugins showing up in the logs and they were both plugins who are released within the last hour And the rest of it was all Jenkins war files Now are the Jenkins war downloads ultimately serve from mirrors as well or from Jenkins wars older than a year. Oh Okay, older than so outdated Jenkins wars. I see. Thanks. Yeah Okay Yeah, not necessarily older than a year. It's whatever. I think it's a year, but I'm not sure The other thing was so just look so it seems like Mirabit says two stage On on how it gets rolled out. So Mirabits does a refresh of the local repository And at that point it will hash the file if it doesn't know about it And then it also periodically scans the mirrors doing a full r-sync of against the whole mirror Which is when it notices whether mirrors have it or not So basically the issues it seems that only does full scan something doesn't appear to be a way to do a partial scan So it probably needs a improvement in Mirabits, but then that could get our Time to mirror two seconds instead of an hour and a half And Also part of this I think gone through I think his name is he runs the EU mirror He's increased his sync time from an hour to 15 minutes. I think So basically for his mirror Plugins are mirrored pretty much straight away And we also we're also pushing releases to the OSU mirror as well So that so plugins are available instantly on the OSU mirror for Oregon State University By the issue is that Mirabits just doesn't notice and doesn't start redirecting there for a while So I did ideally we wouldn't have to serve these plugins, but it's also I'm looking at today's traffic. It's not It's not a huge amount of bandwidth, but if something like the AWS SDK gets released that could be a fair bit of bandwidth Okay, so and the place where you're going to gather the traffic patterns from the logs Is that something you can share with us? Is it available to somebody like me without? Yeah, that's in Grafana. I think I shared the do I share the log Chris? Yeah, so At 8 o'clock a.m. My time zone If you say if you look in IRC for you can check in the low key the URLs there Really and then you may even need to be added to some groups. So just have a look If you're not in the group I can add you Should that's one thing is um did you mark or only get access to? Key cloak, I know that was something we talked about before Olivia went on leave. I Think so, but it's a good question. I don't have the answer immediately. Let me put it as a reminder I would check because I'm key cloak because we're going Yeah Next week there might not be anyone around who has access to accounts Right, and if there's anything else that I know I'm the backup on secrets for Olivia But I think we're probably okay. Well, so I'm the global backup. There is other backups for some specific services But I don't think that will be an issue for a week But if there's anything else that you do need while I'm off have a think Right. Yeah, and I'll I'll try to do some safety checks. Thanks. Okay. Yeah Yeah, just make sure you can log in and if you need it if you need any Hence on how to do anything I can show you Yes, thank you very much. I Was going to veto your vacation just so we were clear Kim, but I Enjoy your time off. That's great anything else on the mirrors So yeah, it's just two things so this fixes two major issues with this that have been around for years It's been for 160 and then for 361 I think So Infra 160 is that update site reports plugins being available before they're actually available and Users can't download the plug-in and especially in the Jenkins you why they can't even I can't even install the previous plug-in because There's no way to install a specific version. So you just get a 404 back and uses that use latest or Pull it in a parent with a fixed version, but not the child Get stuck with the dependency for a foreign so that fixes that issue and I think it's 361 Maybe it may have the numbers wrong, but 361 is about plug-in updates being done over HTTPS So now now plug-in updates are into end over HTTPS Excellent. Thank you. I cannot express the delight How how grateful that I am for that? That's wonderful. Thank you to you and Daniel. Thank you. Thank you Now you're no, you're afraid of releasing get plug-in and annoying hundreds of people for the next hour and a half Which today I'm scheduled to release get plug-in and annoy people for a long time You're right is the Google summer of code get plug-in delivery is coming And so there's a potential we're gonna annoy a bunch of people with some really cool stuff Yes This actually So one other fix that we get out of this as we get an updated geo geo IP database So mirror brain geo IP database Was broken and it hadn't been updated in a long time. It was very inaccurate So poor users in the EU always got sent to the server ion Mirror at least my geo IP data and many others It's going to the server ion mirror, which was falling over all the time and that was what got me to fix it in the end so Gunther runs a really good mirror and And so a lot of a you traffic is now going towards His mirror and he said and he posted a screenshot and I see showing his traffic Climbing up a lot And he's okay with that I assume right he's not saying oh wow I didn't expect to actually be a mirror at this level. No, I seem happy with that He's seeing he was seeing seeing a lot more traffic now So all the poor users hitting server ion all the time are not it's still it's still a mirror, but it's It's correctly now a mirror that's that's it's in the UAE. Is that right? So it's it's a great year for Africa or four portions of India But it's not such great mirror for someone in Norway or particularly in the UK I mean your geographically a long ways from the UAE so yeah And so I did some I did Timings just to show the improvement as well. So before we made any changes. I did a build with my work Instance which has 131 plugins and it took just over two minutes to build. I did it after our changes And what was the running against the EU mirror and it was like 51 seconds to build So it's over twice as fast. Oh Wonderful Just probably about the mirror I was heading but That's that's that's fine, right? You you saw real performance. That's great And I think that's everything unless Daniel has anything to add Okay, next topic then update center improvements Sorry, yeah, I mean they these two are linked so we've already talked about the updates and their improvements There was the selected list of plugins that were released in the last few hours that now makes the selective sync possible and we changed a bunch of redirect tools and URLs around so now It advertises HTTPS URLs and no longer links or redirects to URLs that Don't really work on mirrors because they are not getting a sync So this is really really tied together a quick note on the transition I saw some confusion I think from jesse with a Test result from Friday evening There was a window of about maybe 10 20 30 minutes Where it looked like Downloads were failing due to checksum errors That was part of the transition where we messed up and quickly fixed it But some tests were failing on Friday evening UTC with Check some mismatches. So that's perhaps useful to know for anyone who's seen those Yes, we're also failing because they because we changed from HTTP to HTTPS There was a test in core that tested that right That was a mighted to have fail. That was a oh, yes, that's exactly the right kind of failure. We're better now We'll fix the fail though. Yeah, okay. That's fair Great anything else Daniel? Oh, but it's that's it Okay, so that item on key cloak. I've got it as an action item. Let me just put it I'll note it later as an action item JIRA upgrade plan. I apologize I've made no progress since our last meeting the notes are in the plan and I will be working that I've got a schedule the next session with the Linux foundation to Review that here the actions I've taken we're ready for the first test and they will then plan the work They'll set up a JIRA instance using Data for our backup and we'll be able to test that some of the crucial things there are SSL related How do we delegate to them the permission to create an SSL? Name for something we use for tests Okay, Daniel. Oh, no next one change in Docker terms of service So Docker has restated that pull limits Will be real as far as I understand it is that everyone else's comprehension as well that If you don't have a have a pro or a team account, you will be receiving pull limits Yeah, but it's It's it's not it's at the user end. It's not on our end. Oh So this is per this is the person doing the poll. It's not for the repository. Yeah Yeah, so they put out clarification like today or yesterday We are users of Docker hub. I mean To build The images we need to pull the upstream image All of our Jenkins instances pull from our own Jenkins images or potentially Jenkins Infra organization There's a lot more going on than just us publishing images and not caring who uses them sorry, yeah Hey, so so that probably lobbies that we need to to look for funding to get a pro or a team account It's just a pine It's not about the account it's about having to put it everywhere Right, right exactly. It's it's and then we have to we have to use we have to docker login everywhere, right? We have to somehow inform them that yes, we're using it. How do we how do our? Agents work when we use Azure ACS what's its name? Yeah, as your container instances, right? my understanding is that so Azure is doing a poll. I don't know how they delegate they're good. So Basically the Microsoft account would be doing the polling in that case is my understanding Yeah So I get the impression that for our own polls We could just Not care and see how badly things break because nothing is really Relying on that that much Well the plugins like the docker workflow plug-in a lot of its tests do polls some other plugins as well in their tests they Create docker images and and poll images and things like that so the other question is like if a an image has you know Ten steps in the docker file is that ten poles? Is that counted as ten poles when it pulls in that information or what is how is the counting done? Do we know I? Think from one pole. Yeah Unless you have it cached even right, so that's the thing This is just some nightmare On the other hand even even even this isn't catastrophic, right? I mean if a few plugins see I just breaks It's not like our site is going down as a result Yeah, I think I think it's gonna be hard to Understand all of the issues we're going to run into before we actually run into them Okay, so I propose we leave the topic on the agenda and annoy each other every week with it so that we talk and Share what we currently know. I don't see any immediate action that we should take right now Anyone have specific actions? I should be flight listing here I think one thing we may want to check is with Azure and possibly AWS. We don't have docker On AWS right now, but it's something we've thought about with ECS. But maybe if they have any information how it will affect their Different technologies, whether it be ACI or ECS I don't know if we have any contacts within those organizations. Okay, there. You know, while their recommendation is probably going to be to run a local container registry and import what you need. Yeah So about that We do have Artefactory and it has docker support. I've never used it, but That may be something where we could offload some stuff too. Do we have any limits there? Oh, sorry. Go ahead say even if it's just for internal use You can just say sync them have a script. Yep. Have a script that just has everything we care about and sync it But it's a bit of a pain Do we have any limits? I know we're on a like a sponsored account or something like that Do we have any limit on the size that we can store there I Don't know if we are beyond the terabyte of data so far and We have not heard anything from JFrog that this would be a problem It might come up in the future, but I don't think there are real official limits At least I haven't seen anything the problem that we have is that our instance also predates any official Open-source sponsorship program they had it was basically KK doing a deal with Yarov and or the the founders of JFrog like eight years ago or something Yeah, they start with a plan we exist Okay, so is this one where we need to we need it? Well, so is there an action? I'm there We should talk to Artefactory talk to JFrog I doubt they'd know we exist without a lot of pushing Okay But yeah, that could definitely be an option would be Using Artefactory at least for internal use. I don't think there's anything to do right now I don't I also don't know if it needs to talk about every week either. Oh, okay Good. All right. So I I'm just not yet comfortable that I see what's going to happen That's cool. We'll we'll leave the topic that should go to the next one except when we need to Okay Docker image use of version specific update site is Daniel But so a few weeks ago. Well months by now. I announced Plans and then implemented them to have final grained Version tiers and update sites. So previously If you had looked at updates Jenkins IO, there was one tier per LTS baseline for the last five baselines or so and now What happens is the updates in the generator? looks at the core baselines declared by all releases of all plugins and Grabs those released in the last year and makes them into tiers and So I call them the dynamic tiers because their existence depends on the actually distributed plugins So right now in the Docker image the old pattern with the naming scheme stable dash LTS baseline is hard-coded and I request that this be changed so that The Docker image queries the basically the root URL for the updates and the JSON to get redirected to the appropriate compatible tier and then Continues from there rather than just assuming there will be this magic directory name And the way I propose we implement this in the update center is that The existing tiers fixed tiers will still be generated But any new ones would not So I propose this we started with Jenkins 2 to 49.x the upcoming baseline That's due to be scheduled to be released in two weeks And at that time there will be Not five, but just four fixed tiers And over time For each future LTS baseline that exists the oldest one will be removed until there are no fixed tiers left And that's that's my proposal how this can be implemented. There is a pull request that I linked For the Docker image that I think accomplishes everything we need here But it is not yet merged And I think it's open for like the last three or four months So I think now with the next LTS baseline upcoming is a good time To talk about how to proceed here So any Downsides I don't see any anything. We're sacrificing by by switching this technique Old containers will no longer be able to install plugins with the install plugin script On the other hand nobody support well old containers that would be obsolete any way through due to the The fixed tiers being dropped rights. That's the with with the way I propose this works is I think to 2.190 is the last the oldest supported fixed tier for the next three months and That image will continue to work But to 176 will no longer work, but it would not work anyway. I don't think there are any downsides It might involve an additional Redirect rather than guessing on the side of the Docker container, but it's basically free That's fine to me Yeah, that seems reasonable to me And then after the three month period the 2.190 is working it would fall off the edge and stop working Is that right? That would happen anyway So the way it used it currently works is we support the latest five LTS baseline So right now that's 235 222 204 190 176 once 249 is supported. We drop 176 once 263 Gets released or whatever we drop support for 190 and so on so it's always the latest five and my proposal is we changed Docker containers to no no longer use these and then stop creating the new fixed tiers and Retire the old ones on the existing schedule and the year from now. There will be none left I don't see a downside, but I also don't see the pull request that Alex proposed merged. So Maybe I'm missing something Okay, and that's this this 964 pull request Yeah Okay, great. That may just be that I need to be more diligent doing code use Any any other discussion there on Docker image use Alex any objection from you on the proposal? No, it sounds fine to me and Sounds great to me. Thank you. Okay good Anything else on specific update sites Daniel nope that was it. Thank you. All right Oracle cloud Conversation nothing to report. Oh Sorry Oracle cloud conversation nothing to report. I have the action. I'm still to reach out to them See if they're willing to provide infrastructure for us and at what level I'll continue that release status reports We'd release 2.254 today It went without any bump or bruise 2.249.1 release candidate is coming needs testing and I've proposed an online meetup to highlight Features Tim the idea was to do it like September 10 or 11 and include tables to divs as part of that Would you be willing to be part of that or is that? I'm not really working on tables to do this right now Okay, I would probably leave it out for now you would okay So then that may justify that it's not and not a big enough session to do an online meetup then Because if I remember rightly other UI improvements are not so dramatic as table to dids is Yeah, it's probably not a huge amount in this one. I think Markering what else changed? I mean, it doesn't be just UI improvements, right? There can be other stuff I just don't know off the top of my head and and I will I still need to do that with those support changes substantial, but But that one needs more investigation before I propose a meetup with it. Yeah, there's theming possibly I was I was gonna say the theming stuff. That should be the first LTS of theming sport, right? Yeah Oh, right. Okay. So this this is the first LTS to support dark mode for instance and solar eyes. I Don't recognize solar eyes. What's that and dark mode? On the call Solar rise there's a fairly well known theme for a ton of different text editors And I created the theme plug-in for Jenkins It's a bit weird because it only supports like Four different colors Plus highlight colors. It's a bit difficult to make the Jenkins to make the Jenkins UI to restrict it that much But I think it looks fairly okay Okay, so really might Might justify being part of an online meetup and because dark mode is certainly very very much interesting to people So theming. Okay. Good. Thanks All right. So I'll refine that any other topics we need to address if not I'm gonna call us an end. Thanks everybody Thank you Thanks