 This is the Jenkins infrastructure meeting for the 8th of September. Let me share my screen and we can look at the agenda So topics incrementals on Kubernetes Docker terms of service JIRA upgrade Oracle cloud and release status reports Tim anything else you want to add Is there anything on mirrors? Oh Oh Good question. I don't actually have anything other than they seem to be running well and successfully, but let's Double-check that mirror status Well and and the old mirror solution as well mirror brain Yeah, so mirror bits Just the new solution and mirror brain Yeah, so let's put that on the list Excellent. Okay. Actually, I think for benefit it may be best. Let's put it right up here. Great Yeah, maybe put on there to bring in these upgrades, okay Upgrade and this is upgrade of the major of the version of Kubernetes itself Okay, any any other topics All right So you want to take the floor first with incrementals now running on Kubernetes Thanks very much for making that happen. Tell us more about it Cool. So most of the work having while I was off work between Gavin and Alex mostly I think Gavin converted the function to a web app credit a home chart for it and credit DNS PR and Markey and Alex they've got that the DNS set up So when I got back the main little two issues left which I think people weren't sure I didn't know how to do So one of the issues well, there were two issues left plus other things I had along the way So number one was the secrets needed to be to creating a new secret file requires someone with is your fee access So that it gets encrypted with the The signing key with the encryption key In key vaults so that it can be decrypted in the pipeline So I think I don't know if you're asking you how to do that, I think you would have had access And so the balance But anyone else Well, I hadn't done it before either at least For a new father, so I think Olivia is anyone's credit new file before And I don't think I'm in that if I remember looking at that vault So that's an action item for me. We need to document and test drive it by having me submit something Yeah, I think I think you have answers Mark access Yeah, so you have access to do all of it Yeah, the other thing was that market you couldn't find the pod and it's the first price there So for some well for some reason the job DSL conflict isn't getting reloaded for Jenkins and for when the job runs And sometimes seems to need to be rebooted and then on startup it loads it So, I mean that shouldn't be happening but for some reason it for some reason it is happening And it was like a question of it was on a different communities cluster. We're only running one communities cluster It's the in all the configuration isn't get so it should be able to be found by anyone It's in the jinkers up for namespace Sorry, was that I Need to capture I was I was left in the awkward position there. Well, I don't have access I I have permission to access the communities cluster. I have access to but I've lost my documentation to describe how to do it So I've got to remind myself. This is how it's done do this Sorry After that So while I was away someone merged the updates to a live PR for cert manager and Cert manager had a break and change in it. I mean it was crashing after updates Hey, now, tell me forgive my ignorance on that one. What does cert manager provide? That's the SSL certificates for sites. Yeah, it's the two things issues new certs and to renew certs So if there've been any renewals during that time, they wouldn't have happened But I think we were fine. It was only out for a week Less than a week and it would have got renewed During that but it is a risk Yeah, so I'm monitoring I monitor SSL cert SSL cert health Pretty regularly. So but I've actually got a problem on that that I should put on the list here that I need some help with so SSL cert Erratic Okay, go ahead Yep, so yeah, that was crashing Eventually well, there was to us Yeah, there was two issues with that It took a lot of fiddling and running it from my shame and whatnot, but had to update the API version that we use for Certificate objects And the other issue was It was complaining about not being able to find the DNS solver. I thought they changed the syntax Turns out that they've made it they've made it case sensitive and encrypted in the secrets so the the The DNS configuration is encrypted and charts secrets and encrypted it was all lowercase and didn't match the brick case So that one was a pain took to the wildfire So I fixed that and Then today again, the pipeline wasn't green because the LDAP certificates Was failing to upgrade the home release And so I went there the old apps that was missing because of all the delays in reiterating So the cert was there just a certain object that we use it was missing And so tried to update it. It was impossible to update because the old ones didn't exist anymore Possibly there's a way possibly they were bridging it between versions So I'm guessing for a few versions. They had both versions available. Hopefully. I don't know. It's hard to tell I could take it through history, but I So I thought I was gonna have to put easily LDAP, but it didn't really want to do that It would have taken LDAP down I would have had to make sure back up to working and taking the local taking a snapshot and everything And didn't really want to do that. So I ended up finding a way to I use it I use that this home release and Up and told and told the current version that was actually using the new API version and then I was able to update it fine There was documentation on the whole website on how to do that It's kind of like Hopping into the database decoding it Under thanks you basically have to you have to decode it you have the ones that but you have to Modify it and then re-encode and Resurface So lots of dark magic was done there in that world yes, but I didn't have to take all that down Okay Thank you very much. I would never have been able to do that that level of delving in deep dive. Thank you. Thank you So those are all the issues that I had and Yeah, we just we we tested we created a PR to pipeline library pushed it as an origin branch and Tested it without authentication and then with authentication and all worked fine and I haven't seen any complaints Yeah, it's been happening on it a bit There's a few improvements and cleanups and whatnot Like I closed the ticket that Jesse has and close another ticket I found I also closed a whole bunch of other input tickets that were just sitting there open, but they were done ages ago closed like 30 plus tickets I think Thank you. Thanks very much Anything else on the experiences on incrementals on Kubernetes? No, I don't have some Hey, so next topic then Kubernetes upgrade Yeah, so I'm wondering about doing a communities upgrade. It's running on a outdated minor version of 1.15 1.15 in general is going out of support at the end of the month Not that we have Microsoft support, but it's no longer supported by Microsoft at all at the end of the month and I was thinking about doing an upgrade of it Should be fine In general and upgrade would be to like 1.18 1.19. What's what's your probably 1.17? Not sure it might have to be a stage to upgrade across multiple not sure The stage to upgrade would mean it would Iterate across multiple Components might have to go to like 115 latest before going I'm not sure. Ah, I see okay To 1.15 dot latest Now it doesn't require a transition through 1.16 as well or it's enough 1.15 dot latest Potentially and then all the way to 1.17 This is the upgrade menu, but I can't see the upgrade menu Mm-hmm Very sure here we go So we can go to 1.16 dot latest and then and Then I think we can go to them. So if you go to 1.16 and then 1.17 I think Okay, so it you already know it is a it it would be a two-stage upgrade. We would go 1.15 Currently outdated to 1.16 latest and then 1.16 latest to 1.17 Yeah okay, and in terms of do is is the idea there we propose a plan for when because that I assume would take an outage It's all rolling upgrade and you know, I sold nodes Okay. All right. Thanks. Okay, and Is there any threat any significant threat there from incompatibilities between 1.15 and 1.17 surprises that we might encounter I Think there shouldn't be anything on 1.17 1.16 I think there's one which I don't think will hurt if we do this it's mine affects Is this one that you want to speculate a timeline when you'd like to do it so by end of month So we've got roughly three weeks. Yeah, I was planning to just do it one morning this week like early morning my time Just wanted to raise a VMS Is it okay if I just say London time you may not be in London London is close enough It's the time zone. Okay Yeah, I'm an hour away from Denver, but for me, I'm in Denver time anything else on Kubernetes upgrade Hey a mirror status report. So this is one where I think I need I need synchronization to understand More details about where we are. So we've got we've got get Jenkins IO, which is definitely running the new solution Yeah HTTPS and it's mirroring to Six sites at least And those sites include sites in in Asia Europe and the US Yeah, now the the old solution is still running. It's still running on updates dot Jenkins dash ci dot org Or is that has that mirrors dot Jenkins dot see I don't mirrors. Oh, thank you mirrors. Thank you It is HTT only Yeah, the mind the mind problem with the mirror by still being running is People keep going to the old mirror the old mirror status hides and Getting confused because there's new mirrors on mirror bits. Oh Right, right, and I have the action item to fix at least one place that causes that confusion. Sorry. I haven't done that yet Mark to update the Jenkins IO mirrors mirrors page It's sort of a freestanding page that looks like an old really old blog post But it has a link to the status page that's misleading The only thing that I'm aware of that is Still using mirror brain is release candidates for Jenkins LTS versions Yeah, I thought that Jenkins LTS can and I thought that the windows releases were not yet all there Let me do a quick check just Yeah Jenkins IO If I remember correctly when I looked at windows This is the wrong one I need one that will show me the list of files. No Just try go to update center and get it Get dodging a size not the best for browsing, but if you've got updates. Ah, okay, so updates Like that, yeah and Index of all releases of Jenkins core, I guess Okay, so if we look at No, I was looking for Yeah, I think I'm looking for windows stable aren't I now I'll have to look for it separately But I I thought windows releases were still on the on the old infrastructure because if I go to download Here This link takes me actually through a page that uses here I'll do it and Cancel and this page Is mirrors dot Jenkins dash CI I'm pretty sure it is Available on But I think we may need to update the links. Okay, great. All right, so that's That's I know there's an info ticket for that one to remind me that I need to go do that and so Needs link update and confirmation That of availability on get Great. Yeah, so what I would like to turn mirror right off just so we don't run through systems, but so I think we need to Yeah sort the windows and the Jenkins LTS release candidates Well in the LTS release candidates, I think that's a negotiate with Right just a matter of Hey persuade him either we need to create a Release dot CI build process for LTS RC's Or we need to somehow give him permission to upload to to the new structure Yeah, I assume the proper way is that he uses the release infrastructure to build assays Yeah, I mean he can already upload to the new infrastructure. He's already have access He's gonna access to upload to package. He's got access to upload. Yeah The other benefit if we build with the build infrastructures, we get signed war files right now If I remember correctly, he can't sign the war files because he doesn't have access to the signing key Yeah Yeah, they'll be properly signed. They'll be they'll use all the same distribution and right Okay, anything else on the mirror status report No All right, so I've got one this wiki dot Jenkins dot IO SSL certificate I have tests that regularly check for the SSL certificate expiration date and Oddly enough my checking tool is using a W get call from check MK and it Periodically reports that the certificate will respire in seven days expired seven days And if I refresh the the thing it says no, it's got a long time For instance, if I open it now on my web browser It's going to tell me That the certificate is valid until oh No, now it says only six days left. Okay, so I see the problem now and where my web browser. Okay, I get two months You do okay, so mine does not so mine got the outdated one So there's some some issue then because mine says the certificate expires in six days. I get 16th of November That okay, so something's wrong on that thing you're getting Yeah, mine is expiring the 14th of September. So six days from now and now if I refresh again Now it's the 16th of November. So it just switched So there's some there's some configuration problem on that computer I restart a patch It what just rest restart the web server. It's probably an old process or something Right, of course because yeah, that's good idea hadn't thought of that. Thank you. I can take care of that great And if that's not it I have to investigate further. I can't imagine it could be anything else I don't know how that system set up really Okay, so we had carried over change of docker service terms of service. I don't think there's anything to discuss there is there Tim there's one suggestion from Tyler which was Running a just run a pull-through registry mirror. So for our infrastructure, we could just run a just run a cache Yeah, and we had used something like that before where we ran I think we ran an instance of an artifact cache Yeah, not a docker registry, but an artifact cache in the infrastructure and chose to switch it off so that That's a that's a topic for discussion there probably do it for artifact tree. I would say Yeah, actually, I wouldn't point users at it But for our use I think we could have it as a mirror and that probably solves the concern Daniel had that Not so sure about if they would want all that track of going to artifact tree, but if it's just our infrastructure, it's not very much And so that that's an alternative good then Let's see. So and I've got conversations going on with J frog about Artifactory there's been some concern expressed that our usage is quite heavy And so Daniel Beck and I will have a conversation with them about their continuing sponsorship Of artifact repository for the Jenkins project. We hope to keep them continuing We very much it's very central to the to the Jenkins development flow Yeah, that one's quite critical Yeah A lot of bandwidth that we don't want to have to run Exactly and and that was their concern is they see a lot of bandwidth use and so now we'll have a discussion Hey, is there a way that we can reduce the bandwidth use? What could we do to continue how do you continue hosting so that we don't have to host our own artifact repository? I sure was yep. I seem to having it I I don't know and that's where we'll have the conversations with them to understand. What is the source of the traffic? Hmm All right, then Jira upgrade plan. My apologies. No progress since last meeting I've I've asked linux foundation to schedule this The session 2.249.1 releases soon. So I should have time this week to work on the jira upgrade plan I had a conversation with oracle cloud last week and They are focused on enterprise enterprise use cases for cloud including c i n c d and are interested in working with the jenkins project One of the things that they noted is that their bandwidth costs are significantly less Than bandwidth costs from most other providers Which was interesting to us because as I understand it roughly a third of our azure bill is bandwidth So that might make it very very helpful if we could Reduce costs by having them help us reduce bandwidth costs. Yeah, so last month that bandwidth cost was 1000 us A month before that's all the month of the month before it was 1800 Ah, okay I don't know why I'm not sure whether it was security releases. Maybe I don't know but The summer isn't two months ago the bandwidth cost was huge and then last month we swapped a lot of traffic over On some more we should have taken us more bandwidth Well, and I was concerned about this month and had looked at the at their billing prediction And we were under budget for the month of august. So we've got 10 000 that we budget And as of the last day of august their estimates said we were well under the ten thousand dollar limit even with The bandwidth changes you made for mirroring. Yeah, we went over the month before that All we did. Okay Not by much though. All right Okay, last topic I had was release status reports 252 256 release today, but doesn't have a change log yet I'll create that And two got 249.1 is LTS tomorrow Thanks, Tim to you for your fix for pipeline stage view. That was the one glaring thing that I had seen Yeah, I think that was a combination of Beaming and tables rework It was using it was using the tables header background color and the table color change, but it wasn't using the table header text color Oh Yeah, this core change I could there's another one about transparency, which I couldn't reproduce I'm not sure if it's fixed in new weekly Um Yeah, so the the transparency thing was definitely fixed in 2.249.1 with your change when I checked it Mine didn't change transparency Well, I guess what I I should say it more clearly The place I saw the transparency problem was in 2.235 And and that problem was resolved in 2.249. It was no longer a transparency problem. It was I think that must have been fixed in a weekly as yeah, I couldn't reproduce that one Okay, great All right, any other topics? um Just maybe what's a what out what else is going on with Oracle? Is it Is this just more talks about what they could offer or what we could do together with them Right. So so one of the record the two requests I offered to them were Hey, we would love to have compute capacity Have you contribute compute? So that we could use it with uh for ci dot jenkins.io I suggested having them contribute mirror capacity Would help reduce further further bandwidth demands for uh get that jenkins.io And I've also asked them if they are interested in donating to google season of docs To provide a cluster for our writer. Uh, they may say no, I've got another plan if they do Yeah, so the other thing that Olivia was talking about I think before I went away was People like they've got anyone that can even contribute for a bit of time on a project On mirrors or whatnot Like company contributed company contribution is The biggest benefit that we can get really And that's a good point. I did not discuss with that that with him But let me bring that up with him in the next meeting. That's a very good suggestion That hey, if you've got someone that you could and they they have someone Who came to them from Susie linux and is very much deep into into Infrastructure might be a great contributor if we get five or ten hours a week of that person Yeah, definitely Excellent Anything else But not on that just you know in the k8s cluster Shouldn't we can shouldn't be a problem to get a k8s cluster? Is there is a reason not starting locally? Is it? Like laptop power or So one of the one of the needs is a jenkins x user who doesn't or jenkins x season of docs contributor Who does not have enough laptop to do it? Um and the the jenkins google season of docs contributor She will do it with her laptop initially probably for the first my assumption is the first month So there it's not a not the crisis for her that it is for the jenkins x person Um, and so that's that's the the constraints there are We've got a little bit of time to work on it for the jenkins portion of it The jenkins x people do not and it turns out I've learned I've got to find a different solution for them anyway I mean I'd speak to you can Just sign up for a free trial of one of the cloud providers in the meantime anyway Right and and that's that's a that's a good fallback for For the case I confirmed thursday or no yesterday with zena That she hasn't done a free trial yet with any of the providers So she could use that for a for a month's worth of cloud resources already All right, if this was if this was like a month ago I was at kubicon and they were throwing trials out on everything right exactly. They were giving them away, right? It's a I should have got you like trials for every provider probably to like into the year or something right Um, but yeah Time didn't quite work out on that. Um, yeah apart from that. Um, there's like, yeah k3s and It's not very heavy Well, I'm sure we've got room on the bill to chuck a cluster on anyway either either on aws I'm sure there's room and on a on kubernetes. I'm sorry. I'm curious when I zero There's we've got room on kubernetes on a zero bill as well All right, I'm sure they're not doing very much. They're just running it with some small projects Exactly. Yeah, that's that's my assumption as well the the estimate and the experience with the last google season of The google summer of code I for jenkins x was it was about 150 a month Spend for them, which is still not but not a huge spend Yeah, I I'm sure we can get that down as well. I don't think that you made as much as what car recorded, but um I thought you want to go for that. It's fine I'm just trying to find the simplest path that I can to get to get to a solution So I'll I'll keep looking for that just to have everybody aware And I'm I'm I love to hear suggestions and happy to act on those suggestions to explore all their ways to do it Yeah Cool All right, thanks tim No worries. See you The recording will be posted. Thanks everybody