 Oh, my machine is frozen, can you still hear me? Yes. OK. OK, everyone, welcome to the Jenkins Weekly Infrastructure Meeting. I'm currently trying to open the correct windows on my machine. We are the 4th of October 2022. Today we have Mark Waite, Stefan Merle, Daniel Beck, Erwe Le Meur, Kevin Martinsen, and I, Dimentic Portal. OK, the notes are being started. I'm having issues today. Sorry, I'm a bit jet lagged, so I might not be able to carefully do things. OK, now I'm having a real issue. OK, let's get started with the weekly release. So it's in progress, right? As I can see. Yes. So the tag has been placed. The change log has been merged. So it's proceeding no known failures yet. Announcements. So the DevOps world and the contributors did not happen because Yuri can try to go to that conference. So yeah, I think that no one was hurt and everyone is safe. So that's the good news, which means that we have a few tasks that have been delayed, especially discussing and working on the election process, on the artifactory system, even though they did quite a great job on some of these areas, and the Twitter account recovery. Because Kosuke never fly to the conference. It was cancelled before, so we weren't able to exchange on the way. Announcements. I've rotated all my keys, but I mention it now. I had a security issue on my auto room. So my laptop wasn't touched. My security key were with me, so it was missing a second factor. I checked the boot. It was shut down and no one rebooted it. I saw no access on either the Gitter blogs, in a commit and push, even read access. And I never saw any access on the SSH to our machines. So the AWS account, Azure, I've checked both. They were untouched and no CloudBees account on the CloudBees account that I used. So that should be OK. But as a matter of safety, I still rotated my SSH key, all my passwords on any account related to Jenkins. So now, if you think about something I should have rotated, don't hesitate. Because with the stress of the trouble losing my things and being a bit ill, I might have forgotten something. So I used my, I got a cheat sheet for such events. But still, if you think about something, don't hesitate. Do you have other announcements, folks? So I'm on the DevOps road topic. I've started a think document about how we approach the contributor summit. I haven't shared it with anybody yet. It's going to have to be online. I think we need it. But I don't think we should attempt an all day event. Online things tend to work better if we do one or two hour segments. I'll start that document, share it around, and invite people to comment. And then I'll share it more widely in the community once I've gotten alignment with a relatively smaller group of people first. So I think we need the summit. It's a great help. We've got lots of topics and then we've got to decide which topics go where, et cetera. OK. OK, other announcements? OK, so we can just check the calendar. So next weekly is able to happen. I need help for the calendar. I'm not thinking clear enough. Next October 11. I'll put it in, Damien. OK, cool. Next LTS is tomorrow, right? That is correct. So that means we should restrain from merging anything from now until the LTS is finished. That may think because what issues did we add during the last security issues for the LTS? Where there are some, yeah, that will be worth it to double check after the meeting if there are things to be back ported. Because last time we had issues, at least on the Docker images, minor issues not blocking the release process but still annoying for the end users. We need to just check because I remember some things where to be discussed. I think it's related to the GDK8 being overridden. OK, so that sounds like that should probably be part of our weekly checklist for today that we look into those things. Maybe you can send them to me or drop me suggestions. I know that the LTS packaging was being reviewed carefully by so Chris Stern, the release lead, proposed a pull request. Alex Brandes looked at it and Basil Crow looked at it. So I think we're in place there. However, it's a very healthy thing to use the weekly to be sure we check it. I like that idea. But there are things that cannot be checked during the weekly, especially the missing back ports from the master branch to the stable branch. OK. What I will do is that after that meeting, I'm saying it aloud so it's shared in terms of knowledge, we'll compare on the genkinsinfra slash release. That's where my concern is. Genkinsinfra slash release, the release script themselves and the genkinsci slash docker officially made. So for both, I will compare the master branch with the stable LTS branch and do a thoughtful review of each item if it can have an impact or not. I'm thinking about the docker image used for making the release itself, which is updated quite often. But that can have issues, especially if we have diverging LTS GDKs, if we have a patch change or something. Last time, I remember the issue was that we accidentally had a more recent version, but that opened a C-group version to support for container image. But that created bugs that we were lucky to have a Bazel and JC, at least, who caught the issue quite early. So they were able to push outfixes on the plugins, impacted plugins with that version. But that might be one of the impacts. We need to be careful on that. And no official security release. And next major event for them, maybe. So it's in a few months, so we have some time. Yeah, that's February of 2023. Here's March. I forget which. February, correct. OK, just a word on, unless you have something else to add on the announcement on upcoming. OK, so a word about the tasks that were done or closed last week. We had a few issues related to, let's say, repository permission updater, which weren't really issues. So I saw that team open. You had an issue with expired credential, which is word. But the repository permission updater, we checked it at least once a day, was suffering no issue, no fail build. So it was working as expected. And it's monitored. So I don't remember exactly what was the root issue that team detected, so we closed it. Ah, OK, so he had a failure in the upload to the repo on his plugin. And so that's why Maven repo said, oh, no, you are not allowed to push the release because it was partially released. Release are not important with that setup. So we had to increment the number and do a new release. So no issue on our case this time. Some login resets, some people answer, some not. Thanks, Erwey and Mark, for taking care of that. Same for the multiple commit permission. Most of the time, people ask for permission, and then they are directed to repository permission updater documentation, where they open a pull request, which is a separate topic. That shows that the LDesk is really nice and true point or easy because people are opening issues on that one. Let me try to remember. Some issues I don't know. Great work, Erwey and Adrien and the old plugin ill-scoring, which is now deployed in production, ready to roll. Now it's being updated constantly. Acceptance test harness, it was already done before the DevOps world two weeks ago, but it has been confirmed that everything is set up and went because they published a new tag and the image was pushed. And I know that Alex Brandes was annoyed by the failing linting of the Docker file when he tried to change the operating system of that image, which shows that it's working because that's a nice negative test. He was breaking things and it was blocking his pull request. So work very well. Thanks, Tim, for the older archiving. And finally, core release, staging, not working anymore. Yes. So Daniel, I'm not sure. Yes. Are you OK with Bazel analysis about the fact that with all the Maven plugin changes, not the one I proposed, but all the other that you and Bazel at least worked on, now the staging should look good, right? Or it should be solved. I don't know if we were all we could test it in real life. I think his analysis makes sense. I have come to the same conclusion when I read the log and which commands we're using that we're always staging and always staging to a custom repo. It's just that sometimes the custom repo is also the default repo same URL. And so Bazel pointing out it says using custom, using alternate deployment repository is what I think we wanted to see. So this seems resolved. Cool. OK, so that was also my understanding. But I have less knowledge than you and Bazel on that area. So I just wanted a triple check. Thanks, Daniel. That's quite clear. So comment to another issue that I just saw. The anti-spam system was triggered the first done entry. So the response to report of God was that there is no user registered with that email. But the anti-spam system would prevent user registration. So that's kind of expected, no? Yes, yes, I miss my question wasn't really as you said. Yes, I checked the blacklist in the app. And I didn't see any relationships between this person's name or surname. And the main with the blacklist. So I don't see immediately why it has been triggered. Have we deployed the changes to the denialist that I opened the pull request for about a month or two ago? If it has been merged, yeah, it has been really, really. Yes. Yes, it was deployed. I don't remember something to be checked, Hervé. Do you mind trying to check on the logs? I don't know by like this. When someone hits the spam check, an email is sent to a private mailing list. I think it's called a column admin or something. Yes, about this email, I also wanted to change it to a private infra team one with eventually a redirection, an automatic redirection forwarding to the original email address. What do you think about it? Because I don't have any visibility on this email address. I'm not sure. Sue, I mean, as long as it's a private list, I don't particularly care the thing. So correction, this is a very high volume list. So I would suggest to not send the emails to a regular list. Like this is just an archive that you occasionally need to search. So I would leave it as a separate list and you just never get emails for it. In your subscription settings, you say you don't want emails. And whenever someone hits the help desk, you just browse there in the interface or something. That would be my suggestion. OK. In the case of this person, they hit the IP blacklist. So nothing about their name or email address, but an IP that 10 years ago a spammer probably used or something. OK. I forgot about the IP address. Yeah, we already opened it. But what can we end blacklist this IP address? We can probably just clear out the list of IP addresses. I don't know. That might be something to check, you know, spend half an hour or so to look at recent blocked account signups, whether the IP block list is doing anything that looks useful because we have so many IP addresses blocked. And this should be a time-sensitive data. It should be blocked for a time being not forever. Excuse me. Yeah. Yeah. Something like code sec could be eventually implemented instead of a static IP list, IP address list. I know this link I've put in the chat allows to have a crowd in an open source collaborative blacklist. I don't know if it would be useful or not. I mean, sorry, I was busy looking at the IP address. So we're blocking an entire slash 16 here. So that's bound to hit some folks that are innocent. Regarding any other integrations, is it worth spending time on the account app? So what's the situation with the other system? We didn't progress on KeyClick or anything since a long time. The main issue is that we are constrained by an infrastructure network issue is that that application should run on a private network, which is not today. So that's why we're not moving it. KeyClick was going on that direction, but we have infrastructure issue. And then it requires time. And right now, our time has been written by Gifr Augustev. Makes sense. So I think in the short term, we can probably do without the blog list until we see actual spammer activity again. Some of the blog list entries have explanations, like comments pointing to IP scans that look like a spammer. But I don't know how useful they are like five to 10 years later. Yeah. Cool. Thanks for the explanations. More info on that topic. So then let's proceed to the currently opened issues that we are working on. So as a reminder for each issue, the question is, what is the status? Are we going to walk on until the next weekly, which happened next week? Do we have time or is it lowered in priority? So first one is about Jen Kinz released Twitter accounts. So we were planning to ask Kosuke to do a two or three person key rotation so we could access the DLVR account. We currently have access to the Twitter account. The question was, we need the DLVR account to see why the automation is failing since weeks. That's the status. LVA came with a proposal that was let's remove DLVR token and integration, and we can rebuild a new automation. As a reminder, the goal of that Twitter account is only to tweet about new plugin updates where the source of truth is the RSS feed from Jen Kinz. I often don't mistake a plugin Jen Kinz. So anyone who will want to help, we need to add a new automation that does this, basically. Yeah, also the Twitter together action has significantly been updated this last time. And it seems now really compelling. I think automation for this account and eventually the Jen Kinz official account could be really great. You can check the tweets, you can do polls, you can add media. You can do a lot of things. And you pull a low to have a review on a repository with different team reviewing it and validating it and scheduling it for a long time or anything. So yeah, that could be a good entry point to see how it works. So regarding the official Jen Kinz account, I think that's worth going to one of the advocacy SIG and or at least sending them an email and work with them on that area because that's an account we don't have access to. Regarding the Jen Kinz, yep. Yeah, I know Gavin was really interested to use this particular repo and it was a little bit in standby because it has some problems but there has now been resolved. One of the Matt Coway from Digital Océan really became a maintainer and push great updates or something. Okay, cool. So I raise it, okay. So are you able to continue on that topic and explore the new automation? Because right now it's still an issue because since the 8th of August, we didn't add any tweet on that account, which we have an account. I just need to check out the assessor. This was transformed as tweets, but yeah. Okay. So should I add you assign you there? Yeah, no problem. We can delay or it's only a question or will you have time to work on it? Given, okay. I haven't created a new milestone yet so I will take care of that. I'll and create it. Next topic is still for you, survey status of the artifact caching proxy for CIG and Kinsayo. Can you give us a quick status? In next week, I have successfully created the AKS public cluster dedicated to, for now, this artifact caching proxy. As soon as I have some, I'm currently working on the network balancing integration. And as soon as it's working, I'll be able to create a pre-request on the dozen or so plugins where we will test this functionality with all provider available on Azure Digital Ocean on AWS. Okay, so as a reminder for everyone, one of the reasons why it's taking time, it's because we are hosting it on Kubernetes clusters, but for Digital Ocean and Amazon, EKS, we have that issue that's the threat model of the container agent from CIG and Kinsayo are expected to run on cluster, Kubernetes cluster, which is only targeted at doing that. So adding a service with load balancer exposed publicly is not part of the equation. That's why we have to create separated cluster. The overhead cost in terms of infrastructure is low because we don't pay a lot for another cluster. But the issue is that we need this cluster in the same network area, but in the same network region, but not the same networks. So that took us a bit of time to create a new infrastructure. The good thing is that now we are able to manage way more clusters, which is a good thing in terms of reliability and validating our processes. So I assume that we will continue working on that because that's our top priority. Next topic unless any questions. So publish pipeline step dogs generator and backend extension indexer. So these two repositories generate static files that are then used for generating the Jenkins.io website. The issue is that these files are generated inside CI Jenkins.io in the form of when the build is on the main branch, if it's success, then it archive the artifact and since it's public access files, then the generator for Jenkins.io downloads this file from CI Jenkins.io. That creates a lot of problems and adherences, mainly with the security release process because it requires CI Jenkins.io to be always up and available. So the goal of that issue is to ensure that these things are built somewhere else and pushed on the public reports Jenkins.io website alongside with all the other reports. So we started working on migrating these jobs inside the infra.ci, the private controller enabled with the build log on the GitHub access. So the build log and not the deployment log but the build log is available to contributors. And now it's a matter of ensuring that we have the same feature sets between CI Jenkins.io and infra.ci. That's a problem we never had until today but we have it now. The problem was between CI Jenkins.io and release and trusted until today. Reason is we need the same tools with the same name and the same inside Jenkins and the same label also. So it's not complicated, it's just we have to carefully review and it's back and forth trying the pull requests. But almost there and once it's done the reports will be published. So we'll be able to open a pull request to Jenkins.io. So instead of downloading the static file from CI.G it will download them from the public website reports Jenkins.io. The reason on that is also because we don't want any private credentials on CI Jenkins.io. It's considered a public instance. So relying on it can be dangerous in terms of either quality of service, resiliency and security concerns. So I will keep continuing working on this one unless there is a strong opposition or any question. Okay, next task. Finish cleanup of mirror brain. I worked on it two weeks ago. So that one has a potential of great mayhem especially on the plugin and release. So I was waiting for the LTS to be done so tomorrow and then I will work on that part. The main goal is to ensure that we don't have any leftover from mirror brain. But the investigation show that that user was manually managed on the virtual machine for almost five years. So there were some hidden Chrome tab which are quite important because they run every hour to synchronize all the latest plugin updates to the mirrors which can be quite useful, right? So the goal is to create a separated new account with a dedicated name which is not mirror brain and try to put everywhere. So the goal will be first step, create that account and then we will start to moving the permissions, scripts and tasks to that new account and see if it breaks. So the goal is to have at least one change per day when it's will be ready. And in any case, we can roll back to the current mirror brain. And then after one full week of this taking care of synchronizing everything, we can delete safely the mirror brain. The main reason of that change is because we need to move the updates to your website from that machine in AWS to a new virtual machine in Oracle that Stefan prepared a few weeks ago. The main thing is that these scripts have to be part also of the new machine. So that means the new machine in Oracle will be responsible for every hour or when it's asked remotely to synchronize the change to the host U.S.L. mirrors, to the get junk in SIO mirror reference and cache. So that part will move away from AWS as well, which means in terms of bandwidth, we won't have download public bandwidth that costs us a lot, but we might have some outbound bandwidth from the trusted CI agent that generates the new plugin updates to that machine that bandwidth wasn't existing or at least not costing us a lot because it was cross regions, which won't be the case anymore. However, we were already pushing to host U.S.L. or to Azure. So the outbound won't change a lot. But keep in mind that this might create some mayhem. So we'll try to be careful and ask at least someone else on each time to review it carefully. I won't try to push things too quickly there. Next task unless you have any question. So I plan to work on that task next week, but after the LTS. Next one is realign repo Jenkins CI. That one I opened the draft GEP to explain the proposal, but I wasn't able to fill with too much details. The goal was to work on it during the community submit last week, which did not happen. So I need some help for reviewing that end of week. But my plan, I've put the proposal, which will be enabling authentication only on the mirrors on the repo Jenkins maven and try keeping public the rest. That might have some impact. So the community and contributor must be asked for advice is there because we are clearly some element that will be impacted. Either the main build pump or at least the way we're under documentation for developers if they need to be authenticated. The goal is to follow up the assumption that we don't cross the GIFROG download limits only with the Jenkins artifact themselves. But as they told us, most probably that's the free mirror of most maven repository for free for everyone on the internet that will cause some downloads. So the goal here will be avoid making mandatory authentication. So we won't be overwhelmed by the amount of companies crying because now their mirror of Jenkins repository need an account and it's complicated and stuff. Well, in fact, we only target the developers. If you are a developer, there should be no issue for you to encode your Jenkins held up password in your maven setup. So you can use the mirrors that we provide or see a Jenkins, of course with the work that Hervé is doing. So that's the earth of the proposal. Now let's see, we need advice, lot of opinion and go to a consensus on that part. So Damien, I'm not sure how to configure my or can I get some coaching, not here in the meeting, but can I get some coaching on how to configure my settings on XML? I certainly have credentials that I use to push. Are those same credentials then usable when I pull in this new world of authenticated access? Yes, the credential that you define on your home M2 settings XML, they are scoped by using the repository ID. So if we have an exhaustive list that come from mostly from the pump around on Jenkins that define today the repo Jenkins CI slash public, which has an ID and there is a second one for incremental. So it's like, I think Jenkins public and incremental are the IDs inside maven setup. So once you know this ID that doesn't change often, you can set, okay, that's couple of encrypted credentials is for the repository with ID Jenkins public. That one is for whatever my company private, whatever my personal private and then maven links them by ID. So you can have multiple, that's the idea. So that will work if you use the correct snippet. And so that means my CI jobs scattered on agents all over will somehow need the same or will need access to a credential that will allow them to access the repository as well. So CI jobs, my personal and whoever will need credentials to access that. Thanks, okay. So it's, I think I can see how to test that, thanks. Collect data dog metrics for ephemeral virtual machines that's your area, Stefan. Yes, and that's linked with the other one down. And I'm working on the Windows part of them. I managed to deal with the Azure ones with the cloud init injection for the API key. And now I'm working on the EC2 version of it. And on the same time, I'm working on improving the checks on the availability check that we had before and to have more, to improve all those checks for the Linux parts and for the Windows ones. Cool. Nice work because the way it does, it's not easy. Were you able to pay or not with the dashboard part on data dog, which is the last one? I will need your help. I did manage one thing, but I cannot mix within the same dashboard, different kind of information. I might need your help for that. Okay. So we keep on working on that. Yes. Okay, yeah. So let's keep continue working on these two. And finally, an issue that was open earlier today. Thanks Mark for handling it. So someone say they lost access for publication to the crowd to plugin. Okay, they changed their account two weeks ago. Yeah, most probably they messed up the encryption of the password on Maven, that's the usual. Is there, except telling them to use CD, is there something else that we could do? I checked that that person was in the plugin maintainers in the repository permission of data, which is the case. So I'm not sure, Daniel, is there any way to check for the authentication on the Maven repo since I might mean I could at least check for an error or something? Yes, if you check within five minutes of the error occurring because... Ah, yeah, correct. JFrog uses extremely aggressive log rotation. And it's also stupid log rotation because every few minutes they truncate the file to nothing. So there is basically a next to 0% chance that you see the log messages from the two seconds before they do that. I would emphasize Mark's response because I think there are ways to encrypt Maven passwords that are very different from what Artifactory does, which is why we have very detailed instructions. So I would like the reporter to confirm specifically that they followed the instructions on this page and not some random other Maven encryption instructions. Yeah, Daniel, to support what Daniel said, that's been a consistent pattern that we've repeatedly had people who needed to be reminded these instructions are exactly this way because they're the ones we know work. And there are lots of other places where you can read about instructions that ultimately don't work. Yeah, so recently someone also emailed the dev list saying the instructions don't work and I was confused about a repository they set up and they said, well, I copied it from the XML file from your instruction and my response was, well, read the last step that says ignore everything in this file except for the password step seven. So, yeah. Okay, fair. So thanks for the knowledge sharing. I didn't realize that, but yeah, seeing the instructions was clear, so let's... It is the nature of detailed instructions like that, that it's easy to make mistakes doing them, right? And so, but all we can do to tell them is, look, you have to do exactly what the instructions say. We understand it's easy to make mistakes, but they're exact for a reason. Right, yeah. So, and the instructions are weird. Like download this file and ignore 90% of its content. That's not what would usually be on the success path of any same process. But Mark, you commented there, then there was another comment and it seemed like the reporter just responded to the last comment there. I would like them to acknowledge that they have done what Mark asked. Yeah, the reporter just responded to Damien's comment. So, I suggest you ping them and ask them to confirm that they followed the instructions exactly that Mark linked. Cool, thanks for the tip. Here we are. Okay, so that was the last issue. Many thanks Mark and Daniel for the explanation. Many thanks checking the new issues or the new important thing that we could put. I saw one to three new issues since last meeting. One by Gavin is just triaging now. We see if we have to add it to the upcoming milestone or if we let it in the backlog. So, there is an issue in the configuration of the infra job for stories, which seems not to handle pull requests. So, that means that we should have look for a reminder. The configuration of this job is not infra as code. I hope we will be able to do it one day. But right now it's manually managed still on CI Jenkins.io. So, the goal will be to look. I fear that's infra being a GitHub repository operation scanning that we might need to move stories away from that repo. So, we can fine tune its configuration if it doesn't map the GitHub organization scanning once. I don't remember the pull request policy for this one. So, better to check. So, something to check there. Is there someone volunteering by defaulted foldbacks to me? But if anyone is interested in trying this. So, Damian, if you're okay with this one, I'm, so what you, what I heard you just say is, I could test this by configuring CI Jenkins.io interactively. And I'm actually reasonably good at that. I'm happy to do that. So, how about if I take this one? Would that be okay rather than having you do it? Because it's, this is not an automation one. And I'll take it on because I'm also interested in this site as part of advocacy. Okay. Maya ask you if Stefan or Harvey is interested just to pair for knowledge sharing that will be a great exercise for one of them. Also, a reminder to see the manipulate the Jenkins concept such as a GitHub organization scanning which might have impacts on stories if you have to move it. And that will show so since you are really experimented with the UI that will help them to learn on the process. And you are as good at that as I'm bad at that. So, I would really need it. Yeah, great, happy to pair. Happy to pair. I'll schedule some time. Second one is pipeline graph view on release CI Jenkins.io. I guess everything was done before the weekly release. So, once the weekly release is finished, we will check but I think they did everything correctly and we merge. So, to be checked. Stefan, can I ask you to check this one since you are handling the plugin and core updates? Yes, it was pleasure. And Jenkins.io account verification on Twitter. I guess it's not really infrastructure there but the advocacy part of the board even. So, it's about asking Twitter from that account. So, they have the blue tick to say that's a validated account by Twitter themselves. I have no idea what are the proof that are needed. Some person are validated some or not. That's when I'm not in these things but we don't have access to the account. So, not sure. Should we send an email to advocacy group? Since I'm a member of the board and I'm permitted to post to that account, why don't you assign it to me? And I don't know that I'll get to it. I'm not sure we should put it in this upcoming work but I think it's a reasonable thing to say, yeah, we should do it eventually. And as a board member, I'm a good candidate and since I have access to that account I'm probably a doubly good candidate. Cool. Thanks, Mark. I think we covered all. So, do you have new subject that we should work on prior to early this week that we didn't mention? Or do you have any question, more information for that meeting? The only topic for me was elections and that's one Damien where I took an action item in board meeting yesterday to get with you since we missed doing it at DevOps World. I used the excuse that Hurricane Ian disrupted us and of course, Gavin Mogan said, how can a hurricane interrupt an online election? Damien, I didn't get together. So, you're okay. I assume the two of us pair up later and do some prep for elections. Yes, absolutely. Damien, you may want to speak about the VPN and certificate something you spoke this morning. Yes, to wrap it up, we will open an issue on that milestone. We have to regenerate the CRL for the VPN. It's a year or six months. The goal is to have Stefan making it and me in support. It's well-documented, that should be uneventful. We received a notification on a calendar that should happen in three weeks, roughly. So, that's the idea. I messed them. So, since we are out of time, thanks for the reminder, but I will add the issue on the current milestone. I know that Stefan won't forget about it. Thanks, Stefan. You're welcome. Okay. I think we can close it up then. Thanks, everyone. See you next week. Bye-bye.