 Hello, everyone. Welcome to the weekly infrastructure meeting team for Jenkins. We are the 7th of December. Today we are four persons. So Mark Waitz, Erwe Le Meur, Stefan, Merle, and Hai. So warm welcome to Stefan who joins the SRE team at CloudBiz. So he will work full time on the Jenkins infrastructure. Welcome Stefan. As usual, this is recorded. You will find the notes and the recording on the usual links that are put there on the invitation. And we will try to start publishing them as well on Community Jenkins. First announcement. So I've seen mentioned that the weekly release has been done. I don't know if it has been finished yet. But I saw some requests opened automatically. So I assume that at least the POM and the war have been released. We didn't change anything since last week's releases. So that should be very good. And I don't have any other announcement. Except the election results, maybe. Mark, I don't know if you have a word about the election. I think it has been published on the blog post. So it has. Congratulations to Damian for being elected with uncontested as the Jenkins infrastructure officer. That's great. And yes, there have been some others as well. So I've accepted the position as documentation officer and I'm a member of the board. And thanks to Oleg for continuing to serve as a member of the board as well. I don't have other announcement. Is there any other announcement for you folks? Yep. Okay. So let's go ahead on the task list. So I'm starting with the notes we took last week. Sorry for the notes we pre-pretake. What about census.org? So that machine was forgotten from recent puppet upgrade or appetite upgrades during the past month. I have no idea what census is doing. But I recently thanks to team and from the note from last week. Team gay confirmed that Mark and Hyde were able to connect to the machine. So we worked on adding that machine back under puppet management. It has been upgraded, rebooted and it works now. So we will need some help from someone that know what that service is doing. I don't know Mark if you have any idea. I don't. I'd have to do some I'll have to do some research. So the it's I think it's a good good point to realize, oh, we've got a service there that I thought it was something about under surveying and understanding what's what's distributed, but how it's different than stats dot Jenkins.io. I don't know. So we're just going to have to do the research. So we might have some no, that's what I remember. So the notes are on the on the runbook. Right. So let's ask Olivier for help on that part if it's okay for everyone. Unless team has an idea, I will start by asking on IRC. And I assume team and Olivier should be able to answer. But the machine is up to date at least. And there is an Apache server that answer HTTP not authorized that ask for a password. So it's a private service. Or there is there might be some specific endpoint that I'm not aware of. I haven't checked the access log. So let's see what is the goal of that machine because it was on AWS. So we have to see if we have to move that service on Kubernetes on the VM on Azure or keep it there. Is there any question about that part? Next one. So Puppet cleanup. Thanks, Hervé and team for helping me. So the Puppet cleanup is we started by removing the some part of the Puppet code that is not used since a few weeks, month or years for some. We started by Kubernetes. So we've put all the pull requests on the notes that we have done until now. That was focused on the roles. Now we'll start starting next week. Also removing machines that doesn't exist anymore from the inventory. We had to do that on the I think that was the last that has been one pull request that we were removing. The role was not sufficient. We also had to remove the machine in order to be sure that it wasn't breaking it or at least not the test. So let's continue. Why do we want to remove that code? Because then we will have to work on upgrading Puppet dependencies and test harness. Because right now it's using it's not using the same dependency tree for production and the testing environment. We cannot upgrade without breaking the test environment and the two kind of test unit test and operational tests are based on bricks and foundation that are to hold and that should be changed. So the main goal will be switching all the testing harness either unit or or end to end to the Puppet development kits aka PDK, which is a toolset that works on multi-platform that allows testing and managing all the life cycle of the test for Puppet rules, profiles, and also simulate machines as Docker container. Once we will have switched all the test harness, the existing tests that can be reused as much as possible, then we should be able to have one line of dependency and we can start improving things such as running end to end test on the master branch or like this. And we should, we might be able in the future to go back to a staging environment. But this is longer. Just to note why keeping a tool like Puppet, because we have machines such as the PowerPC machine, the OSUSL VMs. We have a few machines that cannot be managed as code with Terraform. There is no cloud API to manage the life cycle. There is no cloud API to build to pre-build images every week or for these machines. So we need tools for provisioning for these use cases. So you mentioned PowerPC and I know about S390. What was the other one that you mentioned? We have the two OSUSL machines that were used for Confluence and Jira back in time. We plan to keep this machine and use them as for C-Igen, Kinsayo, Kubernetes agents. But that could be used for something else. I don't think we have other machines like this. All the machines are managed by Puppet, but could be managed by Terraform at the moment in time. And the agents that we use for C-Igen, Kinsayo, those are started and stopped through Kubernetes or through plug-ins on the Jenkins configuration. So they're managed by code as well in that sense. Correct. There is a fifth machine. There is a static agent for trusted C-I that is the machine that holds the cache for the data center. We could have that machine cloud managed, but right now it's not managed at all. So at least we could put it on the Puppet management. Is there any other question about Puppet? No. Okay, let's go ahead. A word on post mortems. First, the post mortem about archives.jnkins.io, the issue we had last week due to Apache starting to enable the bandwidth module after a few months not doing it. The post mortem has been pushed a few days ago. We should freeze and publish it next week. So any comments is welcome. Thanks, Arveille, for the initial review. We had I have to publish the outage two weeks ago from Jenkins.io. Mark, did you have time to go over that one? I did. I made my comments and I think it's... So this was the www.jnkins.io. I don't remember the date, but yes, it's been reviewed and at least by me. And I think it's ready to publish. Okay. So we can still come back and update it afterwards, but publication moves it from HACMD where it's collaborative and it committed inside the GitHub repository under post mortem section. I'm taking the note from last week. Outage, post mortem. Here's the link. I'm opening it on the screen. When you say you commented, did you had some question inside the document, Mark? Yeah, there were one or two places where I added a question. The description seemed reasonable. I think I understood it. I liked that your plan for the things we should do. There were one or two things about, okay, should we also consider this? There was in the long term, I added a question. And I think it's you've already mentioned it. We may consider in the future long term having a way to do prototype setups. Yep. Okay, I don't see where the comments are, though. They're just listed as text. So scroll down to the very bottom. Would it, would it, would a prototype of PKG that was me asking a question? Okay. Okay, so, okay, so we'll take care of answering this question before publishing if it's okay for you. Great. Yeah, and you're welcome to delete the things that I added. If, if it turns out they were just not sensible, don't be shy about removing it. Keeping it is interesting, because if you have the question that means someone else would have, would have had it, so better to answer it instead. Okay. Is there any question about the outages we had during the past week and the post mortem? One, two, three. Okay. Next topic on the agenda, accelerating build and test some more. And so at DevOps World 2021, Gradle Enterprise did a 30 minute presentation on their build and test acceleration product. And, and they, I happened to talk with them afterwards and I said, Oh, yes, we love to have open source projects, use Gradle Enterprise, we let them do it for free. We grant them a license. And we run a series of experiments to identify which things we should work on and then work on those things and watch the results and then based on the results, decide what to work on next. It, the thing that was most interesting to me was not build acceleration, but rather test avoidance. The, the, some of their technology has the ability to decide that a dependency has not been changed and skip tests as a result. And we've got a major cost in our, a major portion of our execution time in on ci.jankens.io is in fact running tests. And I know in the code that I maintain many times, the, the test that's being executed is absolutely unaffected by the code change that was just submitted. But while I know that as a human being, others don't. And this, the computer certainly doesn't. Gradle's discussion hinted that, and there's some concern from Jesse Glick that Gradle's technique may actually not be that beneficial to us. And, and Jesse would know of all people. So, but I would like to have the Gradle Enterprise present to this meeting next week, a 15 to 20 minute overview of their techniques and their technology so that we could ask them questions, understand what it might mean if we were to choose to engage with them and, and what the, the hope, hope for gains might be. Similarly launchable Kosuke Kawaguchi's current company has a test avoidance technique. There's is using some machine learning to decide to, to make heuristic guesses, which tests should be run and which should not. And Jesse thought that that was probably more likely to help us because it's not as strictly dependent on whether or not a dependency changed. And so my thought was we also invite someone from launchable to present to the, to the infra team after conversations in the, in the mailing list. My goal there is really accelerate, accelerate jobs by reducing the number of tests we run so that we only, we, we run tests that are important or useful, not every test every time. Would, would it be okay with the three of you if we invited Gradle to attend this meeting next week for a, for a, an up to 15 or 20 minute presentation and question and answer? Yes. Yes, that will make sense. Um, do, do you think that we might want to, to do a specialized meeting with some people from development teams such as Jesse or James or Oleg, Oleg, uh, because here the thing is that as infrastructure people, we might not be at use of, um, how could we consume Gradle? So I, I fear that we will have a, a well shaped discourse that will be, oh, our tools is serving this and this, but we might not be able to do the correlation with what we is being done on the, I don't know the bomb or all these costly jobs. We don't really know what's under the hood for us. So we might need the expertise, at least for the usage or the way it works from Jesse and the other. Um, because in fact, for us, it will be more, okay, if we have to use this as infrastructure team, or can we, uh, what are the requirements for us to install? What will be the impact? So if we do only during the Jenkins infrastructure meeting, that will be only, okay, uh, what are the steps to install? What are the constraints? And are we okay? But then we cannot decide anymore because yeah, we don't manage that, um, that software stack them. I think you've got a good point. My reason for coming to the infra team first was, I think that the infra impact might in some cases be an absolute deal breaker where we say we simply cannot proceed with, with the experiment because the infra impact is too heavy. I think if we then say, okay, we think the infra impact would be acceptable, we, we may want then to consider a separate session after infra assessment, a separate session that says, okay, how do we, what will our objectives be? And what are the, what will our measures be? And how will we decide if we were successful or not? And, and that would feel like a great place to involve, um, skilled developers who understand more about the details of bills of material and dependencies. Okay. If, for me, it's fine. I don't know, Stéphane, Hervé, what do you think? Yep. Uh, be careful to tell maybe, uh, both companies that they will need to do two meetings with two different interlocutors so they can come with a discourse that will be adapted to the interlocutor they met and they won't lose their time. And in our case, we won't have to try to understand the under the hood that Cradel does while we are interested on the infrastructure impact. We'll do absolutely good. So I, what I'm going to ask you to do, given that we keep this meeting so brief, I'm limiting them to a 10 to 15 minute presentation and five minutes of questions. And then, um, we accept that we'll, we're assessing infrastructure fit, um, much more than technical viability, right? Technical impact. So, and, and a separate meeting for a separate meeting with others for technical viability. Basically, what they propose is to switch Cradel bills to Cradel Enterprise, right? No, no. Actually, they're, they said that their biggest acceleration is by letting Cradel Enterprise help Maven bills. So, so the, for me, that was very, and, and I'm less interested while, while bills may help, I'm much more interested in the test avoidance attribute than I am in the build acceleration. Build acceleration will, will certainly help us a little, but avoiding tests could help us an awful lot. And so that was the intent of the conversation with them. And that's why I think a presentation next week on, on what their techniques are for, um, the first 10 minutes that they present should give us a, at least a high level overview of why they're able to accelerate Maven and what their techniques are. And you'll ask questions about, okay, what would that mean on our DNS infrastructure, on our service or service infrastructure? Do we have to host this or can it be self-hosted for the experiment, et cetera, or, or can they host it for us? And the answer is they can. Those kind of questions, I think we come out of next week's discussion. So the only ask is, not draw it out for any delivery pipelines without wider discussion in the Jenkins developer amenities. Because basically you would be introducing proprietary software into the built pipelines. So, so tell me more about your concern there. So I'll like it. So if we just replace current Maven builds by, um, Redwood Enterprise Potentially, the goal is not to replace just to be the goal of the intro team is to evaluate if we can install it next and then hand it over to development team if they think they can do something with that based on the exchange they will have. That's the only goal for us. No replacement, unless development team to ask for that. Well, and no switch from Maven to Gradle. That's the, they, we continue using Maven for our build. So that's that that's the piece. Sorry, there are two different things. Firstly, retaining the same format, secondly, changing the engine under the hood. Because let's assume there is supply chain vulnerability on the Gradle site due to whatever reason, it means that whoever corrects Gradle, they can instrument it and basically modify Jenkins deliverables without us knowing about it. Yes. Yeah. That's why we need to invite them to, for them to show us an ask and see the technical requirements. And then based on that, we can see which part of the offer is a fit or is not. So yeah, not again, the principle, but yeah, just rolling it out would have some implications. Yes. So that will be the goal of the meeting next week. Yep. Thanks for the, for the warning point. Yeah. So I will, and it's, it's a good point that we should probably invite. Well, do we want a separate session with the Gradle Enterprise team to specifically address security topics? I'm not, I think it would be more than we've got capacity for to fit that into a 15 minute segment during the Infra team meeting. But I think security concerns are a valid place as well, not just developer workflow and platform using security, but actually inviting developers of the meeting. Because if you just sit in the infrastructure management, you know, almost now, but from the developer management is following it, or Jenkins contributors in general. So yeah, they are likely to be some strong opinions because we have a grand deal employers in the community. We have people for Gradle who left Gradle due to some issues. I am not going to disclose right now. So let's say it's likely to be a virtual topic. But if the point is really to allow developers to choose Gradle Enterprise or medium, then you should try to get developers on the call. Yeah. So I think your guidance there is more discussion in the Infra list and the developers list. Good. Makes sense. To be honest, in the currency state, I don't particularly see point in infrared discussion. So I guess in terms of from the infrastructure team site is clear. We use the cost for the bills and probably shorten them. But yeah, I think that the vast majority of input should come actually from developers here. Good. Okay. All right. I'll bring the discussion to the developer list as well. Good. Thank you. So do we still keep them to present at least the infrastructure and technical requirement at first? That will be a first opener. I don't see any harm. Keep doing what you propose the 10, 15 minutes next week. I mean, worst case, we lose 15 minutes of our life. And that will start triggering discussion at least on the security and infra part and requirements. While then in parallel could the discussion for developer to see if there is value on the developer workflow could happen in parallel. I don't see any harm to do in parallel. Is that okay for everyone or is there any risk? I think that's good to me. All right. So I'll mark to invite presenter to next meeting. Great. We'll do. Okay. We are three minutes ahead of the end. So unless there is other points for that one. So for unshable, let's continue discussion next week. I haven't heard from KK for the TLS cert for repo Jenkins here.org. We still have time because it's March for the end of life of the current certificates. I don't know if Mark or leg you can try to reach KK by another medium because I don't know outside email how to contact him. Maybe not publicly by Twitter that might not be the best thing for this task. Oh, he's still lurking around on the cloud. This luck. Well, while I was on this like he was. No, I'm not sure. So what else? I have his phone number, but I don't think it's appropriate to use it. It comes quite hard. Okay. Yeah, I can I can provide you because the only slack I have is the professional one. I have one, maybe two. That's all. Yeah. Yeah, good. Yeah. Private message. Okay. Thanks, Mark. We'll see if we can reach KK. No emergency, but yeah, that could be good if we can do that. If he's busy, we can totally take over again. That's the message. He proposed this help, which is really nice. It's just that we want to be sure that he's not rushing this thing. Oracle cloud. Yes. Mark, if you can ensure I used to have an account, but I cannot get back my password. The password I have is mentioned as not working. So I assume that either I don't have a safe to correct password or my account has been disabled. I can tell with the error message. So if you can just check that team are very high and we have the access. And in order to confirm that it work, my goal will be to create an account for Stefan as soon so that we'll confirm that we are multiple person that can manage the account. Yes. So we're ready. Good point. I forgot to send an email to Tyler. So still to do I.O. Domain renewal. So I still haven't emailed you. If you have question on that topic, please read the news from last week. There is an email to get information and legacy knowledge from Tyler about the domain renewal. That's the main should be the transfer to the links foundation. I'm not sure whether it's still on the list, but we should get rid of the ownership of that. OK, basically the links foundation expects it. Yeah, I was going to ask, OK, I haven't seen. Yeah, I'm new on board. So I don't know the relationship between the Jenkins community and the Linux Foundation for this kind of task. So that's something that should go to the Jenkins board meeting. Yes. So the rule of thumb, if it's something important for project sustainability, the Linux Foundation wants to have administrative access to that. Just in case something really bad happens, so we can review the summit and met our falls on our heads and bring to rest of the community. They can do that. OK, so the question here is what projects do we have and what is the how is it managed and renewed? We mentioned that it was automatic, so that's why we want to send an email to Tyler because Olivier wasn't able to tell us. So the idea is to see, yeah, where is it because if it's an Azure, for instance, it's already available for the Linux Foundation and there is nothing to do for us because they already have access to the whole Azure infrastructure. DNS zones are managed on Azure. So my guess is that I'm sure we have also the Azure as registrar, but we have to check and I'm pretty sure it's actually not registered with Azure, but we may want to transfer there or transfer it elsewhere as part of this this eventual sustainability effort with Linux Foundation. Yeah, that could be an easy way to get that to get that without spending too much time. OK, so on the last the last point for me I had is there there has been a huge work on updates and the pipeline shared library for infrastructure. So thanks, survey for that work. I hope it should unblock gave in the task for new repository. We should be there soon. So thanks for making our life easier. Great work on that part and the upcoming terraform things will start again next week as I will be half and we are onboarding Stefan so our availability this week is slow. I thought. So that's all for me. Is there any other point action point you want to mention? OK, so we can stop the recording. So I see Mark that you posted the on community from last week. Thanks.