 Well, hi everybody. Welcome for this new Jenkins infrastructure meeting. We are the first week of July, which is quite interesting summer vacation way. So today to the agenda, we don't have major announcement to share. So we have few topics that we want to cover. So first one is we had an issue with reported Jenkins.org over the last week. We had few stabilization done on trust.ci. We have one issue with updates or CLI. We also have a topic on the Docker image that would be in ship for the Jenkins organization. We currently have someone going to work with puppets and a few other things. So the first one is let's go back to the GFrog incident. So what happened last week while we were doing the security release, we noticed that we could not publish Maven artifact under the release repository. And at the same time, we discovered that GFrog was doing some maintenance on the service. So the service was available. We could download artifact. We could publish some plugins, but some repositories were like the problem is because Jenkins is looking at the Maven repository to know if a new version is available. Nobody would receive a notification that a new version was published. And also because the way we build Docker images will fetch the latest version from the Maven repository. And so we could not build and publish the Docker image in time. So we were able to publish every other artifact, like the deep end package, readout and so on. So those were available. But the problem was for people with an existing Jenkins instance and for to build the Docker image. So what we did is because the security release is built in advance and store on the private Maven repository, we just created a virtual Maven repository in front of the release one. And that's a secret one, the one used by the security release. So that was the workarounds. I think I looked yesterday and now we were able to publish the repository. It appears that the service was degraded since the 24th of June. So yeah, it took us a while. So we had discussion with people from GFrog. One of the things is I still have to update the runbook. We know how we should have sent an email to support GFrog.org to create an issue on their site. So that's something. But obviously because it's a free service, we don't really have a city or whatever. So what I would like to be sure in advance is that we are notified for such maintenance. I mean, it's not a big deal to delay the security release. I mean, this is something that we would like to identify in the future. Because, yeah, security release involves a lot of different people. Daniel does a great job to do that coordination in advance. And yeah, this one was not something that we expected. This is the first time that this issue happened during the process. So yeah, we just want to be sure that it does not happen anymore. Any question? Is there a channel for GFrog to be informed about the upcoming incident that we could subscribe to? So that's a question that I asked. And I got no answer to that. So the problem is, we've been moved to the GFrog Cloud recently, several weeks ago. But because it's a sponsored instance, we don't have a GFrog account. So we don't receive the email. Sorry for the dog. That's interesting. I have two dogs right now. No problem. What do you think about starting by sending an email to support at GFrog.com to acknowledge that we use that address to contact them, since they asked, and use that initial email to ask for how do we know for the upcoming. So that will be an excellent support for the question. And it will show them a good example that we acknowledge their request. Yeah, that's a very good point. We should definitely send an email to support at GFrog.com to identify ways to identify ways to be notified. At the same time, I know that there are discussions with their VP of engineering or something like that to see how they could work with us. Because yeah, that's a pretty big instance for them. And it's a very critical instance for the Jenkins project as well. So yeah, definitely. Any other topic regarding GFrog? Sounds good. So the next topic is about trusted CI and CI stabilization. So this one, sorry, Mark. Damian, you put that topic to the agenda. Yeah, it's just to acknowledge that it seems that we did not have major incidents during the past two weeks. I was in holiday the last week, so I don't really know. Just confirm or tell me. But it seems like that all the fixes we did the past months are now stabilized instances. So since we don't have much more time to spend on this, unless we have a new something upcoming, but it seems like it's okay. Yeah, that was very nice. So nothing major to report last week. I mean, most of my attention was with the GFrog with the security release, but I accepted that. Yeah, that was pretty smooth. That means many thanks to everyone who contributed, including Daniel and team that are not there. But a lot of people put a lot of effort on this stabilization, which is a bunch of issues. So thank you, everyone, for your tips, issues working. The next topic, which is update CLI. So we identify an issue with the CLI since last Friday. So something changed on GitHub side. So I'm not able to authenticate with a GitHub. I mean, the way the way I wrote the CLI, I cannot authenticate with a GitHub API anymore for the GHCR. So we generate issues. And so right now, builds that need to fetch Docker image from the GHCR, Docker registry are failing. I have a fix coming, which should be released today or tomorrow. So that's very brief. But until there, just that you know that that affects the way we deploy Ham charts, because once a day, we also when we when we deploy and manage a cluster, we use Ham file. And part of the process is also to run update CLI to fetch newer, to fetch latest dependencies. And so because of the CLI, it's failing at the moment. Yeah, that affect one of the job. But most of them in for the rest of the day, because we don't run update CLI the rest of the day, it's fine. So it's right now, just some noise that I have to fix. The next topic is about Docker images. So the Jenkins, Jenkins Docker image. So in this time, team Jacob started working on how we build and publish Jenkins Docker images. So Jenkins slash Jenkins. So this time, it's not really directly related to the Jenkins infrastructure. But what's what team started here was to use a new feature provided by Docker, which is Docker predicts to improve the way we builds and published Docker images. So there is a PR that Damian put here, which is the Docker issues 1139. So I really encourage you to look at it. It's really improved the visualization of the number of combinations that we want to do. And it will also simplify how we build Docker image for multiple architecture. So the really nice thing is PR prepare the work for a lot of things coming. So we have a lot of exciting feature that will come. So one of them will be to build to use Docker predicts to build Docker images for PPC 64 S 396. But I really encourage you to look at that PR provide feedback because that will have an impact the way we build. One of the biggest advantage that we have right now is it drastically reduced the build time. That means that because we build Docker images in parallel, we can also add more. We can also add more Docker images. So that's really nice improvements. Anything to add Damian? No, we are also one kick on accelerating the test. Right now, we cleaned up, updated some, let's take our tasks. So don't hesitate to look at the issue tracker. Just one thing, it's not the pull request link you have there. It's an issue where team, the effort of listing everything, all the steps, each element of the tool list might be related to a subsequent pull request. The advantage is that we have a bunch of tiny pull requests that can be deployed to master really soon. So we can see the effect right now and it's for split work. So anyone interested, don't hesitate to discuss and comment on the scope of that issue. There's a lot of details. So that's interesting. The impact for the infrastructure, there is still a link. It's about the machines we propose for these architectures and the security associated that in particular using different patterns. So I'll let you read. But the impact is related to how do we use this virtual machine and sure they are not full in terms of hard drive, that they are up to date and how do we manage safely the credentials? Nothing else to add. The next topic is about some work that we are currently doing with Stuppets. So just to bring the context about why we are spending a little bit more time with Stuppets. As I mentioned last week, we, the Rackspace sponsoring and that's, and we now have to move to machine archive the Jenkins.io to another location. And we've been investigating either migrating that machine to scale way or to a record clouds. The reason why we want to bring a new cloud provider is because we have some credit there that we could use. And more importantly, we currently have, so the content that we have an archive that Jenkins.io is already replicated on Amazon. So we were thinking to move that to Azure or to another location. We decided to move with Oracle Cloud right now because we need some experimentation with the new harm infrastructure. And I mean, that's working great. That's not expensive because we are still in the free tier and we have one gigabyte of traffic, which is enough for a full back service. So the idea now is to deploy that machine on Oracle Cloud. But we have two constraints here. The first one is in order to use a harm machine, we need to run Ubuntu 20.04. And we are still running Ubuntu 18.04 on our infrastructure. So we need additional testing to be sure that everything is working. I mean, that's, we need to be sure that all the code can work with Ubuntu 20.04, which imply bumping some puppet modules. And the second thing is we also need to be sure that the base configuration also work on our architecture. So that's an interesting work, but that that forced us to go back to the puppet codes and update the dependencies and so on. While I'm testing and running the tests, I really realize how much time we haven't updated that kit repository. So we have quite a lot of updated dependencies. And yeah, it's like updating the let's encrypt module imply that you want, that you need to, to upgrade. I mean, we need to upgrade the firewall, let's encrypt module, we need to upgrade a lot of different puppet modules. And so we have, I mean, we have some work to do here. So that's what I'm doing right now. So most of my work is on the testing at the moment. I know that Damian has been working as well on the testing environment for the puppet code. So what Damian tried to do is to replace a vagrant with a virtual machine to replace that by Docker images. So it's faster to run the tests. And it's easier to replicate. Just the point of that, the goal is about the request I did along with Steve on the mailing list about do we need the staging branch or not. Because on one way the staging branch is slowing us down. On the other way, we need some tests to be sure that we don't break the puppet. The issue is that the master branch of that repository doesn't have a deploy step. As soon as the code is on the production branch, puppet pull it without the CI finishing or the test passing. That means we need the pull request to have some tests. So the idea is to use Docker instead of virtual box. So we can start spawning containers that will tests do integrate and when testing. And then we can decide if we remove staging or not. But we need a safety net. That's the goal of that part. Does anyone do any testing on staging anyway there? No. So the reason why historically we had the staging environment is because Tyler at the time was merging PR and staging and testing the staging branch. And usually on Friday he was merging the staging branch to the master branch. That's just for that. So the idea was to have a buffer. But I don't think that the staging branch still makes sense today. What we did at some point was to automate. So we were automatically create PR from staging to master branch. But yeah, I think we should just get rid of staging. Yeah. It seems we all agree on that. So I think I might proceed next week on that part. And here it's only adding more test harness to feel safer on the request itself. Yeah, definitely. Do we answer your question, Tim? Sorry. Yep. Thanks. Yeah. And so the reason of that is the last point I'm working on adding the agent as code for CI Jenkins. I'm still ongoing. I started and I didn't have time. So that will be my main focus for the next weeks. Next topic is about Terraform 1.0. I see that you bring the... So yeah, I think that's Damian who added that topic. So the idea would be to upgrade to Terraform 1.0. I mean, that doesn't make sense. I think we need to go one git repository at a time because we have Terraform code for Azure, for Amazon, for Datadog. So I don't know if you want to start working with one in particular, Damian, probably Amazon because that's the one you're the most familiar with. Yep. And because this is the Amazon and Datadog are the two one... We have a CI running. Even if the CI fails once per week, but still it's running automatically. So we need to start with this one. Datadog works without an issue. I've tested it already. And I have some work to do on the AWS. Might have one other request. If you start working with that Terraform, we should be sure that we have some automation in place to update on a regular basis when a new version is available because what I notice with Azure is that we bump the version and then we just leave the git repository there and we don't pre-test. And we should be sure that we identify when we need to upgrade the code and so on. Okay. Right now there is automated update of the latest patch version of Terraform. So we are using O.13.something and it's kept updated by update CLI. It's just there is a lock on the minor version. So the lock will be on 1.0.something. But yeah, that could be an improvement to have something to notify us. But the idea here is that 1.0 will freeze the syntax change. That's the premise of Terraform 1.0 for all the branch 1.0.something. That's something that the first prediction already. And there is LTS and that's there are discussing about. So the idea here would be to detect when a new major version is available and to run some tests just to be sure that it's working. Because in the case of the Azure codes, we have something to upgrade the code. And yeah, what I'm mentioning, the Azure code, I still would like to take some time to upgrade and to put it on track. One thing at a time. So I'm not sure who added. So I think we should put the name of the person who had a specific topic to the, so that's easier to have the context. So do you want to present that topic, Marc? I don't know if there's anything actually to present. So what we've got is a report that metadata signature checks are failing. When referencing the plugin, it's tool installer. So it's the tool installer check that's failing. But as far as I can see from inside the JSON data, it looks like there is no SHA-256. So there's no checks on at all. And I've got to do more investigation. I don't know that there's anything for this meeting. I was worried initially it was a systemic broad brush thing related to certificates or something. But as far as I can tell, it is not. I'll do more investigation and report it separately. So nothing else needed. Okay. Thanks for sharing that information. There is one last. So I have one topic that I would like to add to the agenda. I don't know if you saw. So Gavin started a discussion on discourse about should we replace Google Analytics by another tool. Last week or two weeks ago, we did some very quick experimentation with plausible. But yeah, I quickly run out of budgets on my accounts. And now Gavin started hosting Matomo too on his own infrastructure. I think that it would be better to move that service to the Jenkins infrastructure projects. That's one thing, especially because it rely on PostgreSQL database. And so we could deploy something managed on Azure or Amazon. We just have to, would be nice to just spend a little bit more time to evaluate other options like sponsoring for long term. Because what I fear with that kind of service is that you have some state, I mean, you generate some data, and that's interesting to have those data on long term. I know that, for example, in the case of Google Analytics from time to time, I just try to identify what's the trend for example, who's coming. I mean, the visitors sometimes are coming from Azure, sometimes they're coming from America and Europe and so on. And so it helped me to identify where to deploy service. And so if the idea is free to replace by something else than Google Analytics, then we should be sure that we can create data on long term. So that's all I want to say. So I really encourage you to look at the discourse. It's under the infrastructure. So it's under contributing infrastructure topic. And that's all I want to say. Any other topic that you want to bring? So I'm hesitant to sign up for more infrastructure like a PostgreSQL database hosted by us, but I think you're right. Let's have the conversation in that channel. So having more, let's say a database, one database is not like, I mean, that's $100 or something like that. So it's not really that expensive. So I'm not really, I don't really worry for that service. I mean, the RCI environment costs us a lot more, a lot more. So that's the thing. Yeah, it was less worried about cost and more about overhead of management, but understood the reason. Yeah, the reason what I was saying that's PostgreSQL database, you have managed service and they are really stable. So I mean, you deploy that and just use them. And that's really easy to use. An example is a rating application. You choose a PostgreSQL database since seven years, eight years. I mean, I don't remember when that service was put in place. But as long as we don't have to maintain the database, I'm fine with that. We are five minutes before the time. So any last topic that you want to bring? Otherwise, we still have the opportunity to start discussion on this course, as I mentioned, under the contributing infrastructure topic category three. And we still have the infrastructure mailing list if we want to discuss. And we are still on RCI. So if we don't have additional topic, I propose to stop the call here. Thanks, everybody, for your time. And goodbye.