 Hi everybody. Welcome for this new Jenkins infrastructure meeting. We didn't have one for several weeks now, so we have quite a lot to cover during this session. So the first one is I want to remind so it's I'm not sure if it's an announcement or notes, but we have the October fest coming pretty soon. And the Jenkins infrastructure project is definitely a good candidate to receive contribution from people. So if you have some ideas about small tasks that could improve the infrastructure, feel free to either open a small ticket or a GitHub issues, if it makes sense. And I'm planning to work on that in the coming week so we could have as many small tickets that we could that we could share. I'm going to just move this to the notes. So next thing that I want to share with you. Anton from the Linux Foundation will do a maintenance on JIRA database. So he has to change the charts in order for Unicode to work in issues, comments and description. So the service will be done for two hours maximum. But yeah, we never really know for sure with with JIRA, but so that that maintenance will happen on the 5th of October 2021, which is the day before the next stable release will do it will do the maintenance at 4pm UTC. So it should not affect us too much. But that's something to keep in mind and if you want to move the maintenance to another to another time feel free to raise your voice. Yep, difficult to take note and to talk at the same time. Next topic unless you have a question regarding JIRA maintenance. So I mean it's not it's not a big Mark if you can take the notes that would be that would simplify me a lot here. Thank you. So the next topic that I want to cover briefly cover because it's not all about infrastructure, but we have the election coming so I wrote a blog post to announce the beginning of the election. So I had a bunch of discussion about community the Jenkins that I know community the Jenkins that I know. So I just took all the feedback there and write a blog post so I would like to publish that blog post and Monday. So we officially start the election process. If you have any feedback, please do there. So we can can adjust. So I put to link one real link to the PR to publish a blog post and the second one I created a small tool that I named EFT for email from discourse. So it just a small tool to collect to retrieve email addresses from members of a specific group. So the idea is, when people join a specific group, they agree to participate to the election, the register to the election. And then we'll use and then we'll fetch email addresses for from that group. So we can send them invitation to participate on the condor say voting system service. But everything is explained in the blog post so that that's it. The small tool. I mean, I don't think it will evolve a lot once the election is over, but maybe useful for different purposes. So that's why I created that I created it. Next topic is about key cloak. So we discover so due to the confidence security issues that affected us two weeks ago. We asked people to reset the password by using quick cloak so beta that accounts the Jenkins that I own and we identify several issues were with the service. One of them is, it's using send great to send email. But the problem is, sorry, we send greats, we are using the default plan, the cheapest one, which is where we reuse the same IP than other sacred clients, and sometimes the reputation of the IP is quite poor. And so our messages are emails are considered as spam by some some some some some some mailbox. So we have to find and we have we need to find an improvement regarding the same group plan. And the second annoying thing with that same great that counts is we are limited to only two people who can have access to it. Which is at the moment because okay and me and so ideally I would like to have more people are at least all the people who can help with the Jenkins and fry and email issues team jack on earlier today made a very good suggestions which is to integrate send greats with our Azure accounts. In that case, we will be built directly from Azure. And that will not easily affect the Azure invoice. So that's really nice. And also we would be able to use Azure and Azure active directory for authentication so more people will be able to access the same with accounts. So that's something that I'm planning to work on in the coming days, weeks. So to come back to the key clock. So that was one of the shoe emails sent from key clock. The second one we identify some issues some integration with that which is a service that contain the user and passwords. And we still have some improvement to do there. I'm not sure what's the current state because people are still affected by it. But what is in my case is sometime we have a very, we have time out issues when we try to update user from key clock. But that's something that I need to investigate. Any question. So yeah, I'm the same grid points. I covered it during the key clock topic. So the next topic is about Azure. So the quick updates on Azure, we successfully reduce the cost of the Azure accounts. The last invoice was something like $7,000, which is really nice. And I would like us to continue on that bus. So we end the year with a good balance because the previous months we were above $10,000 per month, which is the limit that the CDF asked us to keep. And so now the idea is to keep that number low. Olivier, do we have a corresponding dramatic increase in AWS spend, given that AWS donated 60K, have we consumed all those funds? So we haven't consumed those funds yet, but we definitely increase the cost on the Amazon accounts. We have another project that I want to cover. The green here, we receive the budget from digitalization. So we now have the money for digitalization and we would need to configure CI, the Jenkins that you have to use that project. And I think that would be really nice to have that for October Fest because during October, because we have more contribution, we have more pull requests, because we have more pull requests, we have more loads on our CI infrastructure. So that's something that we need to work on. Digitalization allow us to use a small communities cluster with three nodes. We still have room for improvement there. So if we can justify more budget on digitalization, I think we may, we can, yeah. That was just a first estimation when we talk with digitalization. So definitely the Azure account cost reduce, the Amazon one increased, not as much as Azure one, I think, and we still have some fine tuning to do, like the size of the machine, the number of the machine and so on. At the moment, that's something that I have to double check with Timon, but the limit at the moment is should be quite high with Amazon accounts. But that's something that we may reduce. But yeah, that's just a number that we have to adjust. Thank you. Next topic is about October Fest. So as I mentioned, we have that coming in October. If you have some ideas, feel free to share them with us. Otherwise, if you want to help the Jenkins project, that means specifically Jenkins infra, we have some ongoing discussion happening on community Jenkins that I also that's the place to go. If you want to coordinate effort there. The next topic is about backer images. I know that Debian has been working on that. Unfortunately, it's not there to do an update. I would not risk myself to explain all the things there. So I guess, yeah. So I guess it's just better to keep that topic for next week. Digitization, we briefly go over that one. So I propose to just continue to the wiki exports. So as you know, two weeks ago we were affected by. Olivier, could you could you scroll your screen up so that we've got. Sure. Thanks. Thanks for the reminder, Mark. So, as I was saying, two weeks ago we were affected by your major security shoes. And that's on service that we did not really maintain for years now most of the data that that we have there is quite old, but we did not really find the time to be great. The content elsewhere. So the current state of the machine now. So our VC stop confluence is stopped. We we restore we need a copy to another location, a machine, a bigger machine that that we can do some experiments. So now the purpose is to to find the best way to extract all the contents that we have on the machine and then to to migrate that content to the right location. So giving has been doing some work on that topic so he created a git repository named Jenkins infrastructure slash plugins dash wiki dash docs. So it's exported plugin documentation on that git repository that was not yet exported. But we still have so the plugin documentation issue is still it's getting solved now. But we still have a lot of other pages on confluence such as meeting notes, like first time organization contributors and it's in quite a lot of information. So that we don't necessarily want to lose them. So I've been trying to export confluence pages to HTML so that's that there is a tool directly in confidence that we can use for that. Let me let me show you something. So at the moment I exported one of the contents. So the only thing that I have at this time is the exported files don't necessarily match what you see so you should see my screen here. So the CSS is slightly different than what you have on wiki URL URL as well. But in this case you can see that we have a small checksum page and with that HTML. So we may have some cleanup to do so what I was suggesting was just to export or every pages to HTML pages and then upload them to archive the Jenkins. So we would have archive the Jenkins.io slash wiki and we would just export all the contents by default there. So that's a suggestion if you have a better idea. I'm really hoping to suggestions at the moment. But what I know for sure is we won't put back we won't bring back confluence. Any question. So I'm still hoping that we could get some way of easily doing redirects from wiki Jenkins.io URLs to archives Jenkins.io. Given the URLs in in the base it looks promising, even if we dropped the slash wiki, but I'll take it up with you separately Olivia I think you're on the right track we need to extract the HTML so that we don't lose the HTML. Okay yeah we can work on that together. Next topic which is about Artifactory issues. Oh so actually since Damien has joined us maybe we could spring back the topic on on the packer images. Maybe sorry it's already there. Yeah that's the last topic. Let's proceed on Artifactory. So, so the Artifactory topic was mine that I put in. I see that we had a failure to download an artifact from repose Jenkins.ci.org. Is there a way we would like those flagged should I be. Should I raise an issue. What what and I've seen these before this is not a completely isolated insulin incident. How would you like them reported what's the best way to do it. So the question in this case is we are not. So do the file is supposed to be there. Yes, yeah definitely. Because something that that affected us in the past, like several years ago, we had the same issue where we were not allowed to do that amount of requests. So we put in place a report Jenkins.io so we put in place at that time a proxy cache on the CI environment. So maybe that's something that we could do again, especially in the, especially considering that. But that's maybe something that I have to investigate with Damien. Because everything is running on a community cluster so we could deploy a cache local to the to the agents. Because that's definitely the problem here. It's really clean and such solution to be quite honest, because the, the combination of management, the time to spend on managing such an instance. The IO that it requires the costs in terms of infrastructure and the amount of problem that it can cause due to caching to the developers because you're that's a special area where the developers cannot control the cash and cash. Which means unless we've, yeah, I don't know, I had a pretty bad experience and cooperation before on that one. So I would prefer to see if there are other alternative solution but the goal is to make the developers the users pretty autonomous and this remove this from them because they won't be able unless they bypass the cache. So that's something, sorry, but yeah, Tim, go. As I say, we've just ran an artifact tree on our CI cluster for the last three years. And we're going to need to touch it like five times in three years in general. So we used to, we used to have been treas are upstream but we haven't been around reliability issues. And so we just deployed an artifact tree on our CI cluster and it's been pretty much completely hands off and just work. Yeah, that's my suggestions otherwise something that I did in the past was to deploy an engine proxy cache on the cluster as well. So it's pure caching in that case. But the problem in here is we don't really control. So we can open, we can open issues with two frog. But at that instance is fully sponsored. So we don't have any essay with them. Yeah, fair. It's just that we just, we just exited a CV issue on something that we dumped somewhere and didn't touch for free for a few years. So honestly, yeah, right now it feels that we are overwhelmed by the amount of tasks. So yeah, I would prefer asking G frog to increase their SLA or maybe consider another service or consider our only artifact or a from scratch without the help of G frog. But yeah, a mix of these solutions seems the worst case to me. Because if we have to pay for an instance we have to do. Let's have one that we master from scratch. In the case of team superposition is just to have an artifact tree, which would just work as a proxy cache. So that would just reduce the amount of connection but yeah, I mean, maybe maybe this is something that we need to discuss with different book. Yeah, that would be an idea also to check with Daniel and the security people, because maybe the argument for a spawning instance will be yeah, we need to have our tiny artefactories as a service inside the infra, especially for the release priorities in order to build them. So if we have to build this, then that means we will have to to spawn two free instances so in that case the cost will be worth it. Okay, don't think there's any need for anything like that if we just keep us if it's just pure proxy. So keep repo as canonical and then this is just a CI proxy. It's not used for anything else. Yeah, the instance in itself will be separated. That's correct. But the amount of work automation or action to use it. If we have to reuse it two or three times that it start to be worth the effort if we only have one instance then the question still is kept. And the problem if you have one instance is in the current situation we have some agent on Azure summit and less unusual now but we have agent on Amazon. And we're going to have agent on digitization soon. So the suggestion here is just to build other current image that is just a proxy cash with release. I think in Seattle org slash release no configuration no authentication just read only. And so we would deploy it on digitization and Amazon and elsewhere if we end up using a different cloud provider. So if if it's just a proxy cash and don't think, but I agree. I don't think it will be. Again, it's not that easy. How do you uncash. I think that it's synchronized with the upstream system. That's, that's why G frog and an ex sorry, are building their business. I mean it's not as easy as having because the content is not immutable. The content especially the snapshot builds or the snapshots the meta that has always changing right. Only your engine is not sure. Yeah, we don't want to use such snapshots there. So it is. Don't we. No, no use incrementals in general, which is a which is a which is like a release but not in the official release repo. And those are immutable. Okay, so if it's immutable then can make sense but. Yeah, again that's I mean I know I know I saw some discussion with Jesse as well because that's a recurring topic. I mean, we are not the only one to have that kind of issues. So, yeah, I think that would be a nice nice project to to investigate the different options that we can have and which one we take is just the problem is there. If we have to do it. Let's do it for multiple use for the benefits of the end users and not only for the benefits of saying we want better performances. I mean, if it happens one or twice a year the issue on G frog that's totally was it not wasting time on maintaining something else or paying for something else. We have too much services right now that we are. I mean, there are more production on QA issues that we should focus on, unless it really solve, it really brings more values for the end users that's what I mean, we just have to balance correctly. Yeah, I don't, it's, I haven't seen it too much recently there's certainly been off and on quite a lot of issues ever since that credit us to the new platform, just random instability. But I think we're having more problems with like core builds and remoting builds, the tests. A lot of random issues happening, possibly since we moved to the communities agents. That's kind of hard because they're having different issues on ACI. So they are not, they are not prone to security issues. Not choosing ACI. Okay, perfect. Thanks for your insights. Any last comments on the report of Jenkins yet or We have a written discussion on this course or about this topic. I remember. So I saw that that's that's a discussion. I saw this question happening on Jenkins from many lists Jenkins that many that's that's a record issue because we face the same issue like three or four years ago. And then we realized that I was not able to handle the load. And then we realized that after a while, the proxy cash performance was bad compared to the improvement that she front did. So we decided to remove the proxy cash at some point. And maybe the situation will just improve on G for excited because, yeah, as we mentioned, they've been created our service to a new platform so maybe does the time for them to do better control the I mean to better size the infrastructure below the service. So we can just wait few weeks and see how it evolves. We have two minutes left before the end of the meeting so I propose that we quickly cover, quickly cover the last topic, which is a packer images now they mean that you are here. Review and review my last pair that should enable automatic upgrade of the VM on CI Jenkins IO. If it works then it will be automatically opening a pull request with the latest API builds today. And once the second pull request is merged and it will update CI Jenkins IO. So let's test the full end to end. Then next week. I'll focus on the team feedbacks. So Jenkins here. Jenkins in front of the one on the one. Yep. Jenkins in front. Oh, yes. Sorry. Exactly. And the second thing is that Windows Server on Azure is the worst. I've counted 11 different errors this week that I had to fix. Some are random, some are less random. That's a nightmare. I don't know what other feedback. Yep. Which we are able to make. Yes, this one. Yeah, perfect. I've tried to write down as much information as possible. It's me. Put the link to the PR here. I tried to find some time to review it. Yeah, let's cover this next week. I already put a link to next week. Jenkins and from meeting. So if you have to be topic you want to cover, oops, that's not what I want. So if I can right here, I will probably miss next week's meeting. I'll be in Alaska. Okay. That's a good excuse. I just put that into the next week meeting at the bottom of this page. So feel free to add some to be there. So we are a little bit over time. So thanks everybody for your time. And have a great weekend.