 I'll try to do a better job this time. So hi everybody, welcome for this new Jenkins Emperor meeting. The purpose of this meeting, I mean, the number of topics that we want to discuss is pretty low. So basically, the main thing that I've been working on and with other people on the infraside was to migrate Jenkins agents that are running on Azure to Amazon. And we are facing few limits, few limitations, I mean, different behaviors. And so I would just try to mention all of them. So basically, one of the things that we did was to create a package image for Windows and Linux machines. So there is a GitHub repository called Jenkins infras-packer-images. So basically, the open to machine seems to be working kind of well, which is definitely not the case for the Windows machine. For them, just to go back to the open tools, an error that we are getting right now, that is not really clear is that after a while, the connection timeouts and the nodes is disconnected. I mean, consider us disconnected, even if the nodes is working well. No disk space issues, the connection is fine. No, yeah, everything seems to be working fine. And if we go back to Jenkins and just click relaunch the agent and it's working again for a few hours until it's disconnected. So that's one of the things that we are investigating. There is a GRI issues related to this. So I will just add it to the nodes. So if you're interested to follow what's happening there. On the VA? Yes. I realize you're doing project things, but I was worried first, should we first discuss the outage over the weekend? I was more concerned about the ci.jankins.io outage specifically or will you get there eventually? Is that on the agenda already? So I was getting there, but we can just, yes. We can get there, no, so long as we get there, that's great. But you're right, maybe you can just bring the context of what's happening on ci.jankins.io. So just everything is kind of related to the work that we're doing with Amazon. So because we provision instances for Amazon, we also reduce the number of machine that we provision on Azure. So that's one of the things. And because we reduce the amount of machine that we can provision on ci.jankins.io, we end up in a situation this weekend where we had no new nodes because open to machines were broken and we did not provision Azure instances. So that's one of the issue. The second issue is something that we investigate with Mark this afternoon. I mean, this morning for him. But basically what happened is for some reason, now when we try to deploy a new machine on the Azure account, it also provision a public IP. It also creates and attached on public IP and the limit of those public IPs is very low. It's 20 public IPs across the region, including ci.jankins.io, cert.ci, and all the other machine. And so for some reason, we did not have enough public IPs, so we were not able to provision new windows machine, open to machine. And so it was kind of also related to the outage that we had over the weekends. So basically what we did to fix it, we first asked Azure to increase the limits, even if I do not understand why we were affected today and we were not affected in the past because we've been using Azure for years now and we never had these issues in the past. So I don't know what changed. So maybe it's related to the fact we changed the Azure icon type, but yeah, right now I'm not sure. So basically what we did, we did two things. We asked Azure to increase the limits of public IPs that we could use. And at the same time, we updated the configuration to not use public IPs, but just use private IPs. But for that, we had to provision agents in a specific virtual network with specific security groups and so on. So right now, windows agents are working. So at least it's working for now, but the final goal is still to use Amazon infrastructure instead of Azure. So right now, we just have a fallback in case something is wrong with Amazon and then we can go back purchasing on Amazon. Any question with the Azure parts? No, all I asked a question in chat if it's time to switch to IPv6. And I think that's a... No, no, no. It was partially a joke. When configuration changed at a time and yeah. But and so now to switch back to Amazon, we have issues with Ubuntu and Amazon and Windows machines. So as I was saying for the Ubuntu, machines of MI are automated. So they're in MI automated with Sparker, those Sparker image, those MI are published. Sorry, so we can provision machine. We can use those machine. It's just that after why the agent is disconnected for some reason does not reconnect back to the master and so either we delete the machine or we just click on the button to just reattach the agent. But this is something that we are currently investigating and if you have some time to help with that, that would be nice because yeah, there is a behavior that I'm not sure to understand there and regarding Windows and regarding Windows, it's a little bit trickier because we created an MI for Windows with all the things that we need to occur, Java and so on. But there, what's happening is for some reason, Jenkins start to use a password to authenticate on the machine and that authentication does not work. And so we are still investigating with Alex to see what's the root cause there. But normally we create a security group with the right port open. So it's either an issue with the way we built the precary night or just ongoing physically. So and it looks like I just checked and it looks like there's a minor configuration issue on the Windows images. The tests are passing now that we're failing over the weekend, so I'm delighted at that. That's really great. It looks like there's a minor configuration issue that Git LFS is not installed but command line Git is. I'll work with Alex Earl on that one. These are the Azure images. So I probably will also adapt the tests that depend on Git LFS to not depend on Git LFS. So if you're spending some time with that, it would be nice maybe to open a georatica so we can try this work. The second thing is it should be pretty straightforward to fix if it's for the Azure images. There are some, there's an eight script that you can update here. So it's probably just installed Git LFS there. I just put the link of the file in the chat. Perfect. I just somehow or other Git is there but Git LFS is not and I just got to work with Alex. It's usually a part of Windows, of command line Windows on Git. Yeah, so it's probably just for Windows. It's probably the script that needs to be updates. There is just one thing that I forgot to mention. Last week we discussed about having harm 65 architecture on the Amazon. And so I also created an EMI for it. So if we use the label harm 64, we can start testing whatever we need to test on that specific architecture. I still have to write a blog post or at least somebody made always more information with the different issues and how we solve them. But yeah, it's hard to stay focused in the current app periods with the current activity. I wouldn't mention something else. Harm 64, harm 64. No, any other question on cedegenkin.io. So yeah, there's something that I want to say. The reason why we're first for support using an agent and we are not planning to move the master is because we don't have any repository with some to our home to configure the Amazon account. So this means that all modification right now happens manually. So this is me doing that in the Amazon accounts. And so I did, I would like to take some time to bootstrap a project similar to cedegenkin.io. Where we can automate the configuration of the Amazon accounts. And so right now we don't have a lot of configuration. It's mainly I created the credential for provisioning institute instances. So we use that credential from cedegenkin.io I created the right policy. So that credential only have access to a region and some. But when I have more time, it would be nice to, for example, maybe work provisionally cluster work with Spark gates or whatever. I mean, really use the Amazon account more than what just provision in the situ instances. But as I was saying, it's step at a time. And so first for just to have into machine and Windows machine working reliably on Amazon. Any other question? Nope. The last part that I wanted to mention is that I'm planning to take some weeks off during the next summer. So if you have any work pending work depending on me, that's probably the right time to plan it to see how we can do some knowledge transfer or whatever. Even if I'll be available, I'm not planning to stay out of my computer during the next summer, so. Yeah. It would be great if we could ensure that by the time you go off, we as infrastructure can help. It's basically everything. So that we don't have one person bus factor. A lot of the factor, whatever you prefer, but it would be helpful for everyone. So basically, the way I see it is that if you're interested by a specific parts, that's probably the right time to ask more permission. So I'm maybe asked to have the knowledge to work on that specific area. Obviously, depending on what you are interested, if you want to help with a specific, I mean, let's say that you need specific credentials or whatever. We'll have to, because I mean, we have to decide if we trust you enough to give you the right permission or if there is enough people working on this. I mean, it's difficult to say, but while we try to be as transparent as possible, I cannot give root access to the machine to the first person who asked. Yeah. That's perfectly fine. So the idea is really like, if you already work on a specific area, whatever, maybe that's the right time to move to the next step. For me, the main question is there, where we have no contributors at the moment. So for example, for me, it's less of a problem whether I'm interested or not in a particular area. It's a rather a problem, what to do if something goes wrong there. And for me, reaching out to you would be a last resort. And before that, if we have a list of these gaps or whatever, probably we could spend some time towards the summer to reduce the number of them. So we don't disturb you. Okay. So this is something, I mean, we still have the time before summer, but I just want to bring this in an enough in advance so we can keep this in mind. So I was having a conversation with somebody not long ago about the Datadog monitoring that we're doing and just saw an email yesterday that highlighted that it had the Datadog had in fact detected the outage that we had over the weekend. I just didn't look at it and didn't see it. So I was really delighted. It has a monitor already for some of the things we were having failures on over the weekend. This is thus reminding me, I've got lots still to learn about infra. So typically the errors that was reported by Datadog is the size of the build queue, the job queue. And this was a check that I put in place like something in September, something that is just a way to know if something is wrong. So I think like if we have more than 180 time in the job queue means that something is wrong somewhere, but yeah. Yeah, would it be possible to create a list of errors where there is no other contributors who could help at the moment? The thing is difficult to say because we already have a rent book. So if something, so something that I was thinking maybe I can improve is inside the alert, inside the error that is written by Datadog is directly a link to the right rent book. But we also have issues that are not necessarily document and that is not necessarily easy to document. So for example, we had an issues over the weekend about a JIRA. So basically JIRA ran out of disk space. And so we just had to SSH on the machine. But because that machine does not have a lot of disk space, there are sometimes different. So for example, last Friday I cleaned up some old logs that were not useful this Monday. I just deleted all Docker images and stuff like that. So it's more like there are bugs, there are errors that we can easily document and so say do this. But most of the time it's more like you go on a machine and try to understand what's happening and then investigate how you can fix it. And that's where it's difficult to document. Well, I'm not talking about document and how to fix that, rather document and where the gaps in terms of ownership and access. Because yeah, documentation on run books is definitely important. But first step would be to understand the way we miss people. And after that we can probably see what we could do. For example, me and Mark could dedicate some time to study this area, maybe do some knowledge transfers with you, record, or not create run books. But yeah, first item first would be to understand what are these areas. Okay. So could the idea of using the run books as a first, which of the run books are already hot or a topic related to the run book is hot or is low coverage to use Oleg's idea? I like that idea a bunch. Oh, sorry. Personally, I really like the idea that the way you said the ownership gap. The idea is really to identify what are the things that I'm the only one. So a small example is Amazon account that you are running at the moment for some reason. We can only have clubbies employees. This is something that you have to fix. This is something, this is an account that needs to be transferred to the community. But for now, the gap is if you're not a clubbies employee, you don't have access to the log. So this is definitely a place where, for example, Mark and Oleg could help. Where, for example, CLE Jenkins.io does not mean this is the only thing. CLE Jenkins.io, for example, we could have more people. The Azure account is the same. We could have more people. So typically for the case of the Azure account, I was maybe thinking to add Tim Jacob on that part because he has an Azure to not break the things. So yeah, I really like the idea that there is of having people with specific ownership on specific areas. So I try to give a list and send that online from the list. Any other topic that you want to? Yeah, I put some topics to the list. They're just in the proper, such as the changes section above identify ownership gap, somebody ideas. I'm checking this card. So yeah, regarding the approving custom auth app, there's something that I would like to do is to enable the GitHub accounts. I would like to restrict third parties access. And I'm not sure if it will impact the custom auth app. So, yeah. So in my case, I thought the word specifically about Jenkins CI GitHub organization and about how use cases core change generation because the current situation that if you open Jenkins CI Jenkins releases, you can see that GitHub actions is listed as an account which cut the release. And I would like to fix that. I already put type of fix which basically replaces the GitHub app by another auth app. I can do that. There is no problem with the STPI limit, et cetera. But I need approval or whatever it's enough from other interacting members to do that because there may be concerns I didn't consider so far. So I just have a question. Would it be easier to run the release drafter for Jenkins? Or is it something that could be possible? Well, technically it's possible. Practically, it's not only about the release drafter. Right now we use GitHub actions and we change several lessons in order to generate change logs. And this is the current implementation. Obviously, we could move everything to Jenkins. But firstly, it would create additional STPI consumption. And in the current state, I would rather prefer to avoid that. If there is strong requirement to run a change log generation from Jenkins, of course we can do that. So anyways, in terms of permission and Jenkins here organization, you probably have to contact Daniel for this because I don't have the keys there. I have everything I need to configure the organization. I just need to send off from the team that there is no problem with it. I think Olivier, what Oleg is looking for is agreement, I agree, yes, you should have your authorized to do it, Oleg. I think I already replied to that in the email, so for me it's no? No, you didn't. That's why I edited this, the agenda. So maybe my email was not sent, but yeah, for me I thought that I already replied to this. Yeah, you just, I was wrong. You said that you need to update templates. So yeah, sorry I forgot about this message, but it just didn't seem as yes or no regarding the old app. No, for me it's fine, we can go ahead. Okay, so I will just implement that. And so there is one, there is another point that you asked. There is another, so yeah, under Outs, it's totally fine. The other day on the RSE, you also asked if it was possible to enable GitHub issues for Jenkins Infra, GitHub repository, so the perpetrator repository, and yeah, not just something, I mean, I don't have strong opinions about using GitHub issues, it's just that I don't want to have to, I does not like yet to have multiple tools to switch, because I mean, it makes it more difficult to generate reports, it makes it more difficult to find the right information and it's not young. So the thing is, there is that discussion about using Gira, getting rid of Gira or whatever, or fixing Gira, but this is something that we still have to work on. Right. So if we don't want to use two trackers, the main problem is issue migration. It's possible to make great issues to GitHub, but sometimes it's just an overhead. Practically what I've seen in other repositories, once you enable GitHub issues, the traffic to Jenkins Gira basically vanishes. So it would be an archive for historical issues, but I wouldn't expect too many activities there. Okay. So I'm not really concerned about duplication, especially if we say that, okay, now GitHub issues is a source of truth for new issues. No, yeah, but my point about duplication was more, for example, okay, you have GitHub issues for the pipette repository, but not for the VPN, for the Docker image and stuff like this. And on the infra, we have a lot of repositories. So my point was more either we enable for all the infra work or we don't, because I don't want to have, when I start working or when I start to look at this, I don't want to have to open two tools, and I don't want to keep track of the two tools. It's already difficult enough to follow all the notification and all the things that are happening. So... So for me, it's quite straightforward. If it's a user-facing repository where we can facilitate contributions of feedback by using GitHub issues, it would be in favor of using GitHub issues for repositories which are backend and which are not really visible to users. I don't care. And I'm fine with using Jenkins Jira. But I like the statement that you said, do not care, because that means we could switch on Jira issues for all of Jenkins infra as suggested by Olivia and not violate the constraint you gave Oleg. And I am all for turning on GitHub issues. The experience with Jenkins.io for GitHub issues has been positive. I think it's been very good. And I think we would benefit by turning on, by enabling GitHub issues. So as long as we don't have to migrate out of GitHub, it's good. Migrating out of GitHub would be so big issue. Yeah. But yeah, I don't think we need, I mean, I don't really care about migrating all the issues from Jira to GitHub, because if it's really an issues that come back, then we can just reopen the issues and just put a link to the one from Jira and that's fine. And anyway, we have a lot of old issues that are not relevant anymore because we thrive old and the project is old and some, so just in different ways to start from zero. But yeah, for me just like, except that for security work, everything is public anyway. So I'm personally, I'm fine to use GitHub issues. It's just that it's a different workflow and something that I try to think about. So with regard to GitHub issues, I gather that we're not settled on a decision yet, but it's an open topic. Do we have? To me, for me it's an open, it's still an open topic. For me it's still an open topic. I'm not sure yet, but I mean, there are so many people that are still strongly in favor of Jira that I don't want to force a move in either ways because for example, we'd say that I know that Daniel rely a lot on Jira and because he created, this is for the same reason that me created dashboard. And so if we decide to switch everything to GitHub issues, we just to re-implement some work that we did for the projects. So for example, in the case of Jira, yes, I have dashboard to say who's working on what are the most critical components that we have and things like that. So it's a way for me to say, okay, I spend a lot of time on Seattle Jenkins.io instead of trusted or whatever things and so if I decide to switch everything to GitHub issues, then yeah, I have to also change this, which is not a big deal, but yeah, I mean also some work and changing my process. So yeah, for me it's still an open, but I would like to have that discussion because we need, we have to find a solution before September, October, or November. I don't remember, but there is a deadline for Jira because the current state is Jira need to be upgraded. We need to work on the database. And so yeah, just to remind the current states, Jira is using a deprecated my script or post-created version. And so the new Jira version do that supports the database that we have. So we have to upgrade the database. We have to update the collision on the database. So there is some work with the Jira. Then you have to upgrade Jira. And so this will require some work. And if we don't do that before November, we won't have fixes for new vulnerability. And we know that Jira is a popular target instance. So we always find the buzz that we try to be as much as possible updated. So anyway, that's not different topics, but yeah, we have to keep this discussion open. Okay. Yeah. Specifically I have questions about the update center because currently we have issues with plug-in site. So plug-in site itself uses GitHub issues right now. The update center, which is in many means a part of the update site uses Jenkins Jira and it complicates everything. And it's a public-facing repository. So for example, if a plug-in maintainer wants to replace the plug-in or something like that, now they have to use Jenkins Jira infra project. But moving to the request management that to GitHub would be reasonable. So who is the person who solved most of the issues regularly related to the update center? It depends. Security mostly Daniel and Vardek nowadays regarding the rest, whoever handles them. So my guess would be to convince them to switch to GitHub issues. So personally, I don't spend a lot of time. I mean, this is not a project that I'm tracking. So for me switching that project to GitHub issues is fine, but here you have to convince them for that. Yeah. Another similar topic is redirect for Viki because you're right now redirects for Viki. They are managed in Jenkins Jira and they believe we are not open to open and get help issues in Jenkins infra slash Jenkins Jira. So you want to use GitHub issues and that's why you wanted to enable GitHub issues and Jenkins Jira slash Jenkins Jira? I have, I want to enable it in some particular repositories. I don't want to enable it globally. I believe that it's up to maintainers of particular components, but yeah, I'm just exploring some components where we potentially have problems or where we can potentially get benefits from using GitHub issues. Because typically enabling GitHub issues just for this, that's what I mean by some issues will be reported in Jenkins Jira and others not and that's why I'm not really in favor of enabling it. But yeah, if other people think that I'm wrong, I mean, I'm totally fine to enable it, just that this is my concern. This project is not only used for playing for Viki redirect rules. Yeah, that's right. So in the case of Viki redirects, I understand that it's unlikely that we enable GitHub issues there. For me, it's just a potential problem, but I think that our guidelines will just request a making a pull request. It's easier than submitting an input. Because do we really have a lot of contributors that do not have Jenkins accounts? I believe that everyone who contributes seriously has a Jenkins account. So you can write the blog posts and other stuff without it, but in principle, I can create it in a few minutes. Because it's quick to create, you have an access and I just mean, yeah. I don't think the concern actually is about account access. It's rather that, at least for me, GitHub issues are attractive because they're much closer to and very precise to the component that I'm doing work on. But that admittedly does sacrifice many of the benefits we get from JIRA. So I don't think only you're proposing a global switch, rather one repository at a time and only when that repository makes sense and its contributors agree that it makes sense. Exactly. So the specific when you were, I wonder do we need a process by which we say who are the contributors to this repository and how do we get them to unify that we're okay switching this one. I took the preemptive choice to switch Jenkins.io to use GitHub issues. I enabled it because I happened to be an administrator and I didn't consult anybody. I just did it. The danger of course is that now we could get bug reports in both places. I accepted that danger is very much worth it. Yeah. And plus one with regards to it and we can move the entire website project to GitHub issues if somebody has time to do that. So maybe Mark, if you're interested to experiment with issue migration, you have a playground for this. Yeah, which I'm actually not. I will happily work to close out the issues on JIRA that are there and accept new issues as part of the transition. I don't feel any need to have a single repository for that particular project. Now there are others. I'm unwilling to enable GitHub issues on the Git plugin because there are 500 issues in JIRA that I will not move, right? So it's just different. For me, it's per repository is where the decision is. Yeah, right. So I think we are running in circles a bit. So definitely it's an open topic. It's not something we're going to decide today. Yeah, anyway, because we still have to think about it. So yeah, I propose that we stop the meeting here. We are already over time. So thanks for participating in the meeting. And yeah, see you later. Hanar, see you later. Bye-bye. Bye.