 Hi everybody, welcome for this new Jenkins infrastructure meeting and today we're going to talk about the LDAP OTAGE that happened last week and some ideas about how we could manage identity management. So first, let's talk about the LDAP OTAGE, I mean, I guess you see basically what happened. So two weeks ago we had an issue with the Kubernetes cluster where we had to recreate everything from scratch and we discovered that our LDAP database was not back up since February. So the first step was to restore the database then we realized that people were not recreated in the process. Those icons were available for someone else, which means that it was a big deal for, especially for plugin maintainers, for recent plugin maintainers. And so the first step was to first recover every account that was left. So we first focused on maintainers, then we used JIRA and Artifactory Database to restore those data. And once all user accounts were recreated, we tried to reset every password. And for maintainers, it was quite easy. It was a totally different scenario for the whole database. The main reason to this is because we have a lot of accounts in our database, we also have a lot of garbage in the database. And sending 100,000 email address in one day is not something easy, especially when we usually only send something like around 500 per month. So it was quite challenging. And this process also highlighted a few limitations with our account app, which is the application that we use to create, modify and delete accounts in our LDAP database. So now that we restore and we are back to the previous situation, the idea is to identify how we could prevent this to happen again in the future and what would be the different options. So last yesterday, we had a discussion with the Linux Foundation, where Tracy and Alec were also in a discussion. And so the idea was to see with the Linux Foundation infra team what would be the option if we could use, if we could delegate that responsibility of identity to the Linux Foundation instead of managing it by ourselves. And also in the discussion, they also remind us, maybe not remind us, but tell us that told us that we could also, they could also host our JIRA instance on the Linux Foundation. So basically everything we discussed for around one hour about why we would use the Linux Foundation identity management, what would be the limitation, and what would be the different services that we would move to the Linux Foundation. So first, any question until now? You really want to add something on this? So yeah, so then I continue. So regarding the identity management, yeah, someone who is Tracy, we can't hear you, Tracy. Better. Yeah, I've just put in the chat the link to the document where I just want to capture everything with the identity management and the transition. So it's work in progress, but I'll keep updating it with this conversation. And if folks want to go in and comment or use that as just background to what we're trying to do, just put the link to the chat in the notes. Thanks. The thing is, at this moment, we are really open to multiple solutions. So either we totally move identity management to the Linux Foundation, or we find a way to remove the account app. This morning, I was playing a little bit with a tool called Key Cloaks. Basically, it's a tool that seems to be quite easy to deploy on our community's cluster. And so you can have user federations. So we could, for example, use it. So we could, for example, reuse that database and use Key Cloaks to create user in the database, modify user, delete users. And what we can also use it for is to, for example, have it in front of the database. So while we are migrating to a different solution, the Key Cloak could be the central piece for a while. It seems to be supporting a lot of features like user requests and accounts. They can have to reset the passwords and an email. I mean, like all the features that you would expect from an identity management today. But yeah, that's one of the options. The other option would be to see if we really need it because the main reason why we introduced that in the infrastructure was to use it with Tira. But I do it through the byte on my side. Sorry. Is it the same for everybody? Yeah. Sorry. So yeah, we are really at the moment where we are trying to find, identify all the different options. So I really encourage you to look at the notes and put some comments there. Yeah, it sounds a lot of them. Yeah. It sounds okay for you. So I guess we can continue. Otherwise, we don't have, I mean, the main focus were on the account management stuff. So if you don't want to ask any question, I can just continue on the last topic that I want to talk today. I think everybody wants to add anything on this. Well, so in terms of Linux Foundation as a potential ultimate. I assume that's a long horizon because there's a lot of work to transition 100,000 accounts to anything. So the first thing in the discussion was why they can provide, they can provide the identity management. We would not have the control on it, so it would be a black box. So basically, a user just created a Linux account and the Linux Foundation accounts to access Kubernetes project, to access any other project, and it would be the same for the Jenkins projects. So one of the challenge here, first will be to map the Jenkins account with the Linux Foundation because there is a big probability that your username is already taken by someone else in the Linux Foundation. That's one of the things. The other thing is until today, the Linux Foundation always provided, always used identity management for their own services. So they don't know at the moment how we would use those services. And it will not necessarily be easy to create groups and change groups, member and stuff like this. We don't think it's a big deal because we do not create, we don't regularly create groups, except the default one to access Tira and maybe for, let's say, artifactory, but otherwise, this is not something that's regularly changed. But I do not expect it to be like a quick win where we're just working it three days and it's done because that is used in a lot of places. So the biggest challenge I think will be to see how we do for Tira. So if they host Tira, then it's easy because we don't have to handle, we don't have to care for the identity. We just use our identity and that's it. And they provide us the Tira. And if we don't want to use Tira anymore, for some reason, maybe there is other options. And also, they also have experience with their own identity management to manage chief work. So this is artifactory. So this is also something that they already did in the past. So I don't, we plan another meeting with them next Monday. If you have any questions, feel free to ask. So we don't know yet at the moment how much energy we have to put in this project. I'm particularly interested in the Tira thing because if I understand correctly, we need to confront a Tira upgrade within the next six or nine months. We've got a Tira version that we need to update anyway. So good excuse to consider hosting that by Linux Foundation. Great. So that's definitely, so the discussion was just a reminder. So right now we have three options, either we keep maintaining Tira by ourself and then we have to upgrade the database, we have to upgrade Tira. And so it will require some work from us, either we move to Tira clouds, but there we have a bunch of limitation, limitation that I identify our Tira, you know, our Gira, so there is a ticket with that, but the main issues is the number of people who can look in on Tira. We cannot use adapt to a ticket on Tira clouds. So you have to use Tira accounts, there are some limitation with the domain that you want to use for you for your accounts and stuff like this. So it's not necessary to be able to move to Tira clouds. And also discussion about using GitHub issues. So yeah, I think it's kind of related, it's kind of linked all together. But yeah, from for all users, most of the accounts accepted for maintainers where it's way easier to maintain because the amount of maintainers is something like 1,700 user account, which is definitely easier to handle than the 100,000 user account that you have in adapt. But so yeah, I think the discussion about the ticket system you can consider for the identity management. Yeah, on the Gira side, I think I want to say, like I'm in favor of, you know, aggressively getting rid of services that we shouldn't be managing and figuring out how to do that. So we can kind of focus on the things Jenkins should do well. So on the Gira side, I think we can separate some of those discussions. We're already using GitHub issues and Gira cloud. I think we need to figure that out so it becomes clear to the community. That being said, I don't think Gira is going away anytime soon, not with the requirements for security. So I'd be keen to say, well, can we just move Gira to be hosted by Linux foundation and pass on the login as well and do that with a view to maybe getting it done by November. Yeah, there's been a recent discussion on the developers mailing list that some people really love Gira for their plugins. Some people hate it. So it's going to be hard to get a consensus on one single tool. But we I think we can enable people to use both, but definitely moving the services outside of the Jenkins infrastructure management would be really nice. I would be also in favor to move Gira to the Linux foundation. This was something that we discussed one years ago and it was not clear at that time if we could do that. But apparently it's something that the CDF can provide. So I would definitely be interested to see if we can go down that path. The other thing I also think that you have to be clear is what are the priorities for the Jenkins infrastructure projects? Because right now we have the identity management that we have, that the batteries in terms of priority. We still have to finish the automation release project because we have a strong deadline about being able to release stable release and security release. Because once the security needs to happen for the stable release, we need to be able to provide security release as well. So automated release is still on the path. So yeah, there's something that you have to keep in mind. And just this was the last thing that I want to discuss on the automated release. I'm just wondering. So that week we agreed that we would test the security process that we would test the security process using the next weekly release next week. I'm just wondering what's our, I mean, what's the current state with the windows containers? I don't know if it's Alex or me, the blocker in this project, in these parts. All right, for the release specifically, is that what you're asking? Yeah, I'm just asking for the packaging windows, the package windows are on effects. So we can use the default .NET 3.5 container as the build part. We don't have to build our own image for that. And then we can use the inbound agent or what used to be called the JNLP agent. So I just don't know how to just set that up in the pod template. But I can give you the image names to use if that's helpful. Yeah, if you can put that in the notes and then I'll send you the PR with the changes made in order to make this working. So you know what to change next time. Sounds good. That way we're using the official Microsoft image. So it should include all security fixes and everything going forward. Perfect. I think we covered all the part that I had. So I think we're all good for this meeting. So I called three times to see if you want to talk about something specific. That's the right moment considering the amount of people in the video today. Well, so Olivier, do we plan a separate retrospective activity and a separate eventual retrospective meeting on the outage? Howard, what's the vision there? So we definitely have to do a different retrospective regarding the outage. I just would like to see before doing the retrospective to see what are the different options that we may have to solve here. Because we had that meeting with the Linux Foundation yesterday, which was planned really late. I mean, I was at this correct like one or two or before it happened and Trisha organized it in the last minute. We have a second one next week on Monday and then we'll know a little bit more about what does it imply to move to the Linux Foundation for identity management and how many time would it take to have to move it and stuff like this. So I think it would make sense to just do the retrospective once we know a little bit more about what are options. Is it, does it sounds right to you? Yeah, I just want to be sure we go through it because there certainly there's plenty to be learned from the experience. I think we have, we have two retrospectives to do here. We have one regarding the Kubernetes cluster and what happens. And we have one regarding the outage. Right. Yeah, that's fair because they're intermingled and yet too separate, you know, not having detected backups missing, et cetera, is a big gap, not having tested backups yet. Just wondering how would you prefer to do retrospective and we plan to do it next week during the Jenkins Infra meeting? How do we, should we do it by document what would be your preferred way? For me in the past it's worked well to gather as much as we can into the document beforehand and let everyone who is interested contribute their ideas and concepts into the document. So it's been deeply vetted before and then yeah, I would love to hear it if we could be ready for next, next in for, in for meeting. That would be great for me. Let's run this. Can I just add something? I think would it, and this is just a suggestion, prior to doing the retrospective, would it be, and this may have been done and I just have not seen it, would it be good to do some type of a like business disruption report? And in that report, we actually show a timeline of what happened. I think that that would be good for retrospective because if people know where the deficiencies were as they happen, we can then make suggestions. But that's, I don't know if that's been done and if it hasn't, it's just a suggestion. I think that a detailed timeline is in the document that Oleg started. Yeah, Oleg started for the LDAP outage, but I'm not sure that it also covered the Kubernetes part. Yeah, but good idea, Mark, and absolutely we should. The timeline is a crucial part of the retrospective. Thank you. The reason that I use that suggestion is because as I'm sort of thinking from the Kubernetes cluster perspective, how we made better things, I have ideas, but I don't know exactly how the timeline went. I was a piece of a part of it, but I don't know the full scope of everything. Okay, I'll do this. Okay, I can, I can, I can do this. Okay, I think we are good. So in the last topic, one time, good time, great time. Thanks for your time. Have a great day and see you on RC. Bye-bye. Have a good one, everybody.