 Okay, thank you I'm surprised the full house Thank you for coming This talk is called project necromancy It's about how we my team at Red Hat is trying to Sort of revitalize a couple of upstream projects. I'm Dave Wilde. I'm a senior software engineer at Red Hat My partner in crime Doug Mendez Abel was supposed to be here. I talked to him today. He tested positive For COVID on Friday and so he couldn't come so Best wishes to Doug send him a nice note if you know him. He was pretty bummed that he couldn't come So project necromancy I someone asked me Today or the other day They were surprised to see Keystone kind of in the description of this talk because Keystone is Keystone. It's a vitally important project to To open SAC and to and to other things and they were wondering why why it's in here and And what was happening and that's really Keystone is is the reason for this talk And it's going to be what I'm going to be focusing on mostly a lot of this we're going to Try and implement in Barbican But we haven't and really this is just what we've done to sort of Reestablish Keystone get it in good shape and hopefully if you're involved in a project that's facing similar challenges to what We were facing in Keystone. This will help The problem it was becoming difficult to get new code and bug fixes merged in into Keystone Keystone switched to like a committee PTL type of a thing and Several of the core reviewers were were no longer active. I'll get back and I restate the problem later on and I'll get back into some more details about the problem, but that's really what we faced Lance Bragg's dad, I don't know if anyone knows him. I'm sure you do he's he was he was on our team and We really realized that we had a problem when He he let us know that he was moving to a different opportunity at Red Hat And would no longer be working on open SAC. So while it was sad to see him go I'm really excited for what he's doing and he's really enjoying his new role, but that meant that we had the problem of He was really one of the last Last course one of the last stable cores on Keystone So a little bit about our team We are a we're a small team at Red Hat. We mostly work on integrating security and security and compliance features into triple O FIPS and SR back You can blame a day leave for FIPS Lance and Doug for SR back those are the but those are the kind of initiatives that we work on at Red Hat. So we have a We have a vested interest in Keystone and Barbican Barbican that actually Doug is the PTL for both projects right now. So He's on our he's on our team Yes, we we we really needed to be able to get things into Keystone Um what we have done and what we were doing The first thing that we did was get the weekly IRC meetings going they had kind of they'd kind of fallen off the The radar the weeks would go by where there was no weekly IRC meeting And we felt like this was a good first start to get the get to get the project back in shape So it's a Doug started picking up the IRC meetings a couple of a couple of Things that we we did with the IRC meetings we picked the time use UTC makes it easier Scheduling in your time zone is is rough He updated the Open Dev meetings page It's just a pull request if you go there Advertise that the meetings are back on track and that we hold them every week that we can we didn't hold one this week due to the summit And personally I find the courtesy ping very valuable at your at your neck to the list of names on the etherpad and You'll get ping to help me remember The next thing we did was review with thons I'll go into I'll go into more detail, but We were having a problem in Keystone getting getting reviews From people and there were there were a lot of old old patches hanging out about there So we came up with this idea of review thons and we implemented it in around November of 2021 This is this is where I'm restating the problem. So We realized when Lance was moving on that a lot of people that have worked on Keystone Have moved on careers progress your priorities change And so it's not really it's not really that people wanted to not work on Keystone anymore Interests also change, but it's really that that People had progressed and there was kind of an old guard and they had moved on and so we really needed to Take it over again. The other problem that we had is that Myself and Doug both I think Doug would also admit that we had very limited knowledge of Keystone Especially some of the more complicated inner workings of Keystone So it was it was hard for us to do quality reviews on our own That's a that's a it's a pretty heavy lift the Keystone code base Getting to understand it and be able to provide a a good quality review. So we We started doing these things called a review a thon a review a thon is We have a weekly meeting. We use Google meet. You don't have to Google meet works because it's fairly ubiquitous but a weekly meeting where interested parties can come and we sort of Co-reviewed code together one person will share their screen We go through the review we do research talk about it and Put our comments in the review and What an important thing is Keystone has a bunch of sub projects. I think a lot of the other projects due to We try and touch all the sub projects Every week just to make sure at least that there's there's nothing new that were that we're missing that that can get reviewed Takeaways of review a thon. So we've been doing review a thon since November of 2021 I Think my biggest takeaways that is working. I the Stackalitics only has wallabies. So I just did some Garrett math By my my account. We've we've reviewed 187 patches since November Merged 101 of those and abandoned 39 of those more an abandoning later And we're starting to move through the backlog of patches most of the recent patches have have good reviews and Are you know going through that back and forth cycle of communication via Garrett? The other take another takeaway is the virtual face the virtual face-to-face Google meet. I think is really important it gives the It's a more natural feel than doing this over IRC or something You you get to know the people personally that you're that you're doing these reviews with and there is a I Get a sense of mentorship by doing it virtually I'll talk more about mentorship later And and the other thing that I this is this is a personal one I came up with is the communal sense of ownership of the reviews. I Really feel like I have a I have a much better Sense of ownership of the reviews. I check on them more often than I would if it was just me going through and and reviewing It definitely helps the people that are submitting code to get their code reviewed So I think I feel like it's it's it's really helped to revitalize the community second set of takeaways Keep it light and informal. It's our review with on is very light informal. We chat we talk about other things I don't feel like it should be a it should feel like a chore that you're you're doing this this thing it's it's a it's a Community thing. It's it's doing what we you know We all we all really love open source and part of part of doing open source is Reviewing other people's code and making sure that it gets in there. So it shouldn't feel like a burden or a chore If it's an old patch that needs to be rebased Try the button if that doesn't work try taking five minutes to rebase it manually see if you can It lets people know that you know people are kind of surprised when they get a Six-month-old patch that all the sudden gets reviewed, you know And so we let them know that you're looking at it and if it needs work do the work and we'll get it merged in The other thing is don't be afraid to abandon There's a lot of stuff with with keystone that we found that was just not applicable anymore It had been implemented elsewhere or that bug had been fixed by a different patch So it really helps to feel like you're you're cleaning up an old house You know sweeping out the attic getting rid of those those those old those old patches cover all the projects I mentioned that and the biggest takeaway is Goes back to the mentorship idea that I mentioned earlier is that our project knowledge is increasing because Lance will join when he can Christie also wanted to be here also had a COVID scare They're two core reviewers and they've been doing keystone for a very long time gauge was gauge joined our review of thons early on But he he's moved on But being able to review code with people that have that much knowledge About keystone has been really helpful for for me personally because I I I can ask questions about the about the code about what things are doing and Really learn from someone with that experience rather than diving into it myself and trying to Trying to figure it out on my own it the the reviews would take much longer and feel much heavier of a lift Couple of keys to success about review with thons like I said, I think they're very successful Pick a consistent time just like the weekly meeting it won't work for everyone, but you do your best with your your community We our our team at Red Hat has a Upstream Fridays that we like to do so dedicate at least one day a week to upstream All things upstream and so we do the review of thons on on Friday morning Try to announce them on IRC The other thing that gauges gauge specifically told me this was important was I send out a calendar invite to anyone That wants it that really helped him justify it with his boss that time that he was spending You know his that it helps book that time. It helps remind you I think the calendar invites are a really really good idea And the last key to success is make it happen Just like the weekly meeting sometimes you won't have very much participation. There won't be a lot of people there People travel PTO all kinds of things but be as consistent as you can and make it happen and keep it going when when people aren't there I think that that consistency is is important for To make it a success The other thing that we did was PtL ship. This was This was Doug's slide So I'll I'll go over it. I'll go over it quick. It's really just doing the PTO the PTO job He fell on that sword I'd like to take one of them off of his hands when I'm when I'm able But yeah monitor the IRC channel be available Monitor the mailing lists and complete the liaison tasks you could become the the Default liaison for cross-project efforts when you are the PtL the other the other thing that we're doing is This thing called path to core That we came up with It's it's been on my goals to become a keystone core reviewer I think it's on a couple other people's goals to become keystone core reviewers. And so we wanted to Make it a make it an actual business priority to get core reviewer core reviewers on our team So The the path to core is really commitment to keystone monitoring it answering questions working on it doing doing code reviews And really learning the code and I feel like reviewathons have have been very very helpful in this I'm much further along now than I would have been had I had we not been doing these had I just been been doing it alone Stable core This does not You would kind of think as a PTO you would you would get stable core by default You don't really you have to put in a request. I believe to enter infra To get stable core. It's a little bit the reviews are a little bit different for the stable branches and Doug has stable core now. It took him a little while to get it, but we're gonna start doing reviewathons specifically for stable core back ports Now that now that he has that Future ideas, so here's some some other things that we We'd like to look at look at doing we've done outreach in the past we'd like to pick that up again If anyone knows anyone that's interested in Working on Keystone reach out. We can get them Even if they don't know anything about the code we can get them started on that I was thinking of also doing a virtual style hackathon Google Meet maybe once a month or quarter pick a bug or a feature and pair program on it and CFR we get I think that that'd be a Interesting Fun exercise So that's something else that I'd like to try spec reviewathons Specs are specs are their own beast. I think like a dedicated reviewathon for for going through a big specification We have one for OAuth 2 that neither Doug or I fully understand yet, but we need to get reviewed because people are already trying to land code for it so Those are some those are some future ideas that we have So that actually does it for my talk I wanted to I wanted to save some time For feedback if you guys have any ideas if you have any questions or comments I set up an aether pad It's essentially blank right now, but I'm really curious if anyone has any Comments suggestions questions The the the technician asked that if you do if you're gonna speak please use the mic over there So that it's captured on video Thank you, I'm dead horse just about everywhere IRC Twitter you can reach out to me there Doug is moved to Matrix faster than I did so he's demon D's that you may also know him as red robot From the From his previous IRC days Yes, hello from the ironic team. So it's not a question. Just a few comments Review meetings awesome. We do that. It works for specs. Absolutely. Otherwise people don't review specs You also may want to have fewer specs. That's one of things we are doing We're replacing some trivial specs with storyboard items if it so the questions doesn't really have to be a spec As to stable course at some point we just added all ironic course to ironic stable. It was scary. Nothing broke That's a good idea. Yeah Yeah, just a couple of hints. Thank you Nova team. Yeah, we did the same thing with stable core recently. We just merged the two teams. Okay again scary Nothing's broken yet. All good the one question I had is Me primarily but also other people we take make it took an effort recently in Nova to start stripping back features that nobody's using Whether they're old legacy drivers that no longer testing this kind of thing have keystone Have you guys looked at doing something like this? Is it something you think you'll do down the line? Anything like that basically getting rid of stuff that nobody understands and you don't know if it works We have we have not that's a really good idea I think there are parts of keystone that that no one uses anymore I also know that there are chunks of keystone that that really needs some Some review of the the LDAP back end for for instance, I know that has some there's some dragons in the LDAP code So there there there are definitely Places that that I would like to kind of do a you know refactor a thong to go and and go through those with the fine tooth comb But yeah, that's a that's a really good idea about Killing features that we don't need One other thing as well It's not actually directs it at you But if there's anyone in the audience or anyone watching the video later that knows a lot about sequel alchemy Please ping me an IRC Stephen Finn so I can close out the sequel alchemy in my great migration Hi, so from the TC now You mentioned about I know that keystone was in this DPL distributed PTL Models for some time and if I understood correctly you said that that that was one of the issues and problems with keystone So so can you like explain more? What was the problem with this DPL? Maybe this model doesn't work like it should and like TC wanted it to to work So maybe we will need to change something there Yeah Or it was like like just because in this DPL model you have to have some lieutenants for for Laszons for some different stuff So maybe it was just a problem that there was no those people at all. Yeah, it was kind of it was kind of the latter I think it was that the the the PTLs that were identified just didn't have the bandwidth to Take care of take care of everything and and maybe we're assuming that the other ones were we're handling it So I I don't I don't have any specific answers to to what what our problems were but I know that This and this would be a question that Doug would be able to answer much better But see is the PTL and worked with Lance on on taking over the the PTL ship of it Yeah, I think I think that there were I think that there were Especially on IRC I think that there were there were times when people didn't know exactly who to reach out to a single the single source of contact for For Keystone Okay. Okay. Thanks. Hi. I'm Chris Morgan and I look after the ops operators meetups and we also adopted some abandoned docs and You know, I think that this is Tremendous food for thought because the actual ops meetups team has mostly gone away and The people who said, oh, yeah, we'll we'll help you maintain these docs all kind of went away. So it occurs to me. I'm basically In charge of two bodies So this is actually probably the best talk of the week. So I just want to say thank you so much for the Inspiration because we actually have an ops meetup tomorrow The process is so broken that a lot of people didn't even know about it. So the communication And the predictability of where to find them, etc. Is is fantastic. So thank you very much. Thank you Anything else? I am the I'm the meeting runner for our team and I love to give people time back So enjoy the eight minutes Yeah, thank you. Oh, yeah