 So good morning and welcome. I'm going to talk a bit about the new stewardship special interest group we founded a few months ago and Well, let's get started a few words about me I'm Fabio and I'm Called Decathor probably everywhere. So if you look for that nickname you're gonna find me I'm a packager and a proven packager in Fedora and member of the packaging committee the Go special interest group and founder of the stewardship special interest group and I blog about the stuff I do for Fedora in the on my blog at decathor.com You get it now, right? Yeah Okay Short history, how did you get here? This whole mess started in September and I just thought let me include some mailing list messages because they're probably Show what what's been happening first clouds on the horizon were in September when What's it the right I'm planning to often Java packages listed below and the list below was like that long So and there was no reply to that message which was strange and nobody cared and Then things got worse one five weeks after that Java packages were going to get retired because they were orphaned for six weeks and Then we thought this is probably not good and somebody should take care of that because a lot of packages still depend on these packages and Moving them to Maintenance in modules only was Leaving a lot of people behind primarily packages who didn't have experience with modules yet and So I Announced the stewardship special interest group where we take care of Orphans that are Probably too important to just retire because other things to dependent them then The last coffin in the nail was probably non-responsive maintainer processes for a lot of Java packages who don't work on Fedora anymore and Now there are still over 500 orphan packages and Somebody is going to have to take care of them or they're going to be retired as well. So Yeah, I Think you know what What's missing here? Yeah, so oh no, what are we going to do now? so the the sick we founded is now a fast group for Coordinating the packaging efforts we have a Peggio project where we have tickets and do some basic planning we have a private mailing list where all the Baxila email is going and It's private because sometimes there are CVE's and they might be private and private issues and so on We have a IRC channel and three node We have bi-weekly meetings now for almost three months and There are 11 members now and most of them are Maintaining some number of Java packages now and we have as of three days ago 222 packages How many people of those are from Red Hat? I think Miro is from Red Hat I'm not sure some of Cypher boy, it's Alexander. He works at Red Hat at the doc tag PKI stack Dinesh works at Red Hat He also works on the doc tag PKI package I'm not sure about the other ones Yeah, but some of those people work at Red Hat some don't and most of them just Depend on these packages and don't want them to get retired or want them to be fixed. So Yeah Okay This is the number of packages we maintain over time. So it's been steadily growing with with a sharp spike at the beginning and This doesn't include the 500 orphan packages that are still there so there's those are the first 200 and Then there are still 500 orphan packages left and We're Working on that. So We still don't know what we're going to do about this because it's difficult with Three or four really active members to maintain 700 packages that are Mostly out of date by two or three years and haven't been touched in since so that's a bit too much work so we are Thinking about what to prioritize and the first thing we did was to put out some fires Reduced failed to build from source packages in Fedora 30 from 3 to 0 But that's no longer true because we had adopted additional packages in the meantime Then we retire check style package because is what was outdated by I think 20 stable versions and It had a number of open CVEs filed against it. So we just got rid of it and Where it was needed we patched out the dependency. So we made sure nothing important broke there then We had made some adaptations to our packages because eclipse was retired on 32-bit architectures Primarily that Was managed by switching OS GI providers to OS GI core instead of the eclipse provided OS GI implementation And we fixed some CVE issues in C3 P0 and Yeah, so the worst stuff is behind us now What we've been working on the past few months is to reduce the package set so primarily by Disabling optional dependencies so we can drop at least some of the packages for example in Apache common VFS we dropped support for Hadoop and FTP file system support both of these weren't used anymore anywhere and This fixed a filter bit from source issue because Hadoop is broken in raw height and it let us drop FTP server package as well, which is only used by that package and was Failed to build from source since Fedora 28 or something Yeah, so that wasn't good either and mean FTP server has been retired yesterday. So it's good that we got rid of it Then we removed unused support for Spock and groovy from test NG That means we didn't have to fix Spock with which was also failing to build since Fedora 29 I removed removed support for Eclipse equinox from XP in package, which means it doesn't Fail to build anymore on 32 bit architectures Remem removed support for hibernate from X stream because hibernate is not installable on Fedora 31 anymore So that's fixed up and remain removed support for spring and Java persistence API from some packages Because spring is broken and often and this will be retired soon anyway in a one or two weeks, I think So we no longer depend on spring Yeah, and we feel also working on updating some packages we've made 25 version updates We also are handling these via pull requests So someone files a pull request for version updates Someone reviews it and then we merge it because we're not that experienced with Java packaging yet So we like to have a few more eyes on those changes There are five more version updates pending and After all that then there are still 108 out of 222 of our packages are still outdated But it's getting better Yeah, the number of outdated packages is Slightly drop dropping but It looks better that way so at the beginning we had almost 60% outdated packages now it's more like 50% so Yeah, we're working on it Then I have a few bit more things to show Average number of missing updates for our packages is two so Yeah, on average two stable updates are missing from our packages Which means there are a total of 400 stable updates missing from our package set, which doesn't look so good But at least the number isn't growing anymore and On average our packages have been outdated by more than 400 days so That means it's been 400 or more than 400 days on average since a stable release was tagged upstream and it's not in fedora yet so we're also working on that but You will notice that if nothing happens this Line slopes up just because time passes and nothing happens and things get worse over time Yeah Yeah, right. That's entropy and if you add all these outdated since The days together you get that our packages are outdated but more than 100,000 days in total Which sounds a bit scary But still we're working on it Yeah, next steps We'll try to keep our packages building which isn't always easy because they're outdated and Yeah, we try to fix fail to build from source issues when they occur we try to adopt packages that we need before they get retired and We also try to keep our packages up to date now. We maintain them so I've been monitoring upstream releases for all these packages and for those That were up to date we try keeping them up to date and we also try keeping the outdated ones up to date Slowly it but we're making progress Then we try to minimize the divergence between modular packages and Non-modular packages So whenever there's a change in the modular branches, we try to merge these back into the raw height repos just because it makes things easier to maintain in the future and We don't have to duplicate efforts in both modular and non-modular branches Right and we're doing some We are also trying to remove some Unnecessary dependencies, especially some maven plugins for source code linting and test coverage which are pretty useless for RPM builds and Sometimes regenerating the build requires in packages also let us remove some dependencies that weren't actually used Yeah, then what we'll probably be working on either for Fedora 31 or Fedora 32 is to drop support for Gradle It's really outdated. It's currently can't even build itself anymore So it version 4.4 4.4 point one which is packaged in Fedora Can't build version 4.4 point one Which is a bit of an issue? because not even the bootstrap build works anymore and It looks like other distros have given up on packaging Gradle as well Because all distros I've looked at which includes Ubuntu, Debian OpenSource, Fedora They all have the same version of Gradle as we do and the only one The only distro that has an up-to-date Gradle package is arch But they don't build it from source They just package the upstream binary release So it looks like they've given up too. Yeah It's a bit complicated because Gradle added dependencies on Kotlin and it all already depends on Groovy and on Scala and a Lot of Java packages. So it brings in a whole Big broad dependency tree and it's just not maintainable as it is. So We'll try to remove it without breaking too many things There are only 15 packages in Fedora, which are built with Gradle and They can be ported to Maven if it's necessary It's already been done for some packages because there's been some I think there was circular dependencies and bootstrap issues So packages like test in GJ unit 5 and acute B and D have already been ported to build with Maven instead of Gradle So if there's really a package we need that's currently built with Gradle, we can work to port it to Maven and There's also I think five packages built with Gradle that are already often which will be retired in one or two weeks and Three packages we maintain ourselves So this won't be necessary anymore because they're they built with Gradle and they depend on Gradle itself. So We won't need this anymore Yeah, but we'll try to keep breakage to a minimum and We'll work with abstract with packages package maintainers who depend on Gradle to resolve this issue without too many headaches Yeah, I think Yeah, now it's time for questions Yeah, sure Hi camera. I'm Matthew and the Fedor project leader. This is very interesting to me So I had a bunch of questions. So I thought I'd come up and ask them. I have three Things really what order do you want them in the first one is a general question? Second one's a SIG question and the third one is a Java question Okay Okay, the general question is we often have people with ideas there should be a SIG for this there should just Infidora and then that goes nowhere. This was an idea and it went somewhere. What did you do differently that made this actually work? I'm not really sure if that's specific to this problem or not but When I made a proposal there was instantly Interest from some people who said yeah, we really need to do something about this But I think it also helped that it was some kind of an urgent issue Not not something like yeah, we have an idea. We want to do something like Lama based with a llama Fedora system or something But in this case there was really an urgent issue we wanted to fix and Yeah, yeah, right so if things are Yeah, if things are already burning to the ground people will show up to put out the fire Yeah, right we really had a few concrete things we had to work on to keep things working and Yeah, that's that worked really well for us. Okay, so the SIG question is The Java stuff was all the obviously a fire. This all seems very Java focused right now Is the goal for this to be broader to other things? Or is it basically once this fire is contained? Maybe we're maybe we're done Or will we look at other things that might not be getting the maintenance they need to across the distro? Basically, we didn't want to make this specific to Java. We also maintain a few Node.js packages There was a few of them were which were often as well and we'll take care of these but yeah the Critical thing was the Java stuff crumbling around around us. So, yeah, that was the Might see the ignition point But if if things start to break in the future, we might look at these as well So it's not specific to the Java Java stack. So I guess the second part of that one is Do you see this as something that Where instead of individual maintainers picking it up a group picks up this and you you know There was like eventually will own the whole distribution joke earlier And you see that as a useful thing like maybe we want more of the distribution owned in the collective rather than individually I mean that's a difficult question because some people are really hoarding packages and they don't like other people to touch their packages. So We probably won't be able to help these people but Yeah for for stuff like like all those low-level Java packages that we maintain Nobody really wants to touch them, but everybody needs them. That's that's really the problem. We're having with this so I Think here it's it's a good thing to have I Mean that was originally the job of the Java especially interest group but I think it has only one or two members now and Well I Mean for such low-level stuff. Nobody is really interested to work on it and but everybody needs it. So somebody needs to work on it and I think that's easier when we have a group of people instead of having Maintainers maintain some low-level dependency. They don't really care about So that that segues nicely into my Java question, which is I think the Java world in general has Rejected the idea that packaging up Java in RPMs is useful Basically when you're doing you deploying a Java application you deploy it in jars or whatever wars and these things that are you know Lots of bundling and that that that is the approach people use nobody in Production says I'm going to make an RPM out of this and ship my things that way except for a few crazy outliers You really love RPM, right? So The The point of things being in RPM is when we ship those things in Fedora in some way So the free IPA dog tag thing makes a lot of sense because that's an application that uses Java in Fedora so we need it in RPMs even if We don't you know Even if that's different from the way Java is deployed in a lot of places So have we looked at the dependency tree of all these things and made sure that everything we're doing is in Service of an actual application in Fedora, or do we have a bunch of stuff where it's like a we package this jar? So hopefully maybe somebody will use this jar Yeah, that's that's actually something I've been working on since the beginning and I have some We have some scripts in the pack your project. I Can show them since we have a few minutes I have a script that checks all our packages which Which other packages depend on them are they owned by our seek out there an interesting package? Stuff like this and if it's not dependent on by anything we want to maintain or we can maintain Then we try to drop these packages as well So we try to not only cut down the dependency from from the side, but also from the top So how many Applications like eclips or dog tag are there in Fedora that require Java at this point is it tens hundreds Five I'd say probably Low dozens it's it's hard to say because I'm not really That's invested in the whole Java language world but the things we care about are The dog tag VKI stuff because we have members who maintain these packages in Fedora and Can't modularize them in Fedora, but they maintain them in modules in rel. So It's a bit difficult for in this situation and there are some other packages we want to keep building for example Maven is important because just almost every Java package is built with Maven and We can't remove that because otherwise we'd break a thousand packages or something like that So that's infrastructure stuff. Yeah, one of the other questions. How many of the jar Dependencies are actually shared between that like how much you know overlap is there between dog tag and eclips? How many packages are kind of like singleton dependencies? This is only packaged because it's part of this one application I Haven't look I haven't looked at yeah. Yeah. Yeah, I see where I go with that I mean eclips is already maintained as a module in Fedora So they've already dropped support for raw height in essence and it doesn't build in raw height anymore because most some of their dependencies were already dropped and I think Matt maintains the eclipse packages Matt booth and He helped us with some some of the things to Unbreak dependencies which were broken because eclipse was dropped on 32-bit arches But he's primarily primarily interested to maintain eclipse in his modular branch. So that's not really an issue for us anymore Dr. Pka stack has some dependencies in in Java, but It's it's all a bit of a mess because there's a lot of circular dependencies in the stack and we've tried to break some of these dependency circles by Disabling optional support for for things removing sub packages that weren't used Disabling some modules that Just were built because we could build them, but we didn't need them So we really tried to cut things down to the bare minimum that's actually needed So make sure that we actually can support this because it's not sustainable the way it was Thank you very much for this. I really appreciate it. Thanks Okay You Yeah, thanks for coming Right And an issue Why we can't do this in fedora is because the Java packages module isn't really usable for other packages So it's there, but nobody can really use it so people have to rely on the Normal Rohit repo to build stuff, but nobody cares about the Rohit packages. So Yeah