 Hi, everybody. I am Matthew Miller, the Fedora project leader, and this is a Fedora Council video meeting. We try to do our business by email and chat most of the time, tickets, rather than having to get together to do a meeting. But we found out that actually having to get together for meetings is valuable and having a video meeting every now and then helps us all kind of stay connected. And we decided to focus these video meetings on basically looking at some part of the project, either an area which needs some support or an area which we'd like to shine more light on or maybe kind of an emerging technology area. And this time we are talking about SourceGit, which is very much an emerging technology area and kind of an interesting future technological direction for building packages in Fedora. And we have Tomasz Tomacek and Hunar, whose last name I'm not going to mangle again. He's told me that that is okay. And there are – there's not a presentation this time, but we're going to have kind of a discussion about it and see where that goes. So welcome. Thank you for introduction. So we didn't prepare anything, but I guess we can give you a short intro of what SourceGit is and what our plans are and what the current status is. It should take a few minutes. That would be excellent. Yeah, and if you could try and aim this at a fairly – I don't necessarily even know how packaging works level. We can go into the details more, but yeah, this is kind of for a general audience. We'll start there and then we can go into more gritty details as we get further into the call. Okay. So I can start and Hunar, if you have more comments, we can then turn over to you. That's fine. Sure. Okay. So SourceGit, what it is, it's actually a way to have packages within downstream and even the term SourceGit is kind of misleading because it's pretty much upstream Git. It means that you have a project, let's say upstream project that go for Kubernetes and you want to have it in downstream, which means Federal Linux or CentoStream. And the way we have it right now is very far away from upstream, like how upstream looks at their projects. So we created this thing called SourceGit and by creating, I mean, there are people who created it and we just took away and started shepherding in more and building a tour to link around it and ecosystem documentation workflows. And at this point, we're in a pretty good spot. We already have bunches of repos which work like this. We just trying to go further with the documentation and go further with workflows so that it's like end to end. Right now, it's not end to end. Like two thirds or something like that. So in the traditional packaging model for Fedora, what happens is upstream, that's the project that we're packaging up might have version control, like they have their package on GitHub somewhere. But Fedora actually predates GitHub by quite a lot and predates actually that being the standard practice for software. So Fedora's workflow is based, although we have our own Git repository, source control repository, it's based upon the idea of taking basically an archive, a tar ball from the upstream. So the upstream will make a release and they will take it from their source code and package it all up into, you know, if you're not familiar with Linux compressions, a tar gz file, rxz file, basically a zip file of all the code. And then we'll download that zip file and then we put that into a cache and then our Git tree just has a spec file, which is the rules for building that tar ball into a executable program that you can run. You can download and install as an RPM on your system. So SourceGit works more closely with the actual version control of the upstream as I understand it, right? So it goes straight from that source into an RPM or how does it, what's the basics of how it works? May I tell my story? I'm not sure if I interrupted you. So the basic problem that we are trying to address with SourceGit is like this discrepancy between how today upstream projects work in Git. And how Fedora is storing sources and downstream patches. Because yes, there is the spec file, which tells how to build the RPM from that source archive, but there might be also some not encouraged because Fedora applies to be upstream first, but they might be spec files down, sorry, patch files downstream. And then usually what packages want to do when they are pulling some change from upstream, for example, and they have a new archive and they want to apply the old patches on the new archive is that they unpack that archive, they initialize a Git repo on their machines locally, and then they try to get those downstream patches and apply as Git commits and see if they work, if there are conflicts that are resolving them. And then once they have the new history, they basically they took those extra downstream commits and converts to patch files again and commit to Fedora's Git repositories, the disk it. So, and basically what SourceGit tries to do is help with this, to avoid this convoluted way of working with source code downstream. So just to explain that a little bit more, in a Fedora package, if we have to make a change that's different from the way the upstream works, we include a patch file, which basically alters the source code. And that might be to make it build in Fedora properly. It might be to put the files in a location where they need to go in Fedora, that the upstream package wants to put them somewhere else, some of those like just moving things around patches, or it might be like we have a bug that is Fedora specific or a bug that we just have a fix for that we want to get in now. And sometimes in that last case, that bug is actually already fixed in the upstream, but it's not fixed in the release version. So we're pulling in patches that might not be there. But one of the key things is rather than being, so again, we're putting this in source control in Git. And in source control, a lot of people these days think of a patch as going from one state in the source control to another state in the source control. Like this is going from a commit is basically a change and a patch is a representation of that change. So in Fedora is what we call diskit, we actually store those patches as files, so they're not actually source control changes at all. They're by themselves new source files that are applied as part of the build process. And all of this is because our build process was invented before source control was a thing that everybody used or was used to using. So, and actually the thing you described of pulling it down from enough stream, putting it back into a Git repository and extract, I personally have never done that. I always just work with the tar ball as pure patches in a very old fashioned way when I'm working with a Fedora package, even if the upstream is properly using modern Git workflows, I revert to 20 year ago source code workflows when working with a source, I unpack it. I make, I unpack it to a directory, I make my change, and then I do a diff across those two directories to make my patch. So, yeah, so if this, if I were doing that in a Git workflow, of course, I would just be able to have the Git diff to make the patch, but then still making it into a patch and then putting it, checking in that patch and to just get is an extra step. And if you want your work to be reviewed, then whoever reviews is going to review a patch of a patch, because that's what you are proposing as a change. And like these kind of directions and strange things, we would like to solve with the source Git approach. And actually, like, and I would like to be clear because sometimes our team is invited like we would invent the wheel or something, but we don't because people in Fedora already work this way. Like the Python folks have the whole upstream Python code stored in GitHub in their repositories, where they maintain their downstream branches, and then they take that whole repository and have the tooling to convert in a diskit format. But the work they are doing is GitHub in a real Git repo, not this indirect way of working. So what we are trying to do is bring together tooling, which could be shared by different teams within the Fedora ecosystem. Thank you. Tomas, I definitely interrupted your introduction with my back story. So if you had more you're saying, please continue. No, no, that's fine. I mean, it's a discussion. So I didn't want to ramble for too long. But the point I was trying to make was that the way we are doing packaging in Fedora, it's so different from upstream that if you ask someone from an upstream project and tell him, like, please come to Fedora and fix this thing, like they would need to spend hours in reading documentation in order to have a try. And with source Git, we are trying to make the repositories look exactly the same as they look in upstream so that for an upstream maintainer or for like random contributor, it would really be exactly the same thing to open a pull request against source Git as is with upstream Git or like upstream project. So I actually have a question. It's kind of a leading question. So that Python source Git lives in GitHub, which is fine. But right now with our disk Git, there's a centralized disk Git where everybody, you know, you may not know exactly what the processes are or how that package works, but you know, you can go to source.fedoraproject.org and find the spec file and RPM for that. With the source Git approach, do I need to maybe have an account on Bitbucket and GitHub and GitLab and somebody else's stood up thing and Pagger, depending on where the possible, like how do I find where the source code is for something that's using source Git? That makes sense. Yeah, that's a very good question. So like at some point, we would love to have these repositories stored within Fedora infrastructure. So you would go to source.fedoraproject.org and you would be able to discover all of these repositories. Like that's the future I would love to see. But when we tried to make a proposal for this last autumn, I remember that we were already too late with this. And specifically, which was suggesting that there's this data center move being done, and that's bigger priority than us having these kind of repositories in the infrastructure. So early hope that for, I guess, for 34, this is already too late. And also given the fact that there is this lots of discussion about changing Gitforge for Fedora, so we are just waiting for like for the dust to settle in so that we can make a proposal and then having all these repositories within the infrastructure and having like all integrated with the rest of the services. That makes sense. So the idea would be basically to have a level of indirection there. So the upstream would be cloned to a source.fedoraproject.org source repo and then it goes from there. So as I understand this, the source repo then actually gets built into a traditional spec file and patches in diskit. Is that from if I look at the existing diskit for something that's using source Git, what does that look like? Does that still exist? Or does it go straight to Koji from source Git? No, so we definitely for this is going to stay like the definitive source of of true when it comes to what sources were built in Fedora. So we don't want to, at least for now, maybe like this is a long, long term vision, but on a short and medium run, we don't want to replace diskit. We are not there yet. We haven't proved ourselves enough to claim such thing. So current in the first step source Git would be just a layer to to help developers do their work and we would provide the service and the tooling to sync source Git repositories wherever they are stored in GitLab, GitHub, whatever you call it, to sync from there to Fedora diskit in the proper diskit format. Welcome. Okay. So let's say I have a package that I maintain that is kind of it's it's a it's not very active in the upstream, but has patches every now and then it's also it's an academic project and as such I need like 10 patches to get it to actually build in Fedora and become a sensible Fedora package. I guess the first step or for moving to us this kind of workflow would be for me to clone the upstream repo and then create a new Git repo with my patches that currently live, you know, in the spec file in the source RPM, apply those to that and then that so that I've got to get repo with my patches applied. How would I hook that up to source Git today? What do I do to make it happen? Well, actually I would start this work with first forking that that upstream project in whatever Gitforge you are using or forking to a Gitforge where we support source Git and then just create a branch in which you would like to track the your downstream changes that branch could be, I don't know, one branch for each supported Fedora version, for example, but if you have the same patches, then you could have just a single branch. And you have the same patches, yeah. They would do the work in that branch, the downstream work in that branch and then tooling service would come in and you could configure a package, tell it which is the Fedora package, this repository corresponds to which this Git branches you would like to handle and things like that. And then once the service is set up theoretically we should be able, package service should be able to propose a this Git change from that branch you have created and in this Git change patches would be already like patch files in this Git format and archive. Okay. So if I would then make a change in my Forge repo, the source Git Forge repo, do I do anything, does a PR automatically get created on source Fedora project.org every time I make a commit or what do I do to trigger that? So that's like if you configure packages to service which currently can do this and which we use to support this whole source Git idea. And in packet you can configure to take certain actions either when you open a PR or when you have just a new commit on a branch. So once you done the work in your source Git repo and your PR is merged for example, in case you have it configured to take for example a proposed downstream action, if there is a new commit on a branch then packet service would wake up and propose that change downstream in this Git. So yeah, I am probably unlikely to do PRs for myself on the repo because I've forgotten to that I know some people like to do that but it's too much overhead for a small thing in my mind. The effect would be the same. What about releases? I mean that's one of the things that our traditional model has really been focused around. This tar ball is a official release of the upstream. In some upstream projects never make releases these days which okay, some make releases and then they tag the release in Git and sometimes they make releases and there's no particular thing that gets tagged. How do I do that? How does that translate to the disk Git and how do things like version numbers and release numbers happen? Like if my package suddenly goes from version two to version three in the source Git do I have to then manually bump the spec file in the disk Git to make that happen or what happens there? That's a good question. And honestly, we haven't thought about this yet because we are still trying to trade as upstream releases to be the kings but I would say definitely makes sense what you just proposed that people would be able to create releases in their source Git repositories and then this would be carried over to Fedora and build there. Yeah, right now we probably don't have workflow for it but I can write it down and we can create upstream issue for this and figure it out. Actually for one part what you asked Matt we have support like the easy part for us is when the upstream text it's releases in Git because then you will get those tags in your source Git also and packet already has some functionality which probably could be improved but packet is able to figure it out from tags the version number and even like filling in the spec file and update the spec file accordingly. So this kind of works. The things we haven't thought about is okay what happens when the release is not a Git tag and yeah we need to figure it out solutions for this use case also. So actually I have to go back to another basic question which is is there an expectation that will be there will be a spec file or a spec file template in the source tree or does that still primarily live in disk it? And this is a difference between like a lot of Debian packages try to get the Debian control files into the upstream and we've never tried to do that really in Fedora. You have to go ahead to my story. I already muted myself. Okay, actually both you can have all the downstream packaging files in your source Git repository or your upstream repository and that's the ideal thing because you are in full control of everything but you can fetch it from Fedora Diskit for example for every upstream action or source Git action. So it's really up to you. I can see and there's kind of two different major cases and one is where I am both the Fedora Packager and the upstream or part of the upstream where it might be convenient to do that or there's the other case like my project my package where I'm not part of the upstream project I just want it packaged in Fedora and the upstream people aren't particularly interested in that so those are yeah, okay. So when packet service started out it was mainly intended as a tool to help integrate upstream projects in Fedora and the main idea back then was that the easiest way to do this is to have the spec file upstream and then everything works like a charm but and this works very nice in cases when Fedora maintainers is also actively participating in the upstream project but we also learned that many upstream projects just like don't care about what's happening in Fedora and they likely do so because that's not their business they are not creating a distribution they are doing their project. Also the case where they want to make a spec file which will work for Fedora, OpenSUSA, Mandrieva and whatever other RPM based things they are and so they make a kind of a cranking spec file with a bunch of conditionals or a bunch of things or this doesn't really conform to anybody's guidelines but it'll build on all of them so good enough or my favorite all those guidelines are stupid my own way of building RPMs this is the spec file the way they should be so sometimes upstreams have their own ideas and then they have a spec file but it's actually not useful at all in Fedora. Yeah, sorry that got carried away into a rant I didn't even have a point. I guess I do have one question about that too which is having all the spec files in disk it as the source of truth makes it easy to do things like mass changes where we want to like introduce a new like license field into the thing where we do mass changes across a bunch of spec files if the source gets is the source of truth in some ways for the spec file like how does that get reconciled how do mass changes cross packages work? I'll say disk it is still the authoritative source of truth and this is actually hard requirement from Fedora proven packages such as Miro Harenczuk who says that they have scripts built around disk it and they want to be always able to change things in disk it or like for massive things. So we want to preserve this and that's why whenever you do our changes in source disk it like then they are changed into disk it and we want to have a way to do it like the other way around as well. Yeah, it's gonna be tricky and we still need to figure out like all the like all the corner cases but that's what we need to do. Okay. Like thinking back will be problematic in many cases but for like mass changes probably should work fine and if things cannot be reconciled by automation then it's just open a PR and ask a human to fix it. That makes sense. Do people other than me have questions or comments? I was reading the sorry the packet, you know, the documentation. I have something a little bit in my mind when you ask it earlier. What about, I'm currently testing and packaging some, how do I say, codecounter package wasn't exist in Fedora and I just pack it in copper to testing copper to see how the core is gonna be packable, packageable. There's one problem I have. One package I made is actually archived. So the guy himself is doesn't even using anymore is this repository doesn't work. How is this going to work? I might have to fork it again and maintain myself or because this is only using for just small portion of the code itself. So it doesn't actually matter. I just have to build one time and let's just say this is version 001-1 and done. I don't care what's happened because maybe the actual, I can also go to upstream and say to guy, okay, please can you just change this one if this is possible because this repository doesn't work anymore. It's an archive. There's two ways to do it. Ask to him or I need to find a way to have to do it myself in code and suggest to him so we can just remove that part. But let's just say I can't do that and I need that package at the moment. How do I cope with this problem in this kind of situation because this archive actually, this repository doesn't work anymore. You don't want to become the upstream maintainer for it. You just want a copy of it that works. Yeah, I just want to use the code counter and some people asked it to me earlier, one of my friends say, okay, is this package exist? I actually checked. It's actually fast. It's useful, but we don't have it. I just package for copper, so it's there. But yeah, please. I think you are mixing two different kinds of questions here. So one is if package is not maintained anymore and you need to recover it, that's one thing. The other is if upstream was archived and dead and there is still a maintainer in Fedora, what a maintainer can do with a dead upstream and if maintainer can recover the new upstream. And source gate folks, they cannot help you with that. Like if upstream for the certain project is dead, someone needs to become the new upstream. And if it's not you, then someone else needs to become the upstream, right? So you need to find someone. Yeah, you have to fork it and you have to own it or you have to find someone else. Like dead upstream is dead and something needs to be done. So I'm gonna hold it down, put package gates all over the, okay, okay, fine, fine. Okay, is that, do you feel like your question is basically answered? Yeah, but I still have one of the issue what you said earlier about version changing. So it's, I think it needs to be changed somehow automatically to bump up the version inspect file somehow because we have to change it always manually. So I think package doesn't provide that as I believe, as I understand at the moment or I'm wrong. So in packet you can either define a version template or I also think we have an action which can output you a version string and then you can specify that in your spec file. And whenever a change is proposed to this gate that one is used. So if I just change, if I just create and so this is the new version and the tag itself. So everything else going to be bump up automatically and the change look and other stuff. So I just need to point the direction. This is the new guy, new version, sorry, undone. Yeah, well the version template works based on tags. I'm not sure if that was like you need that tag in your source gate repository or upstream repository which like, it looks like a version. But then you can use it from there automatically to fit it into the spec file. So is it counting the versions to incrementally in that tag or any documentation can you point to me too? So I can also test it on my copy repository and see how it's going to be. Or like that tag configuration. I just, in documentation I don't know where can you show it if you know what it is. Just go, yeah, thanks, thanks. Yeah. But in the end, you should be covering the versioning of the software because like fucking can't know like what's the actual version you mean to have. So either it like have it in git tags or whenever there is a new upstream release just like make sure it matches. So that it doesn't change version in spec file for you. So you should be like you are the maintainer you need to know. Yeah, okay, okay, okay, I got it, okay, I got it, okay, thanks. There's some links being dropped in the chat here which we'll try to get into the YouTube archive as well. That's all for me, thanks. Okay. Anyone else have questions or comments? Yeah, I'll take one. So one of the criticisms that I've seen of this model is that it requires a lot more git knowledge than diskit does. So a lot of packages of software now in the diskit model basically just enough to do git pull, git branch, git push. And because they're not necessarily developers they might have just enough skills to sort of do the packaging part but not necessarily the code contribution part. So how much more git knowledge does source Git need in order to work and how can we help our contributors get that? That's a difficult question to answer. I guess because both of us are developers and for Git is kind of like a daily bread and... If I would give some perspective, like I also work with Git a lot. So I'm not in UV perspective but I think that Git becomes quite a like default knowledge in the current way of working with things and not for developers alone but also for documentation writers, for people who create designs and for people who work with all like creative and technological or like just text data. So I think it's not something the source Git folks may like resolve on themselves but it's something which generally happens with Git as technology that it gets more and more rich to wider audiences and the tooling around Git becomes better. We have now web UIs of various kinds. We have now Git helpers. We have now Git clients. We have now a lot of visibility for how Git is working and also like I don't think there is the actual deep dive into Git internals. Like we are not doing a 20 heads merge like some kernel folks do, right? So we don't really require the expert level of Git to operate on that level but indeed you need to be not afraid of taking this system and like looking into it. And personally, I believe that this is something people need to start doing on the level of, you know, like how you learn arithmetic at school and that the 10th year at school you learn Git because like this is your way to open the whole world for you as a graduate from school or something. So we're not there yet but this is I think where we're heading. It does seem like maybe something we could do in Fedora to help people, help on board people. We could do Fedora classroom sessions around this and we could work specifically on some documentation of like this is the level of Git you're gonna need to know to do this and we can, it could be a benefit to be doing this in Fedora because you're building your knowledge that like you said, Alexandria is actually useful for a lot of other things as well but it also, yeah, it also is a little bit of a barrier. Everything, every layer of things we do makes things more complicated. I think Justin has experience with docs writing right now where it happens with Git and I guess like migration from Viki where you could just type in text and to Git where you can actually, you have to actually submit it is again like a barrier but in the long term, it's worth doing so. So yeah. Yeah. On the other hand, I'm not sure if this is like too brave from myself but if I just take the use case you told us Matt that you are unpacking the archives and applying patches basically you would do the same steps with Git it's just like Git commands, not patch and this one you are doing right now. So the commands are pretty obscure too, right? Like patch dash U capital R and then an N because I wanna make sure I don't miss the new files, right? Yeah, it's not like that. The Git command line will help you to get rid of these obscurities because everything is already built in. Yeah, I guess if you are not scared of patch commands you shouldn't be scared of Git. It's like if you manage that you're already like good enough. Yeah, so that might just be a matter of a little bit of hand holding that was something we could definitely work on. Well, and that's something that we can, that's part of the reason we have these meetings at the council level is to kind of connect some of these things like we've got this technical thing that's very much on the engineering side about we are improving the package or workflow and we can see okay, there's a outreach and documentation and education need that's connected to that. So that's one of the things you can do at these kind of things, so that's cool. Also about the Git usage it's already broadly has been posted in some various places in Fedora as a video as a documentation also before. So that documentation requirement is actually in there as well. So making that is probably also easier because the order they have a lot of stuff is actually pretty much ready because I know that Git one and one and further has been short before tons of times as well. So that Git doesn't change too much, so it's there. Cool. And like just one thing, one of the drivers that motivates us to work on this is that we think that with SourceGit we would enable a larger group of developers who are not part of Fedora yet and for whom learning this Git would be yet another bump to take to come and be able to participate. So if we bring the way we do packaging in Fedora closer to how development happens in upstream, then we think we could bring in more people to the community itself. That makes sense to me. I guess the next big question is what can we do for you to help and one of the things is you were talking about priorities and the CPE team needing to stand things up and having data center move and things being top priorities over that. I know the first quarter of CPE time is pretty much spoken for and like everybody under the pandemic, their capacity is lower than usual as well and they've got finishing up the authentication system and there's ongoing CentOS stream work but they will have capacity in the future. So it sounds like we do wanna wait a little bit to see how the Git forage in Fedora thing is going to work out but when you're ready for more of that, talk to us because part of our job as Fedora council is to help CPE understand what the Fedora community priorities are and help kind of rank what the priorities are and this sounds like a good important initiative that we'd be very happy to tell them is important when you're ready to do it. Yeah, that sounds absolutely great. And we would love to because I feel like, I mean, I'm personally struggling with like proposing initiatives or changes within Fedora and carrying them towards like success and especially for things big like this, it's really intimidating. So like getting help from you would be very much appreciated. And the other thing I wanted to say is right now we are strictly focused on CentOS stream and when that work is done, we should be able to raise the same workflow, same code for doing this within Fedora. So, and that would be ideal. Okay. I guess another thing that maybe a thing that Ben is working on, Ben, are you ready to talk about your program manager team? Idea? Not yet. Okay. So stay tuned. So we're looking at kind of, you talked about the, I don't know, intimidation factor or just the, it feels like a lot of workload to help like shepherd changes through Fedora. Ben is working on some things to help teams with that, to help get assistance to people with getting a following up on help, making changes actually happen and making sure all the details are taken care of and things like that so that's something we can probably help with as well as time goes forward. Thank you. Are there other things where we could help? I think we are still in the phase of proving the concept and working on the tooling and proving the tooling to, and make it familiar to the community with this, this whole idea. Okay. Cool. So. Let us know when there are things. Question I have is, do you have already early adopters among Fedora maintainers with whom you work or you're looking for them still? Like, do you have people to work with on this initial setup? So we already have plenty of usually redhead teams who are interested and we are like in touch with them. And this is mainly for the central stream work. And at the same time, we have plenty of people who use this in upstream, like without the source getting that they directly use back in upstream and then get their upstream releases to Fedora directly, which is really great. But at the same time, we definitely look for more people like that would be perfect if more people try this and on their packages because as more and more people use this, we run into different issues, problems, corner cases and it's like, I can see that it's getting better every day with more bugs we fix, like it's getting more stable or like more corner cases being fixed. So that'd be nice. Thanks for bringing that up, Alexander. That's a good point. Maybe a community blog post asking for more testers is probably, it's something that could bring some of that or we could even do, if you're ready for a bigger thing, we could do a develop announce post saying you're ready for that or just develop this posts reminding people, this is an ongoing thing and you need more help or more volunteers. I think there were some of the posts like this recall and at DEF CON you're going to be presenting sort of as well, right? Yes, and we are going to have also a workshop there. So maybe you're not like do additional announcement, but when you will announce this video call, for example from a console side also say, but yeah, everyone is welcome again. Absolutely. Retirate a message. We would definitely would like to do that. We are just like in the middle of some work, we would like to wrap up. Things are a little bit unstable right now. Okay, wait till it's ready. There was a presentation on from last summer and Thomas had a blog post, federal community blog post, if I remember well. And there was also a very fruitful discussion on DEVO last year sometime, I don't know, 2020 is a blur, but yeah, there was one. Yeah, it was in autumn after flock. Yeah, it was actually too fruitful. But yeah, right now we are focused on DEF CON which is in one month. So we'd love to have to workshop there and basically let people try it out and hopefully use it. And from that point, we could actually do some blog post or something to do more promo because at that point we should have like better, but like more things implemented and more things covered. Cool. Any closing words, final remarks? So I'll be honest, it's very late here, it's almost eight PM and I'm really tired. So thank you for inviting us. And yeah, we're really trying to work this out. And if you have any ideas, how can we make better, please let us know, we would love to discuss it. All right, awesome. Thank you very much for joining us late at night and I will let you go. Goodbye everybody, see you next month. Thanks. Before that for a lot of people, but the video call again next month. Thank you, see ya. See you.