 Alright, so welcome to the nth annual packaging and git buff for some value of n greater than 5, I think. So I hope we can have a buff buff and not a lecture buff. Both of us are employed to lecture. That's right, so if we lecture you have to pay us basically. So you have to ask lots of questions or we'll charge you money. And the tuition where I teach is frankly exorbitant, so hopefully my boss isn't watching this. Okay, so we don't have a lot of fixed material, but I think part of what we want to do is, or what Sean wants to do is talk a bit about the git, which is one aspect of packaging and git, widely misunderstood, including by me. So that's why we're going to talk about it a little bit. But I think just a quick demo and we'll see where the discussion goes from there. Do we have someone to take questions from IOC? I mean I guess I can do it if nobody else wants to. Oh yeah, sure. Okay. Alright, well I've got a quick demo of how git packaging workflows can make people's lives easier. It's worth drawing a distinction between git workflows for package maintainers and git workflows for everyone else. Because often the workflow that a package maintainer uses is different from the workflow someone uses who just wants to hack on that package. And this demo is an example of a git workflow for a non-maintainer. In particular, it's using git to sponsor an NMU. Okay, so I made this fake NMU last night. It better be fake. It is because it's David's package. We can see this guy, a contributor, has made a commit on this not-much repository. And what you're meant to imagine is that I'm a DD and I've cloned this repository from GitHub and this guy has said, hey, can you sponsor this upload? I think it makes things better. And the non-much maintainer is ignoring my emails or something like that. Totally unrealistic. And if I'm that DD, one thing I need to do is this repository is essentially untrusted. I just got this off GitHub. I need to div this against the archive to see that the NMU doesn't sneak in some other changes that I know that David definitely doesn't want to go into his package. How do I do that without Git? Well, I have to build a source package from the repository. And then I have to... So I have to de-package, build package, HS, right? And then I have to go to the parent directory. And then I have to apt-get-source not-much. Make sure I get it from SID. And then finally, I can use debdiff. And then I have to tab-complete two horrible source package names. And I can finally see what this guy's doing. And debdiff is cool, but it's simple. It'll only just give you a straight diff of the whole thing. Wouldn't it be great if we could use git-diff to do this instead? Git-diff has lots of options. You can filter by file name. You can ignore things, include things. Well, with de-git, you can do that. So I'm not going to touch apt-get-source or deb-diff to do this. So I type de-git fetch SID, which means take what's in unstable and make it accessible to Git. Oh, hilarious. What package are you dealing with at this point? It reads the changelog. I'm going to try de-effetch on stable instead. Is that any random repository like it's for a package? Yes. Video team guy, use the microphone. Yeah, that's telling me. Sorry about that. Could you do this in script and then publish the log afterwards so that people can read that? If you tell me what to type. Script foo. Do I need to redirect it? No, no. It'll open a file. Can I do that to put it something? Okay, cool. Thank you. Right, we'll try that again. The directory you were in was just on there. Okay, so we managed to work around that bug, which is good. So you can see it's pulling the not much DSC out of the archive, but you can ignore that. And it's telling you that a Git commit has been synthesized from the DSC, and that's deterministic. So you can do that as many times as you want. It'll always give you the same commit. And then to review the NMU, we just use git diff. So git diff, dot dot head. Okay, so let's review this diff. They seem to be hijacking the package. Good. And if we scroll up to the... Well, let's see what they did to the changelog. So we can filter the diff, because this is what you can easily do with Git, to see what they did to the changelog. And not much is mine. Okay, so I'm not going to upload this. But, you know, what did Git let us do here was... Diff against the archive in a secure way without having to leave Git, which I think is very nice. Okay, that was my demo. And then in order to possibly help us talk, I made a single slide, hopefully covering all the workflows that currently exist in Debian. And this is for, well, the main ones. Oh, so I have to interject here. I wanted to call this slide. We are all special snowflakes. Sorry, no, I completely forgot. Here we go. There we go. Okay, so these are Debian Git workflows for maintainers, right? So the demo was a Debian Git workflow for a sponsor, who's not necessarily the maintainer of a package. These are workflows for maintainers, which is probably what we're more interested in talking about. I split them into three groups. Firstly, there's the Git, or do you get, plus a helper. Get build package, get DPM, and David's favorite Git package. There are the workflows that you can do with Git, but DeGit can't handle, and these are on the way out, I would say, except for the Haskell team, which has several hundred of them. Which is Sean's favorite team. Yeah. So repositories which contain more than one package in one Git repository, DeGit can't deal with those. Submodules can't handle those. Never will. It's enclosed, it won't fix. And repositories which only have the packaging in and not the upstream source. DeGit can't deal with those. And then more recently, we have these new pure workflows, right? And what I mean by pure is, you can think of DeGit as an add-on to almost any existing workflow, but these are the ones where you only need Git and DeGit, rather than GBP or DPM or something. One of them is a merge-based workflow, which is usable now, and the man page explains it. And then one of them is not usable yet. It's a rebasing workflow. I can talk about that more if you want, but I'm not going to say anything now. So hopefully this slide will be useful to our discussion. And that's all I wanted to say. That's where we're going to stop talking unless they pay us. Right. Okay, great. Have you got a hat to collect coins in? Yeah. I hope that works better than when I tried in class. So, discussion. Antoine. Wait a microphone. So I tried DeGit a while back and stumbled upon a few bugs, which I believe I've been fixed now, but I didn't see them as much as bugs as peculiar issues with my peculiar workflow. And I think there are so many ways of doing this that I wonder if part of DeGit's goal is to encompass all of those eventually or just restrict ourselves to a certain set of best practices or do we claim there are best practices or where do we play in there? Like, for example, only Debian sub-directory, I use this sometimes. You're a bad person. Yeah, I know. I'm bad. But, yeah, like, if it's a wrong practice, I'm ready to change, but if it's just a matter of, like, oh, maybe we'll support this eventually, then that's fine too, you know? So I'm curious what you think about that. Yeah. So first thing to say is that one confusion people have when talking about DeGit is that they think it's a tool on the same level as, like, GBP or DPM, which gives you a workflow, basically. And it's, although we've got these pure workflows where we're saying, okay, we're going to invent a new workflow that capitalizes on DeGit, the tool itself imposes basically one requirement, right? And the only requirement it imposes on you is that if you take your Git head and package it into a source package and unpack it again, you get the same thing. That's it. So you can see why it's not going to be compatible with these, because if you only have the Debian sub-directory, if you pack Git head, well, you can't pack Git head because it doesn't have any upstream source. You wouldn't be packing Git head, oh, I suppose you could. Yeah, only. Okay, okay, so pack and unpack, and it's still the package you actually wanted, I suppose. And that source package builds, I think, is something that he left out, but is also true. You'd have to build. Yes, okay. So Git DPM doesn't give you that either. What do you mean? So in Git DPM, my Git head has the patches applied. Right. So there's a microphone right beside you, Sam. Okay, thanks. The question about whether Git DPM actually meets that requirement either in Git DPM, my sources have the patches applied in my Git head, whereas, oh no, I guess they are compatible. Yeah, but I thought there was magic to make that possible. Okay, there is some very slight magic going on to deal with some small corner cases that most of us aren't even aware of until we bump into them. But if you're trying to decide whether your workflow is decompatible, what you need to think about is that requirement that it can be round-tripped when you're packing into a source package and unpacking. So just to show my ignorance as boff co-organizer, what does this mean about patches applied? I mean, there is hackery in the Git to work with patches unapplied, right? There is some hackery to deal with that. Okay. Basically, you just pass an option and it doesn't. Right, okay. But if you think of a patches unapplied repository, it's still the case that everything's there and when you pack it, you get all the same stuff back. Well, this starts to get a bit philosophical, right? What's everything there? Is a URL enough? I mean... Well, I mean, as compared to the packages, whether it's only the Debian sub-directory. And a link to the tar ball. Right. I mean, how much magic is allowed? Okay. I used to get yesterday or the day before to upload something in... Great choice. I was really testing it out. For my package, it did manipulate, of course, what I had in my Git repository. If I now would push this to my team's Git repository, it does look weird. It has more history because it... I think what I saw is that it made a branch where it applied the patches. So it created Git commits for every patch. And then in the merge, it basically reverted those, such that I had a merge commit which didn't do anything on my master branch, but now I had an extra commit and didn't do anything and I have an extra piece of history. I think this is at least worth mentioning if you have this unapplied history. Is this actually, if I would do it multiple times, is this going to keep on adding or is this just a one-time only kind of... Does your team use patches unapplied or applied? Unapplied. Okay. Did you tell Digit that you were using patches unapplied? Yes. Okay. Well, I mean, I'd have to look at it. What I'm guessing is going on is that Digit... Digit wants to make sure that what you've got is a fast forward of what it thinks is in the archive. But it lets you override it. So it creates a branch representing what I just did there with Digit fetch and then merges what you did on top of that. It did merge it into my master branch. Your master branch changed? Yes, with one commit, which didn't do anything. Okay. Oh, so it would have been a pseudo-merge commit. Right. So it was just joining the histories. That will only happen if someone does a non-Degit upload before you do your next Degit upload. So the first Degit upload will always have this little bit of history strangeness, I guess, for most people. But also, I guess, if there's a new upstream or not? No. As long as every upload to the archive is made with Degit, then it won't do that. But on most teams, I guess, you could expect... You could expect some skews. Some back and forth. People using Degit, people not using Degit. I mean, that's... Oh, Gregor knows we're always trying interesting things in the Pearl team. Sometimes it's... You know, the coolest technique is only useful if it's useful to the people on the team, right? So... I was also wondering, which I haven't dared to try yet by really pushing because I was afraid of messing too much with my history. I have a couple of patches which actually carry a big patch set, which typically, of course, with new upstream, needs to be... The patch needs to be patched. Yeah. In an unmerged workflow, would that... Right. I'm afraid that I get one merge where I have to resolve all my conflicts which I don't think is good for inspection later on because it does include upstream changes as well. So there's basically two choices in this situation. If you have a package which has a substantial and semi-permanent series of big modifications, which we have in Debian, there's basically two choices. Either do patches applied and use GBP, which Deget supports, or you wait a little bit longer and then we'll have this rebasing workflow. But yeah, unfortunately at the moment, if you're in that situation, it's not the best tool. But this is not a de-git-buff, so we should talk about all the other options that there are. So currently, indeed, in this workflow, we're manually doing the guilt updating of the patches by applying it one by one to see every time what I'm actually doing. Right. You can use GBP PQ to make that a little bit easier. So I'm curious, how many teams or people are using patches applied workflows? Okay, me, but... So it's a minority of people still. How many of you know who I supposed to apply? To the buff or to their packages? Do you use microphones? Can we get a microphone? Sam needs a mic. Actually, why don't we trade stuff here? So, is there an easy way... No, no, I'm unlikely to need a mic. It will be good for me to not have a mic that's... So, is there a way to convert... Like, if I have a checkout that currently has patches applied, and I'd really like to, at least for the moment, look at it, patches unapplied for easier diff stuff, is there a way to get that? I don't need to commit it. There are times where looking at a diff, it would be really nice to actually get a patches unapplied version of a tree, even though that's not what GitDPM gives me. So, I might be able to help there. So I wrote a tool called GitDebCherry, which is included in Git package, but it's not especially tied to Git package. And it can extract a Quilt patch series from a patches applied repository. And, of course, that's sometimes a little tricky to do with complicated history. So... No, no, no, I've got a good patch series already. This is Git... I have a GitDPM repository. I just wish to... I want to turn my perfectly good... I have a great Debian slash patches directory. I don't have the Quilt metadata to do a Quilt pop-a. Right. I don't know of a tool that does that, apart from unpacking the average tower and another directory or something. That's sad. But the other direction is definitely possible. Yeah, I know, but that's not... But I use GitDPM, so that's not the problem I ever have. I see, I see. Yeah, if you're someone who prefers patches applied, and you... Sorry, yeah, yeah. Right, you prefer patches applied, you can always get that pretty straightforwardly, yeah. I prefer GitDPM. I just sometimes need better disks. Okay. I'd have to... If he's just... Hello. Hang on, hang on. Hello? If it's just a one-shot thing, I'm pretty sure, but I have to check there's a Git command you could run to basically check out everything but the Debian directory relative to upstream, and you'd get the tree you wanted. But that may or may not be... That may be too much surgery. Oh, that's great. Actually, that worked. Do you think... Rob, are you thinking like a Git work tree? There's an option to Git checkout that you can, you know, just tell it... I think you can even tell it... I know you can tell it individual files, but you can also tell it subtree and give it a commit, and I think it'll just revert that part of the tree. Right, but Sam would need to have the upstream, the whole upstream history in his Git repository. I do. Well, if he's using GitDPM, he probably does. Okay. Hang on. We've got lots of time, don't worry. But what I want here is... So you're saying I should check out the upstream tree, then graft in the Debian tree from the commit I want. Well, I was actually proposing the opposite, but... How can I do the opposite? You basically tell Git to check out the upstream tree, ignore accepting the Debian directory. Ah. I think we'll have some feedback back here on this topic past the mic. If it's just a Diff you're after, the other approach is just to use FilterDiff. Right. Or I think Git has the ability to filter by path, so if all you want is a Diff of the Debian directory in the patches, and you don't care about the direct changes in upstream, you can just... You need to filter Diff it or specify a path on GitDiff, and that should give you the Diff you want. You can also filter Git here. All right. I mean, so this is getting a bit narrow. I mean, so Gregor. Sorry about that. No, no worries. I mean, it's a narrow bar, right? So... So I'd like to ask a simple question. I haven't really paid attention to Dikit development in the last three years or something. So what is actually the advantage of using Dikit as a maintainer for one's own packages over just Git build package? Okay, I'll try and mention as many as I can remember. You get a bunch of extra safety checks. So there are certain things that keep happening in Debian, like your changes file has different distribution to your change log or something like that, and Dikit protects you from those. And secondly, you're sharing your full Git history with your users. So that's good for them, but also if they send you patches, they're going to be based on your history. So, okay, what do I have in... We need Mike. Sorry, it's big room. You're sharing in a history in such a way that it is automatically discoverable by the... There's the receipt of that. That... No, I... Right, but we can't... So... Okay, so... Then why don't you rephrase that? Okay. Can Git help us with microphone? You make your history accessible via Dikit clone. Whereas right now, if someone wants to get the source of your package, they have to try and find it on Alioth, right? And for... Right, when Debian Checkout can try and find it on Alioth for them. But Debian Checkout can give you a different thing every time. So if I Debian Checkout a Perl repository, it's probably going to be pretty sane. If I Debian Checkout a Haskell package, I'm going to get all the Haskell packages. That's quite a confusing interface. Dikit clone always gives you the same thing. And that's good for attracting new contributors and probably contributors to your package. Someone can always Deget clone. But if you Deget pushed, they get a more useful history. That's what I was going to add, that maybe we need to fix the cases. When you do Debian Checkout, you get something that changes. But with the current control file fields, we should be getting just the package repository. And I think we should aim for that. Sure, it's an interesting aim, but, okay, hands up everybody who has ever forgotten to push their changes. All right, come on. All right. And hands up everybody who's done an upload without a VCS, the correct VCS fields. I mean, these things happen. So even if I'm asked this same question and work, but I think this actually, this uniformity is a big benefit. Even if the get checkout you get is this rather crude history of one commit per upload, it still makes it easier for somebody not knowing the intricacies of quilt and source packages to patch the Debian package. I mean, the underlying assumption here is that there are more people in the world that know get than that know quilt. I think that's a fair assumption. And just to expand on that, so you asked about benefits for maintainers. This is more of a benefit for users, but it's connected, so worth mentioning. If you're someone who's moderately new to free software and you're just getting into the idea that you can modify the stuff on your own machine, you're going to be wanting to do that in get. So suppose I find a package and I find a bug, and there's some bureaucratic reason why it can't be fixed upstream right now. But I want to share that modification with my friend. Deget lets me do that in get. So I can de-eclone the package, apply my fix with a get commit, and then just push that to get hub or wherever and share that with someone else who has the same problem. Whereas right now, because this is one of Debian's guarantees with a stable release, we say, you can modify any of the packages in the stable release and we provide the build dependencies, so you can rebuild them. But we only provide that in the form of out to get source, which is hard to share to get improved on that. You just said you get a raw history of what commit per release, is that already the case? Because I think it just gives you the latest one, right? It does. So an open bug that it could give you one commit per whatever it can find, but right now it only gives you the latest one. I guess you have to implement that first on the Deget server, inject all the packages or something like that. Right, there's some ideas, but if people are using Deget push, then this becomes irrelevant. A question about that, I think there was this overwrite kind of option in Deget. Does that mean that on the Deget view, indeed you lose all the old history and just my Git tree as I now have it, is the Git tree that you get? You can't overwrite what's on the Deget servers because then the history wouldn't be fast forwarding. What the overwrite option is saying is, Deget thinks you're going to lose something here and you're saying, no, it's okay, I've got all the changes. It's mainly designed to prevent you from accidentally failing to incorporate an NMU. That's another advantage. Deget won't let you push if it thinks you might have missed an NMU. You have to tell it. How can I now then upload my package history that I currently have if you already have a Deget push once before? Then you would just use overwrite and it would merge your stuff over the top of the stuff that's there. Question here. I know that you have thought this over because, well, I know who you are. But one thing that I have left pending here is that how is this integrated for groups working? Say the Perl, the Ruby, all those groups that use version control to handle large amounts of packages. Is something thought for the workflows? Sorry, can you repeat the question? Yeah, yeah, yeah, I mean, what I'm missing here is that, well, I know that I am not active as I was once in those communities but I still follow more or less, try to follow what's done there. How does these workflows integrate with things that go beyond individual packages to the workings of the whole groups? I guess in the, I mean, the only setup like that I know is the Perl team but essentially it has tools to iterate across repos and so the iteration is happening at a level above Git, right, as opposed to with SVN when you would say commit changes to a bunch of packages at once, atomic-ishly. So I guess you would have to do the same changes but I think that there's two different questions here. There's how you arrive at the thing you're ready to upload and how you upload it. And the how you arrive, for example, by running said across 2,000 packages doesn't change but then instead of doing 2,000 uploads with DPUT you could also do them with DGIT push, that would be an interesting test of the DGIT server. Yes, it would. Well, it is 2,000 commits now for the Perl team and they're only, they happen more or less contemporaneously at the same time but they're not atomic in any sense. They're only as atomic as the tools are bug-free. I don't know, Gregor, do you have any disagreement? Not on this topic. Okay. More questions. Yes, in the back. Is DGIT a good tool for me to use if I want to collaborate with other people who are not Debian developers? Like I'm working inside a company or just with my friends or something. Sorry, yeah. If I'm working at a company or with my friends or something and I want to add local patches to Debian repositories, is DGIT a good place to start? I've had good success using Git repositories for packages where the upstream, that is the Debian team is using Git but sticking things into Git at work, like importing it from an SVN or just not packaged at all workflow is sort of weird and I would like to use DGIT. Are the tools there for me to do that? So, DGIT, one of the design philosophies of DGIT is that you only have to use DGIT when source packages are involved. If you're not dealing with source packages, it should just be part of something you can do with Git, not DGIT. Right, so we are generating source packages because we're building locally patched Debian packages. So, I'm going to generate version 1.0 plus my company 1 or something and we're going to have some weirdo local patch in there. Right, right. Yeah, I mean DGIT would let you obtain from Debian the thing you're going to base your work on already in Git. No matter what the Debian maintainer does. So that's an advantage. Is it going to give me any tooling for managing source packages in my own repository, my own private repository? I think plug wash might have more to say, but I don't think I know what you're trying to do. So I might have enough distance here to see where you're going. So I think the managing source packages in a repository isn't really a thing according to DGIT. It really likes to think that they're ephemeral, that source packages are disposed. Okay, go. So this goes to the question of how to build a derivative distribution, right? Who does that? And the point is that in DGIT, it doesn't care where it's pushing things and where it's uploading things, as long as you tell it the right URLs to upload to. So by default, it uploads to .debian.url for DGIT repository, .url for .debian archive, .url for FTP master, and etc. And in your DGIT packages, you could change that config to pull from .debian, yet upload into your own private repository. And it will, if you provide the APIs which match the DGIT server APIs, and if your repository looks somewhat like .debian, then it should be quite easy to tell it to pull from here, but push there. You just have to buy an enterprise license for the DGIT server. Okay, in case anybody didn't get, that was a joke. Please do not attack Ian. Yeah, however, like for example, for Ubuntu, we haven't yet worked out how to integrate the merge workflow for merging from DGIT history of .debian and basing Ubuntu DGIT history on top of and merged with .debian DGIT history, because for us, we want to merge packages. Right, right. That's not clear yet. We want Git merge to do the right thing with all the patch cues and everything. Right, that's really hard. More questions, complaints, people who want to revive Git 3.0. Who remembers Git 3.0? Who hears an FTP master? Okay, good, it's safe to talk about it then. Sam needs a mic, despite his previous claims. I guess I'll say that I certainly, there are times when I find dealing with Git 3.0, with 3.0 quilt and Git, frustrating enough that I am so tempted to go randomly hook the package to override the package source so that it'll accept the native version somehow and then use 3.0 native or just drop Debian revisions from my packages because if Gilliam's going to be that way, then I can be that way too. Seriously, there are times where 3.0 quilt just gets to be such a pain. So you're proposing a social solution to a technical problem? Yes, Deb Helper taught me that works well. I could talk about the DeGit main-to-rebase workflow just for a minute because that might be what you want. So I do not fully understand this yet because it's not done basically. But how this DeGit main-to-rebase workflow will work is your repository will have the upstream source and patches unapplied with a Debian directory and then each patch will be represented by a commit and that will get pushed to the DeGit servers. Now, obviously, you're going to want to rebase and move those around, especially when you merge a new upstream version. How is that going to work? Well, roughly, it's actually not going to be part of DeGit because it's not about source packages. Rewind to the upstream source and rebase your patches however you want and then merge that on top of the old patch queue. We don't have a whiteboard marker. You'll get a trapezoid pattern in your Git history. But then in the source package, the patches will probably just be squashed because the thought is, well, if the patch history is a beautiful series of commits on the DeGit server, it doesn't really matter what goes into the source package. So that's coming soon. So our video overlords inform us we have five minutes. So if you've been working up the courage to ask a question and you're really bored of DeGit, now's your chance. What's the point to have something to do with Git? There we go. So is there a plan for how to deal with packages in Git that are inconveniently large? So like data for games, that kind of thing. I'm thinking of open arena data here, which is enormous. There is no plan. It's a hard problem. But I mean, if we're using Git workflows, we've got good Git workflows for 90% of packages. That's enough for me. Is there a plan for dealing with those packages in Debian? I mean, do they work generally? I guess that's off topic. Sorry, I'm breaking my own rule about having something to do with Git. Yes. Maybe switching gears just a little bit. Has anybody thought about how we might indicate or better connect when bugs, something that I care about, are closed in specific Git commits to packages? I mean, one of the things that I keep wishing that somebody would make easy is the ability to graft on the Git tree onto a set of bugs so that you know that, yes, it was found in this version and in this commit it was fixed. So the BTS could, in theory track, the Debian directed graph of versions that depend on which they were mapped onto the Git tree at the same time. So if anybody thinks that's exciting or has comments on that, I'd love to see somebody give me patches that do all that. So it sounds like a job for Git notes, which are unfortunately sort of half implemented. I mean, they're getting better. So we did make some tools, which I don't think were very widely deployed, but we made some tools to experiment within the package Perl team to track not bug metadata per se, but patch work like this patches forwarded and attach that to specific Git commits. And that more or less worked from a technical point of view. I mean, it's a question of did we end up using it? I don't know, but I say we loosely. Did the team end up using it? Seems like a reasonable place to end, if there's anything else to say. Yes, sure. So Alvarez always has a question. Good. Run out the clock. Just an idea that popped into my mind. Would this be a good way of converting your latest SVN repository to a Git repository by just going into DGIT? Like, just doing DGIT clone and working from there? It would be the throw away all your history approach, which I mean, given the way most people use SVN, that's probably perfectly reasonable. I said most people, not including anyone in this room, of course. Thanks for coming. All right, thanks a lot. That was a great buff, guys. And girls and others.