 Good afternoon everyone and welcome to the next talk given by Ian and Sean about Git dep rebase a new tool for Debian packaging Thank you. Hello. So Sean written and I are here to present a New git workflow tool for Debian packaging But first before I tell you about it I need to show you where it fits into the ecosystem of Debian package management software On this slide We have you the maintainer on the left hand side just will off the left of the slide and on the right We have the Debian repositories The slides will be online You don't need to photograph them. You may have heard me Plugging digit once or twice. You should all use to get Digit publishes your git history so Debian's users can use it But that's not actually what I'm here to talk about today Git Debian base does not need digit and digit does not need git Debian base Git Debian base is an alternative to GBP PQ and to git DPM and to other tools in the same kind of area like git PKG And of course, it's an alternative to using raw quilt Git Debian base is a tool to help you manage your git branch Containing the Debian version of the package you maintain Git Debian base helps you maintain a useful git branch with the contents you need for building and uploading Git Debian base is mostly concerned with maintaining your Debian Delta Debian Delta Q that is the changes you make for Debian to the upstream parts of the package Other tools tend to call this your quilt patch series But with git Debian base, there is no quilt and you don't work with patches Instead there is a series of commits. So we prefer the term Delta Q Git Debian base is primarily intended for Debian package maintainers Although you could use it outside or alongside Debian It does not deal with building at all you use whatever existing build tools you like Nor does git Debian base deal with source packages or a rig troubles does not do uploads Of course when you actually upload to Debian you need to produce a source package getting a source package is of course as easy or easier than with other workflows Usually it is done automatically for you by DGit push source So usually you don't need to concern yourself with .dse source packages even when uploading to Debian Of course, you can share your git branch on a service like Salsa without building or uploading Then you don't need to deal with source packages either Git Debian base offers a standard git Rebase workflow where you edit the whole of the source code for your package Including your changes to upstream files and your changes to packaging including your changes Sorry The source code for your package including your changes to upstream files and your changes to packaging all together The experience is very like using Get plain git rebase to edit a topic branch Delta queue editing that is editing the upstream parts of your Package can be done at any time interleaved with packaging work As far as I know there are no other tools that offer these features Both GBP PQ and get DPM required to switch to a separate view to edit the delta queue Some tools have a specific function for git cherry pick But with none of them can you just use plain git cherry pick or git AM on to your usual branch at any time? With git debris base, you can just edit the code and commit it with git in the completely usual way Specifically at any point you make make commits to upstream files and commits to packaging in any order So you can just cherry pick from upstream you may make fix up commits and use the git rebase auto squash syntax to have them Automatically folded in by the next rebase if you wish you make make mixed commits containing both changes to upstream files And changes to packaging files Of course You can always directly edit the source if use a plain git merge workflow and non quilt source packages For example as described in the de get main debris base. Sorry de get main merge tutorial man page But that does not maintain the Debian Delta as a broken down linear series of changes And in the source package such a merge based workflow squashes all the changes into a single Debian changes patch So supporting maintaining a delta queue that is a linear series of changes is what get debris basis for Also unlike git DPM and some other tools get debris base has no entry metadata So it can't get out of date or desynchronized or need any manual changing or fix up as I said Unlike GBP PQ and git DPM. There is no need to ever switch branches get debris base only uses one branch to handle all your Debian work Of course, usually you will have an upstream remote tracking branch as well So if you are working in multiple Debian releases back ports for example You'll have branches for those but it's only one branch for each line of Debian development and no temporary branches or alternative views With git debris base you can always immediately build binaries out of your working tree with deep package build package or whatever other Build tool you prefer and your working tree is never made dirty by git debris base or any of the other tooling Because your working tree always has the delta queue applied it is never dirty by patch application Because there is no metadata. You can never get a metadata conflict Because git debris base treats the quilt patches in Debian patches as an output and handles them entirely Automatically your tree is never dirty by the generation of patches and you never need to read diffs of diffs And the final part of my plug With git debris base Git blame and git log on a file work entirely properly For example, if you do git log on a file from upstream which was changed in the Debian delta queue git log will show the Debian delta queue commits preceded by the upstream history. We'll show you an example of this in the demo If you run git blame, you will see a correct indication of which upstream and or delta queue commits introduced each line Or for a file in the Debian directory You will see a correct reporting of which commits in the packages packaging history introduced each line With git debris base, you never need to use the quilt program. You can mostly ignore 3.0 quilt source format Not having to learn about 3.0 source format is really good for newbies Particularly for people from other software development communities who don't know about Debian, but usually do know git Unfortunately, it's not possible to paper over the cracks completely You will still get trouble if you make changes in git which 3.0 quilt cannot represent Hopefully that's not too often On the other hand when you use git debris base with 3.0 quilt, the generated 3.0 quilt source package is perfectly pretty with your delta queue commits converted nicely into patches just as other people consuming DSCs have come to expect and Finally, of course, git debris base is compatible with DGIT You do not need to pass any quilt mode option to DGIT and you can always upload right away All necessary bureaucracy is done automatically when you say DGIT push source So that concludes the marketing spiel. I'm going to give you a bit more detail About how it works and Sean will be doing a demo later There are some important details. I'm going to be glossing over So if you actually want to know what's going on please read the Reference documentation, particularly the section 5 man page. Everything is fully and formally defined there So this slide shows a likely situation Which you might find in the middle of an editing session There's a lot of stuff off to the left-hand side Which we I've left that off. So we've got space to see what happens when you do some editing The horizontal part near the bottom here is the Called the breakwater. This branch contains unpatched upstream source code Plus the Debian packaging in the Debian sub directory does not contain any Representation of the Debian Delta Q so it doesn't contain any of your commits to upstream files And it doesn't contain anything in Debian patches in the example Commits A and B are packaging work The Debian Delta Q sits on top of that in this example. There are two Debian Delta Q commits I've pulled them one and two because that fits nicely on the diagram These are commits touching upstream files in the diagram your current hit your local master branch is at commit to So your tree contains the patch source code plus the packaging ie. It is your actual patched source package You could build it with deep because build package dash uc dash B to produce binaries For testing you can get grep for things and be told where they are even if they're in the upstream source files But introduced by your Delta Q You can get log dash capital G for things to be told where they came from and showing the relevant commit whether that's upstream or one of yours Okay, then suppose you want to make a change which edits upstream files and files in the Debian directory a Common example might be a change which edits on upstream file, but also adds a changelog entry I'm calling this C3. The reason for this name will be clear in a moment Now with this commit your tree is of course still fine. You can build and test it right away But suppose you want to tidy things up In particular, you might want the new upstream change to come before patch two Perhaps just because it's tidier that way or perhaps because you're about to change Commit to to in some way that makes it depend on your new upstream changes in commit C3 So in order to rebase and reorder the patches you would run get deb rebase dash I which is very like get rebase dash I you get the standards get rebase to do list editor and You see in it what looks like commit C3 and as you can see here in the diagram I've already reordered that to come before change to So you say okay go and assuming there are no Conflicts what you end up with looks like this so you can see that commit C3 has been split into two commits C prime which contains the changelog change and Three prime which contains the upstream change the upstream change is now in the Delta Q in its proper place C prime the packaging part of your new commit has been pushed into the breakwater This is the general scheme of things We have a fast forwarding breakwater containing packaging and unchanged upstream files It doesn't have a ref to itself Instead it is contained within your master branch and each time you get deb rebase the rebase starts on the breakwater So what about a new upstream version? To rebase onto a new upstream version you run get deb rebase new upstream Get deb rebase expects the upstream code in the form of a git commit of course Actually by default it hopes to find a tag named after the upstream version number But you can tell it explicitly if that's not right Get deb rebase arranges to include the new upstream source into the breakwater and rebases your Delta Q onto that So there are new commits here on the breakwater Firstly the at sign is a special merge That folds the new upstream source code unchanged into your breakwater branch This special merge is called an anchor merge The most recent anchor merge is the backstop for rebase processing by git deb rebase The second commit is simply adding a change log entry for you. That's done automatically Sorry, I've lost my place Yes, right. So having provided the new base Which is this commit D here It rebate it uses git git deb rebase uses git rebase dash dash onto to rebase the Delta Q Onto the new breakwater If you didn't ask for an interactive rebase and there are no merge conflicts, that's it You now have the new upstream code with your rebase Delta Q Of course, if you're going to upload to the Debian archive You'll also have to make an a rig table of the new upstream If you're using the workflow I've been describing so far, that's generally just a single call to git deb a rig so Indeed, let's consider an upload to Debian and let's imagine you made or obtained a suitable a rig table There's a certain amount of bureaucracy to be done In the usual case of an upload with D git. This is all done for you automatically. So you don't need to worry about it But it's useful to understand what's going on. So I'm going to go through it a bit Firstly, you're going to publish your history So your history has to be made fast forward from your previous version of the package to achieve this Get git deb rebase will make a pseudo merge Our pseudo merge is a merge commit which takes its contents from only one of its parents You would make one by hand with git merge dash s hours If you wanted to make your head fast forward and know that all the wanted changes from the other branch are included That's the command you would use But generally you don't need to do that manually in this example Get deb rebase had recorded the previous branch state so that it can make the right pseudo merge Your new branch is derived from the previous Branch you had so it's right to declare that it's fast forwarding that doesn't lose any changes The branch with the pseudo merge is suitable for pushing to any git server. You could push to salsa say Secondly When you upload a 3.0 quilt package the contents of debian packages need to be right. Sorry debian patches needs to be right Again, that is taken care of automatically Commit is made adding a patch representation of the delta Q to debian patches You can ignore these auto-generated commits After uploading you'll want to push your branch to salsa if you have a team repository there That makes sure that all the views of your package are up to date so that other members of your team won't accidentally base their work on an old version You can just push a git deb rebase branch which has had the pseudo merge made called a stitched branch with git push It's a normal fast forwarding git branch If you want to push without uploading that's fine, too Git deb rebase stitch will just make the pseudo merge for you giving you a fast forwarding branch suitable pushing to salsa or wherever After upload next time you come to the package you can work directly by adding commits on master If you want to rebase or you just want to tidy the branch up you can run git deb rebase It strips off the bureaucracy commits These remain published of course, but they are removed from your own master branch If you made any commits on top of the pseudo merge those aren't in my example here Or maybe pulled such commits from salsa or wherever it folds those back into the breakwater and the delta Q So once again, you have a nice delta Q to edit Git deb rebase makes a note of where you are previously So that next time you want to push or upload it can stitch the history back in with a new pseudo merge At the start of this walkthrough that ref was indeed present the FFQ prep master ref you see at the top right of the slide and I kind of glossed over that Okay, so let us think about the new upstream. What if one of your delta Q commits doesn't apply During the upstream rebase So IE the change you made doesn't apply to the new upstream source code. I may be a bit small That's a like a rebase conflict output from from git So a git rebase users will have seen this kind of output before It stops at the first commit which can't be applied in the new context and it asks you for help This looks quite bad. Of course, it's it's not good but this is an irreducible aspect of Maintaining a delta Q on on top of a moving target. Sometimes you'll need to fix up conflicts At least with git debris base you get the right tools to help you fix it up Some of the other workflows can involve trying to reserve month merge conflicts during quilt apply or fix up conflicts in diffs That's much less fun Also get debris base new upstream is quite low commitment Imagine like on the diagram here git debris base has applied commit one and stop because it can't apply commit three prime Now if you decide this is too difficult to deal with today You can just say git rebase stash stash abort and everything gets put back the auto-generated special breakwater merge and The change log entry are discarded leaving you just where you were before You've wasted no effort because everything you're throwing away was auto-generated There's one caveat I should mention Right now if to get debris base branches diverge Because different people edit them simultaneously It is not trivial to merge them again the data model I'm describing does not currently allow general merge commits If git debris base encounters a normal git merge It will stop and fail In and in the general case sorting out such a merge is not a trivial problem GBP PQ sometimes handles this kind of situation by expecting you to merge the actual patches I you can end up resolving merge conflicts in diffs Other tools don't always handle this well either. I have some ideas about how to do better at this So watch this space But for now you your team should coordinate to avoid creating diverging git rev debris base branches Get debris base will help you with that by often spotting when divergence is about to occur and warning you about it So that's enough explanation I think it's time for our demo now. So Sean if you're ready. Yes Right Is it can you see get K? Yes. Yes. It's that okay So this is a package of mine called helm. She's an emacs add-on and I recently switched it to use git debris base So let me just walk you through some of the commits that are here at the top of the branch Here is where I merged With get rid did rabies new apps there with git debris base new upstream I imported version 2.9.7 and then if we work our way up from there We have commits like this one which are Delta Q commits So if I just pull that up which just adds a file to the upstream source called changelog and then we have That's another here. We have another Delta Q commit which hacks the upstream's readme file to make it a bit shorter Another Delta Q commit which messes around with a shell script. Then we have a packaging commit So this one is just touching Debbie and changelog as you can see here we have a Mixed commit which is touching dot git ignore which is an upstream file and the Debbie and changelog and Then an auto-generated commit which is which is the one of the hexagonal commits made by git debris base generating Depatches and then these three commits the top. Well, sorry. That's just a packaging commit These two commits at the top are the pseudo merge. So making this branch fast forward from my previous upload Now one thing that's a bit unusual about this example is that like this is all mixed together and In the previous slides you saw that the Delta Q was always at the tip of the branch That's just because I didn't debris base before I uploaded and But that's actually completely fine because I'm gonna do some editing now and debris base will sort it all out And you'll see that happen Okay, two other things to show before I start editing Let me Count the number of patches. So there's a that's kind of a hairy git log command All it's saying is show me all the commits not touching Debbie and since two point nine point seven And you can see there's four of them. So right now we have these four commits and Let one more thing we wanted to show you was a git log on a particular file So this is an emacs git thing, but it's just running git log. Don't Don't be put off Here is the git log for the file read me md You can see all of the upstream commits and my little Delta Q patch on the top which apparently I made three years ago I guess this is an old package And that like just works you just have the log showing the upstream commits and the Delta Q commits Okay, so let's do some editing now that we've seen that so remember right now. We have four Delta Q commits I'm gonna add a fifth commit to save time. I actually have this in a stash So I'm just gonna pop that stash here And so now I've made a change to the change log and to an upstream file. Let me show you that with git diff So I added a comment to the upstream source and I added a Debbie and change log entry saying that I did that and Let's just go ahead and commit that ordinary way to use give So yes be more excited about helm All right, there we go. That's committed and let's take a look at what that looks like over in git k So we now have a this new mix commit at the top of the branch So as I mentioned earlier, this is a mess, right? We have these mixed commits. We have these auto-generated commits we have these packaging commits and Delta Q commits interleaved but get rabid rebates can just fix that for us. So back in the shell get deb rebase dash I and We end up in a git a git interactive rebase edit session So let's suppose that this new commit the last line Let's suppose that that needs to come second in the Delta Q for reasons unstated. Well, that's easy We'll just move it up just like a normal git rebase Commit the rebase Okay, off it goes. We've so says we rebased. All right. What does that look like? Well, let's refresh git K Okay, so now we have something that looks a lot more palatable So let me run through these briefly We so we now have five commits in the Delta Q before before we now have anyone and Here they are they start here Yes, they do so we have. Oh, no, we don't they start here. So here is the first Delta Q commit It adds the change log like before Here's my new commit that we just added You can see it only touches the upstream source now The change log bit has been split off into its own commit And then we have these ones these old Delta Q commits at the top and you can see that all of the packaging commits come Before the Delta Q so here is the other half of the new commit touching Debbie and change log and get every base split it out and reordered it perfectly nicely and also note that the Auto-generated Debbie and patches commit has disappeared because it's auto-generated. We don't need it while we're editing so it gets got rid of Okay, so that's actually it if we wanted to upload this change now I could just type do you get push source and it would go to the archive It's as simple as I literally do that right now, but I'm not going to because I'm a responsible maintainer But suppose that we wanted to push to salsa This as it stands is not a fast forward of what's on salsa because it's been debris based But there's a command to fix that get debris base conclude which says Stitch me back so I haven't pushable We run that and then I have to restart get K. I won't explain why But in order to make it something that makes sense So now we have exactly what we had at the previous view those eight commits are still there and one commit at the top Just you don't know jet. So it's pushable and It's as simple as that That's everything that I wanted to show any anything else you want to say about this. I think that's everything. Okay, great Thank you, of course Right, I just need to plug all these widgets back in Okay, so thank you for the demo. I'm glad that came off nicely. I think we've got the slide on again, right? So just like wrapping up Get debris base is available in testing and stretch back ports and is in good shape The since early versions it's been battle tested to help with security updates to the Debbie and Zen packages Which are quite an exciting test case With a lot of patches and a lot of difficult stuff to do and the documentation is comprehensive So no doubt the user interface and documentation will improve and new features will be added and bugs will be fixed And indeed you'll see we're referring you to the version of the tutorial man page from unstable That's because we've just done some documentation updates But you can start using get their views from stretch back ports or testing or unstable right away The best starting point is probably the tutorial man page D get main to debris base, which is in the D get package And if you want more information or you need help or you're just curious We're holding a workshop Tomorrow morning. I think that's tomorrow Where anyone is invited to come and help get help with get debris base and also with D get if you want questions using D get So if your questions don't get answered in a moment or you want us to like Workshop your problem do drop in it's a drop-in session And also of course I should refer to the reference documentation Get debris base 5 has the data model and all the terminological definitions and get debris base 1 is the command line reference But you don't want to start there unless you really like reference manuals. You probably want to read the workflow tutorial Or come to the workshop So thank you all for coming and I think we've got a reasonable amount of time for questions anyone You want to come down and use the mic, please? I think that's much easier for the video team You can come if you got a question you can come down and stand by the mic in advance that might save it a time The one thing you mentioned for somewhere in the middle is that when you have a team and diverging commits That is not dealt with the tool, right? Right. That's not the approach That happens right now is not brilliant. Usually unless To to unless both of you have been editing the Delta Q separately you can normally deal with this by Using a manual git rebase to rebase one of the sets of changes on top of the other and that avoids Producing a merge commit. So if you you can do also do often get pulled dash dash rebase will often do that for you So It's not completely it's not completely unworkable. It's just not ideal Do you plan to add better support for this? I mean kid is distributed. So I'm right, right? Absolutely Absolutely out. I mean I'm on the train and working. I cannot do permanently push and right, right? Absolutely, absolutely. So you can certainly do that right if you if you're working on a train like that and So it depends kind of what kind of editing you're doing if what you're doing is you're like working on the packaging And adding new commits on top That's really easy because you can just rebase those patches on you know You can get pulled dash dash rebase and then you don't make a merge commit and it's all completely fine but if you edit the Delta Q Non-aditively non-aditively right if you if you kind of reorganize the Delta Q commits or drop patches or stuff Or change the commit messages or something like that and then somebody else also does that Right now there are no tools. I think that will that will sort that out for you in a reasonable way I have some theories about how to improve this I think probably what I'm going to do is add code to deal with some of the easier cases and make that a bit smoother So that in easy cases the merge will be fixed up for you and the hard cases You might very well still get like error messages and then then you'll be left Having to do it yourself, but one thing you can do is you can check out both Get debris, but you can check out both parents of the merge and run get debris based on them separately and to get like Get everything nice on something that you can look at something look at and then you can reason about and then you can like Fix it up yourself Right if that happens to you, please like ask me on IRC I'm dizzy it on free note and of TC and or maybe Sean will help you out. Yep, SP Whitten Yeah, okay. Thanks. That was my question. Is that it? No other questions. Everyone's reading the man pages They are quite comprehensive you could you could disappear into those man pages But we read the tutorial don't read the I mean if you like reference manuals great if you're curious, but Where do new upstream releases fit in? We only saw packaging branches in this. No, no, no here. This is a new upstream release. Ah, okay, right? So so I'm in this diagram V 1.2 is the new upstream release And what's not shown on this diagram because it's way off to the left is is like There was a like a v1.1 So you do a you do a new upstream release with get debris base new upstream Let me take that off Right so it it automatically merges everything together and rebases your patch queue on top And it will automatically drop patches that have been merged upstream say they'll just vanish because get rebased as that If mostly it will do that right if it will do it if get if get rebase will do it then get debris base Right, right. I'm consistently amazed by what get rebased manages to do by itself. So You won't be typing a quilt dash n dash r to remove the thing that applies upstream all that rubbish Right and if you do have one I mean I had this with the Zen security updates if you have a like a really complicated patch queue and you know That you need to drop these three patches because now they're an upstream and they all depend on each other So it's merge conflicts if you don't drop them You can just say get debris base new upstream dash I and now you will get an interactive rebase with the list of the The commits in and you can just drop them there Good, right. So I'm expecting you all like typing frantically trying this out. We have a Conversion tool. Yes. Yes That's what I used on this helm package like Before the point at which I started the demo basically a few weeks ago I just typed get debris base convert from GBP because I was using GBP before and it fixed everything up So I should mention there that it's Not a good idea to not know whether what you have is a get debris base branch or a GBP PQ branch because if you have a GBP PQ branch Then and then you run get debris base on it It will just throw away your patches and now you have no patches and you'll have to dig them out of the get history again But we do have a command convert to GBP if you for some reason you tried it out. Yeah. Oh, right I didn't know about that. Yeah, it's it's it's it doesn't make it fast forward because it can't for reasons But it does work and you can Get merged HS hours if you really want to It's used by the test suite Okay, so well, thank you all very much and I'll let you get on with hacking and see you at the workshop