 Next up, it's going to be helped by Guido Gunter. He's going to talk about what a build package can do for you. Welcome. So thanks for the introduction. So I want to talk a little bit about how you can maintain Debian packages in Git and how Git build package fits into the set. So just before we start, who in this room already uses Git for anything? So OK, that's about everybody. And who uses version control system for maintaining Debian packages? OK, so you're in the wrong talk. This is a schedule tutorial, actually. So OK, then maybe we just can turn around this a little bit. And I just go through the slides. And if you have some questions, you can just interrupt me. And you can maybe go into details. But it makes no sense that I just give a tutorial on how to import packages and all these things when you know all about that already. OK, so just a short overview of what I intend to cover. After a very short introduction, I just want to go through how to import and update and build packages with Git build package. Then I want to talk a little bit more about how to generate change logs from commit messages. I want to mention at least some more tools around Git build package and Git and maintaining packages in there. And I want to find out some things that can certainly be improved in Git build package. So first of all, what is the point about version console and distributed version control system? So in the Git repository, we saw different possible decisions, like different possible trees. And the relationship among them, which are the commits, which are the group points in this picture below there. This will come in a larger version later. And the commits each have parents which kind of build history of how to get to that commit. And you can branch in that repository and merge back the changes. And this actually works quite nice in Git in contrast to some other version control systems around. You can mark, you can tag certain possible decisions with the yellow fields below there. And all what you do there first happens in a local repository and you can commit changes and undo changes. And after that, you can publish the whole repository or branches so that other people can just come from there and share work with you. Overall, Git is dense-based, efficient, and fast. I think the packet over here did some studies when you speak org into Git from CVS and it turns out that basically the checkout isn't much larger than the Git repository the whole history. Is that right? Okay. And so what are the recurring tasks with Debian source packages? So at one point or another, you get to import the Debian packages to the source version control system. So this might either be because you want to maintain your packet in a version control system or because you want to do an NU and you want to be able to get patches easily and don't want to see all versions around or you want to do a upload or you want to do some sponsoring. So what I'm using that for a lot is just to spawn the request from mentors.debian.org I just import the version into the version control system so I can easily compare it to the last upload and see the differences. Things we regularly do is change, build, and test and then find a failure and change, build, and test again. From time to time we import the upstream version we add, remove, and update patches. We have to write a change log before we release the package and from time to time we go back forward or do stable updates and things like that. With all of these, having things in the version control system really helps and through the talk we see most of the topics covered here. So first of all, you want to start and import a package and the tool that chips with the package is fitinfor.tsp and you can just give it a local .tsp file which will import the upstream version and the Debian Disk into the version control system but you can also give it a URL and within minus minus download option it makes it quite easy for example to patch a source package for mentors or maybe from another Debian derivative just to import it into your control system and to have a source and look at it. Or you can just give a package name to it. In this case it will use apgets to download the package via the sources entry in your sources list. I use that quite a lot if I just want to have a look at the package and want to maybe quickly fix something and submit a patch. Let's take a very short look at how this works. So it just imports the package and imports the upstream table and then we have a new directory here with a git repository and we can look at the commits and we see that we have just here an upstream branch that's the green one and the Debian branch where the Debian development happens on which is in this case called master and we have tags here for both versions and in addition to that we have a pristine tab branch that is used to regenerate the upstream table in exactly the same way as it was downloaded from upstream by storing the binary diff. This pristine tar itself is not part of the git bill package. It was written by Joey Hess and it's a great tool to reproduce the upstream table from the version control system. So what I use quite frequently is this also works incrementally. So once you import a version then you can just import another version on top of it. So we can just do that again and pick up another version and we'll get this again. Then we see we have the new upstream version here the upstream branch advance and the master branch advance and this is the type of case which I use for example for sponsoring uploads from mentors.dev.org So I have a base version and then I just want to see what did somebody change and can review the changes this way quite easily. So there's also another tool which is git import DSCs note the additional s at the end here which is used to import multiple DSCs so if you have the history of a package in source format you can just point it to all the DSC files and it would go ahead and use TPKG compare versions to sort all the version numbers and import them one by one into your version control system and this way you get the whole history of the packet displayed there. So you can also pass pass it the minus-minus-debsnap option which uses debsnap from devscript which goes to snapshots.org downloads all the snapshots it finds there and imports them into your version control system this can be quite handy if you take over the package from somebody who didn't use version control or where you can't find the version control system for that and then you get as much of the history of the package as possible and can just have a quite easy look at what changed between the versions we can see how this works in practice too so I'm doing this from local because it's just a bit faster and it basically looks the same as the last one but just import both of the DSCs in one step yes, sure you just have to add the package name at the end yeah okay, the question is how do you use the debsnap option and the example is missing that there should be a package name at the end so you have to pass a package name like in that case python-statutile and it will fetch all the stuff from there so I'm just skipping about the git commands because I think you know all of them so once you have the package in your repository you want to do some changes and that's usually something so let's make just a trivial change which remove some commands from the readme file and commit that now you have made a commit in your version control system but you haven't made any changelog entries and since you anyway have to do commit message when you commit things into your version control system that's actually not really a point in maintaining the devian changelog separately we're on master I think yes this was just an example so in order you don't have to keep all the you don't have to maintain both the devian changelog and the commit messages you just can go and tell gitdch which is another tool from git build package to generate the snapshot for you the changelog for you sorry and what it basically does is if you pass the minus-minus-order option it looks into the changelog and looks for the version number that is on top of there and goes through all the looks for the same version number in the version control system and then adds all the commits that it finds there to the changelog and the minus-minus-snapshot option is to manage to manual the version number and now it created a new changelog entry right here which has my commit message right here which is the comment and because I gave it the minus-minus-snapshot option it put in a kind of a snapshot banner into the changelog to point out this is not to be a released version but this is a testing version I want to use and it for that purpose it also incremented the version number over the last released version which is here but with it till it decremented the version number so the version number of the snapshot will have a lower version number than the next to be released version at least hopefully so and you can if you do more commits you can just repeat that you can also repeat that without doing any commits and it will just keep on bumping the snapshot number right here and if you're done doing all the changes you can say well I just take the long option you can just say I want to release that and then it will remove all the snapshot crap and build you a package a changelog entry as you expected so that's what I basically cover I just skip over building the package actually because that's just calling git build package at the end you can certainly pass minus-minus-gits tag when building the package so it will create a tag in the version control system for you then you might want to import a tarball from the local file system which when in the upstream version comes out and this is just done by import oric and then you can either give it the tarball and it will import it or you can use minus-minus use can in this case it will just invoke use can to check if there is a new upstream version and download it and install it and commit it to your version control system and merge onto your Damian branch if I do have network we can actually so let's say I just had to release that by committing that change use use can and it will look if there is a new upstream version in this case there is one and it will import that into the version control system and now we have that new upstream version here and it's again being merged onto master okay so there's from why would you at all have the upstream sources in git so many people using subversion don't want to they just keep the upstream sources totally separate and just keep the Damian packaging in the Damian branch I for myself find it quite convenient to have the upstream sources because you can just fetch patches from upstream apply them build the Damian package with that and see if that works or if it breaks anything you can just experiment as you can easily with a distributed version control system and since git is space efficient it doesn't really matter much if you have the upstream sources in there and it's actually very convenient to compare version and support changes and last but not least upstream might be using git anyway so if upstream is using git you just clone the upstream repository and then you create your Damian branch in there and do the packaging and in the future you just go and pull the new versions and then you have the upstream sources and the Damian branch together again so how can we handle killed patches the the killed patch system is quite popular it's becoming quite popular in Damian it puts the patches into Damian patches and applies all the patches that are in there hi sorry I had a question about just the previous slide about working with upstream that has there that is using git since Damian technically distributes based on tar balls and not every upstream that uses git releases tar balls that are clean snapshots of upstream's repository do you know what I mean they might ship a tar ball that's been generated after an auto-recont for something how do you recommend combining both upstream's git and the tar balls and the Damian branches so you basically have to do some DFSG clean stuff on your tree or something like that no no I'm not talking about DFSG free, not DFSG cleaning I'm talking about upstream has a revision control system that they use but then when they build their tar balls they they take a snapshot and they run auto-recont on it and then they bundle it into a tar ball and so the diff between the tar ball that they distribute which is technically are a ridge.tar.gz and then the Damian package is different than the diff from the DFSG am I making sense here even if they do distribute a clean tar ball the checks up on the tar ball don't generate because you don't have pristine data information to recreate their original tar ball if you just pull in their grid branches no you're asking the wrong question really if you have an upstream that's using git and you are cloning and tracking the upstream git then when there's a tar ball release there's nothing that prevents you from generating an ridge.tar.gz out of that tag or that branch in your git repo there's nothing out there that says that you have to go grab the tar ball that they built that has its MD5 sum and have that be your ridge.tar.gz what we care about is the ability to in the repo to make sure that we keep building against the same ridge.tar.gz in the upstream release there are a lot of people I know that really like to see our ridge.tar.gz match an upstream release tar but you're sort of mixing concepts if you're tracking and working with the upstream git and then trying to figure that out if you do want your ridge.tar.gz to match upstreams take a look at the readme.source file in the Debian directory of either the open drive source packages Sam Harven developed a script which uses git what essentially what it does is you pull in the upstreams tagged release which usually doesn't contain the auto conf generated stuff then you download upstreams release tar ball and then you run a bunch of git magic which essentially unpacks upstreams release tar ball, commits it and set tells git that it is a merge with upstreams release tag and then all the right things happen from that point on inside git. So one other thing to kind of piggyback onto what Bdale was saying is that the other issues that there are some upstreams that use git and just don't release tar ball period and in that case there is no upstream tar ball to go to so generating it out of whatever in some case when they're doing if they happen well some of them don't even tag they just develop on git and that's it so I just make a date tag and release out of that and that's it but git build package covers that basically because you point it to the upstream tag and if it doesn't find a tar ball it just generates one for you so the only thing you have to make sure is that you generate the same tar ball again and again so if you have generated one I'd recommend to commit it back with pristine tar so you just make sure you get the same pristine tar ball but that's like yeah and the only other thing I throw out is that if you if like me for some packages you're both upstream and the devian maintainer just do the approach I was talking about take the ridge dot tar dot gz that results check it in the pristine tar and then point people to that as the distribution tar ball if they want to build it somewhere other than devian okay so going back to the kill patches so this is actually it's a quite popular thing in devian and all the other built tools we have basically supported it's like that help is supported with minus minus with quilt option CDBS has the patches with quilt and of course there's a source format 3.0 killed which which uses this natively and so many packages you download will have the devian patches directory which matches just these two things like having patches in devian patches and having a serious file and what if you just want to want to update these patches or put them to a new upstream version and comes with a very simple patch queue handling system which just uses basically get killed import and get format patch to redo the patches so you can just go into the source directory and say gbp minus pq for patch q import and we go to devian patches take all the patches import them onto the patch q branch and then you're ready to go to modify these patches and once you're done then which and you have a separate branch and you can always rebase to master the devian branch and to drop patches or to merge patches or whatever you want and once you're done you can just call go back to the master branch and can call gbp pq export which would then serialize the patches again and build your series file which you then can commit and and and build the package you can then if you have this patch q branch and you import a new upstream version then you can use gbp pq rebase to update these patches against the new upstream version um this is just using git rebase internally to take all the patches that are in the patch q branch and after importing the new upstream version then applies the patches one by one against this and if there should be any merge failures then you can just fix them and you can start any patches already applied it will just skip them and drop them and once you're done that and you don't need that patch q branch anymore you can can just drop it and um as I said it's really a simple patch system to get a patch tree quite easily without any additional setup costs it it's not meant as an alternative to topkit or tnt or something like that um it's just to just there to handle these kind of things easily um another thing about the source format 3 and the kill patches is that it by default it applies the patches to your source tree and if you do so you have after the build you have a patch tree and you not always want to have that because if you then do git status or something like that then the source tree isn't isn't clean anymore and there are actually several ways to um to not have that one is the quite recently added minus minus unapply patches option to dpkg source so in in your source directory you can create a file which is called debian source local options and um if you put the unapply patches option there then after the build it will just unapply the patches and your source tree will be clean again um the other alternative is just not to um not to build from the debian branch but from the patch queue branch by just calling gbpq import before the build and what other people do is just using minus minus git export directory which is more like svn build package like behavior which exports your whole seed source tree to a different directory and then build in everything there and then copies the result back um I'm using people the most of the time so I'm not using the last variant okay um we have right before we have seen that um that git dch is able to to generate commit messages from to to generate debian change log messages from commit messages and this is what we have here is just a type of commit message which has the commit ID right here and the committer and the date and then it has the description of what happened and git dch is actually able to parse some meta information like you can add things like thanks and the colon and then list all the people um that you want to thank them and you can have the closes which is even more important where you can just add the bug numbers this bug closes and you can um this is actually a regular expression so you can also put lp for launch pad there or something like that and once you generate the change log it will put all these information together and um generate the closes right here for you and we'll add a thanks line here this makes it actually a little bit bit easier to separate the the actual commit descriptions from some additional information you want to have there um and it features the id length parameter which I find quite convenient so it if you set that in a configuration file or parse that um to git dch with a number it will take that much tickets from the commit id which is which you can see here it puts them in angle brackets here and um it takes the first 7 digits of here and puts them here and the idea behind that is that it makes it easier for somebody to to track back your Debbie and change log changes to the version control system so I find myself often in a situation where I think well that's a quite interesting change in the change log I want to know how somebody did that and in the end I have a hard time just cloning the repository looking at it and and then finding I can't identify a single patch or a set of patches which which did the change and so by adding the commit id here is you can really uniquely identify the commit in the version control system um so that's what I talk about and if you if you have a vcs git link in in your devian control file then you basically have all the information to find the commit itself and um this is what a little cgi does I wrote some time ago which um just takes the devian change log and parses the change log and um then let's see if that works and then goes through it and just add links here to the um to the version control system and if you click on that one and git devian.org should be responding and then we should see somewhere okay it seems the network is okay there it is so um then you can just go through the commit messages this is all a little bit old school I'm working on a new version at the moment which uses a little bit more like Ajax and whatever but it's not been released yet so you can just hover over the commits and all this stuff um hopefully I have something working at the end of devcon um okay so how do you configure git build package there's lots of options to set and um you have a main version file which is the gdp.conf file in etc git build package this is side-wide and all the changes will affect all users and then you can have one in your home directory of course and you can have two of them in the git repository you can have one in the .git directory so all the changes you made there are will stay local to your repository and then you can have one in the devian directory and so since you push them out to the to any remote repositories um other users cloning from your repository um will have the same configuration this might be helpful if you choose different branch names or different tag patterns or something like that um so what can you set there you can just configure there all kinds of things like tag and branch naming patterns you can set there if you want to sign text or not if you want to use pristine tar you can set the build commands like pbuilder um you can just configure there if you want to export your source tree you can specify any import filter so if you import a tar ball you can say if you want to exclude some files or not and git build package like svn build package and other tools features hooks so you can have a pre-build hook and a post-build hook which does things before building or after building and you can actually have a post-tag hook which does something after you successfully tag the version um and should you ever get confused about what the current settings are you can just just run git build package minus help and the help will show you some of the defaults for most of the options um so some more tools around git build package which ship with git build package is a gbp clone and the gbp pull these are just two tools that just make sure just when you clone from a remote side that your pristine branch pristine tar branch is set up correctly that your upstream branch is set up correctly so that when in the future you pull from there everything gets forwarded there um as it should be an example post-tag push hook which I use quite frequently this is um so if you build a package and you're finished building and you're up loaded you usually have to make sure you push the changes into remote repositories because other people you work with will see the changes and in order to not forget about that I've just put in a small hook which pushes out the changes to a remote repository after a successful build and a successful tag so which is quite basically the same than a release and there's another script there which is gbp create remote repository which um just easy to set up remote repositories so if you have created a new package and if you have have that in your git repository and you want to publish it then you can just use gbp create remote repo which will go to the remote side create the repository there and push the changes in there so you don't have to do that all by hand anytime and it kind of defaults to the call out maintainers infrastructure on gitdebian.org so if you just color rating on packages this will just kind of work out of the box then there are other tools which I kind of mentioned already or not all of them first of all there's pristine tower which is responsible for regenerating the upstream table and then there's git pbuilder written by Russ Albury here which is a setting up pbuilder there's a nice git hook by Xenius which you can put into the remote repository and when you push changes there it will go through your commit messages and parse all these commit messages and see if there are any bugs closed and if there are any bugs closed it will send out mail to the devian bug tracking system mark the bug as pending and put back a link to the version control system so users can easily have a look what's changed there and of course there's the central infrastructure gitdebian.org powered by and where you can put your packages so whereas git build package not doing great it's not doing great with multiple tar balls so there's actually no real support for that at the moment there is a patch which I still have to integrate so this is kind of work in progress for at least git build package but it's definitely lacking support in git import oric and these things git dch should make better use of dch itself so you can easier do any views and because and uploads would be a backwards org or qa upload this is something I'm actually working to at the moment and it should probably improve support with complex patch systems like topgit or tnt or something like that but since I never used topgit on a project really myself I have no really good idea on how to proceed there so any pointers would be very welcome okay so I'm already almost quite finished git build package has a wiki page and it has a manual, the manual is actually shipped in the package itself but if you want to go there the online version you can browse it there I'd really like to say thanks to all the git add scripts and pristine tar maintainers because without them git build package wouldn't be any useful it relies heavily on these tools and thank you to all the bug reports there were many of them and they've suggested great features and especially to the OCaml maintainers which filed very very nice reports so are there any more questions and comments? I wasn't really listening because I was doing the sound but I'm pretty certain you didn't mention things like cow builder and pee builder and all that which fit really nicely with git build package and enforce your build depends so it's worth looking into combining them unless you did mention them I was fiddling about with the levels well actually that's what the script from Rust is doing so you can set a builder command to git build package you can say use that and you just git pee builder I think it's called and then we'll just do the right thing yeah absolutely but it's worth exploring the adding that if you happen to be using this and haven't used cow builder yet then it's worth a look at that too for people that realize that they plug together so easily it's what a point was worth making but maybe not so I have an issue with git build package from the standpoint of running it on a machine where my home directory is mounted by NFS I don't know if anybody else has run into this it just doesn't cope very well with that has anybody else run into that? I don't know so I should check that I ran into it a while back and you know my solution was on those machines I just copy the repository over to a local directory and just work from there but I seem to recall and the issue why I didn't particularly pursue it too strenuously was I wasn't sure if it was an NFS problem or a git build package problem but actually I'm using git build package on NFS a lot so we at work we have many of the packages in git and so we have the well actually and the home directory is NFS mounted sorry and that doesn't it doesn't cause you a problem I'll try it again and then maybe I'll send you an email if I have a solving issue with it I wanted to say that it's rather a git issue because git assumes that all its work here and the objects are in memory cached and with NFS that doesn't happen so git rather has some performance issues on NFS but otherwise for a small project I use also git build package on NFS and it works I wanted to ask a question so I've never used actually git dsh because the workflow seems counterintuitive to me if I understand correctly from your presentation you're using git dsh but you're not committing the changes to the changelog and then you're using git build package with ignore new and at the end the auto will remove all the snapshot intermediate and only you commit only that exactly because it helps you if you want to merge different versions like the other way would be to change the changelog every time you make a commit and then this makes it harder to to merge between branches so if you do the changelog at the end it's quite easy thank you I have two questions first is it possible to keep only the devian directory in the is there a feature like merge with upstream yes it's called git overlay in git build package but it's nothing I have used personally a lot so it's just as far as I know other people are using it like that okay and the second question are there any hints for for example in devian python module teams we have a lot of packages is it possible to somehow keep track of all of them at the same time or every package should have separate repository I think you would probably use a separate repository for all of them are there any other tools that help maintaining multiple packages so we should if we want for example I don't know check out every package from team repository we have to write our own tool to I don't know check out a list of packages and check out later I think you can do that with sub modules just keep a list of all of them and then get them but I have not used sub modules extensively too so I maintain shore wall and upstream upstream release is about a half a dozen packages and I wanted to keep them all in a single git repository so if you look at the git repository for shore wall which is on source forage I have hacked up a shell script that does it maintains a devian branch and pristine tar for each of the individual packages all in one git repo it's probably pretty easily customizable for a package set other than shore wall another comment there is also a tool called MR that can handle multiple repository like that so it's easily configurable so it's made by joy s I think so it's called MR a couple of waiting for the camera so a couple of things one for git pbuilder there was a request that I make a devian package of it a while back that seems silly to me it's one little pearl script if you're interested in chipping it with git build package or we should talk basically about whether or not you want to just include it I think there is actually a bug for it and we had just two issues to resolve maybe we can just do that and include it would be great the other thing is the thing I mentioned earlier for both kerb 5 and open AFS we wanted to use the upstream tar ball so that you could double check PGP signatures on it for example for security software but still merge with an upstream tag it seems like it should be reasonably straightforward to include that in git importer rich basically instead of committing the contents of the upstream tar ball directly just as its own commit you have on the command line provide the upstream tag and then in that commit mark that commit as a merge commit and make the other parent commit the upstream tag and then basically everything works pretty well so another thing for us to talk about I saw that there's a git SVN but there's like five or six steps that you need to take to convert a subversion repository to git are there any scripts or tools to just automate it and do it the right way so nothing that I'm better at so I've made a script like that for converting all OCaml packages to git so if you go to the on the wiki look for the OCaml team page there is a link to converting to git so it's quite a hackish yes so there is a page with all the manual steps so it's pretty hackish and actually what it does is so what was the most practical was to generate a shell script that would check out SVN important git and it uses a git import rig and so on to convert so you get a repository that work for and so it's so there are some hardcoded paths to OCaml but you can easily change them and I guess you can so it's converting from SVN to git workflow any other questions? then thank you for your time