 Sir, I would like to talk about where code gets injected into what ends up as the Debian package. If you are not the author of the package, obviously the upstream is sold. So, if you are not authoring the package yourself, there is an upstream source which injects code into your package from time to time, Potentially, you could also have patches that are maintained by a third party upstream. In terms of the Linux kernel, I think of Linux's tree versus the ML tree, for example. In each one of these cases, you are not in control of the code that is being injected into the package. Next, you might actually have one or more features that you are implementing in your package which have not yet been accepted up to date. And finally, the other major chunk of code that ends up in your package comes in from your packaging approach. There are maintenance steps, there are often helper tools that maintenance steps use that get injected in here. I will separate out the Debian packaging stuff from the other sources of code because usually your Debian packaging across several packages has more in common with each other than it does with any of the upstream. For example, every single one of your dot-flags Debian directory has a file called rules, which is a main file if you are following policy. It implements certain targets that policy defines. So, just by itself, you have got a great deal of commonality between Debian directory in different packages. Maintenance steps, no matter what format you represent them in, I've seen post-int, dot-in and so on for people with autofocus. In any case, no matter which format you use, across packages you are probably going to represent your maintenance steps in the fashion. I myself find myself using a skeleton maintenance file in there, which handles all the optional parameters that maintenance steps are supposed to handle. And then I go on from there, configuring the special package. So, corresponding to each one of these code flow files, there are certain use cases, things that you have to do during the course of maintaining packages or whatever. So, if we start with, since we are talking about versions of work, we are going to take your upstream source and various different features and patches, etc. and put them in versions as well. Once you have done that, you have got to incorporate new upstream code, that's really the remain. From time to time, you might see major upstream version changes. This is significant in a couple of ways. Firstly, depending on how widely deployed your package is and what kind of package it is, you might find a need to have two major versions in WN and the same package. So, whatever workflow and version control mechanisms you use need to cater to the possibility of having more than one version of the package in WN and the same package. So, the other thing you need to track is, if you are maintaining your own patch sets and you submit them upstream and they incorporate it, you no longer have to separately maintain, however, as a branch or however you are representing that feature. If you are also incorporating a third-party maintained patch set, then whenever there is a new patch, maybe in response to a new upstream version or maybe they have redone their patch, you have got to be able to take that new injection of code and integrate that into your package. The same thing applies if you continue hacking on the feature that you are maintaining or if you may change it to the Debian package. I would also like to be able to maintain support for stable versions of my package. So, whenever Debian testing freezes, I freeze that particular branch or that particular package for repositories for all my packages and move on to newer ones. I like to be able to make it so that people who are following my code, my packages through the distributed repository that I use, they know that this part has been frozen and they try to import or commit to it. So, this was my contribution to the theoretical part of packaging Debian. I'll go somewhere entirely different and talk about other version control systems. Just give a short introduction so that we all are on the same page. My template can't take a second name. I'm sorry, Manoj. That's why it's only me on there. I have to fix that. Who of you has used distributed version control systems? I hope I'm not going to say anything wrong now then. They are considered to be hit these days. Everybody uses them. What I want to isolate now is which feature of these distributed version control systems is what makes them so useful, specifically to Debian packaging. Because it's not really the distributed feature. It's just that the features that we actually want happen to be implemented very well in all the distributed version control systems, but I want to untie that. So, I'll be looking a little bit here at cleaning up the mess. What is the snapshot versus commit-based version control system and also then of course talk about centralized versus distributed. So, here is a brief overview of the evolution of version control. We all kind of started out with SCCS, RCS on far left centralized file-based, no branching whatsoever. And then went over to the point here. Big Keeper, Git, and Mercurial, BZR and Monotone, which are very distributed, atomic and have proper branching models. I'm sorry, this microphone doesn't really like me. Oops. It wasn't enough space. Where would it be? I would assume it's probably right around where Darks is. It's probably not in the Keeper, Git, Mercurial, BZR thing, because it was there way before these were. So, it has some concept that these other working control systems didn't incorporate, and so it probably doesn't go into the same group. But it's also not Darks, but it would be like right around there. I mean, I'm really not trying to say that Arch is inferior by not including it here. Probably all used Arch. My only default Arch is that I have to type so many hyphens that my hyphen key fell off. But other than that, it's been quite powerful, and as a matter of fact, what we're talking about today is sort of using branching for Debian packaging. That is all Manor's brain work, and I, at one point in time, found Manor's page when he was doing everything in Arch, because I thought it was really cool concept. I forced myself to learn Arch, and then I eventually gave up, but that's just my limitation. So, we had on the left side, we had the CVS like version control systems, and what these are is conceptually. I mean, there are people who actually do version control systems for a living, and they can prove to you mathematically that everything that happens is sound, and so on and so forth. And there are people who do this academically, and they have the proper terms for everything. I do neither. So, I just kind of use the terms that seem normal to me, and when I say snapshot-based, then some people will say that no CVS was never snapshot-based, and SVN is, but we did ARRI's not, and so on and so forth, because it's probably a very overloaded term. But let's say it is conceptually snapshot-based, a revision control system that maintains the certain state of a tree at any point in time. And if you go between revisions, you are essentially jumping between different snapshots of a tree, between different states of a tree, at the point in time when these revisions were committed. These can be actually change-centric, which is what SVN is doing. So with SVN, a commit is actually a change set, and what the change set is trying to adjust here is that it's a collection of multiple changes that all go in together, either all or none, unlike CVS, where they aren't any change sets. There's really only this two single files, and they don't all have to go in together. So merging here consists of computing disks between two different revisions, or snapshots, and applying those, and then relying basically on the powers of the patch utility to find the proper insertion points for patches, and you will have to resolve conflicts yourself in the end. And another important thing is that a commit to any of these CVS SVN-like version control systems is really just adding a log message to a dip, in a sense, whether it's a dip or a snapshot. What you can do is you can add a log message, but that log message is freeform text, so it contains stuff that you, as the developer, find useful at a very high level, but it doesn't really contain any information that the version control system can use to identify this change or this current snapshot. How about the revision? The question was how about the revision number? Well, that's true. You can identify the revision snapshot or a change by revision number, but you may have made multiple changes between, and also as soon as you start branching, then I'll get to an example where this should be illustrated. I may not have perfectly explained what I mean with that, but I'll have a little bit of a graph that should illustrate it better. But maybe it's also going to be a little clearer if I end with a slide, if I'm done with a slide, where I talk about modern version control systems and note that I'm not saying distributed here, because none of this, I think, what I'm saying here is actually of distributed nature. But what happens is that these distributed version control systems actually are so-called commit-based, so there is not really a single snapshot of any tree at any point in time. Your tree is the function of whatever the base was that you started, plus the set of all the commits that you have applied. So I cannot say revision 517 on last Friday, 2200 hours, was this and that tree, because I may have pulled in a couple of patches just before that point in time, and Manosch may have pulled in these patches just two seconds after that, so at that given point in time, my tree contained the patches, Manosch's tree didn't contain the patches yet, and yet we are at the same point in time, we are sort of speaking about the same revision. It doesn't work that way. What is more the case is that a tree is actually identified by the set of commits that it contains, and merging different trees simply consists of computing the dissection of the sets of commits that have been applied to both trees, identifying which ones in the other tree are not already in my own tree, and then only taking those commits and applying them. Sorry for the German on there, I couldn't be bothered to remake the slides from a previous time, but basically this is what's happening with these modern version control systems. Every square that you have here is a repository, and those little black boxes are supposed to be commits, and what these distributed version control systems, I did display it a little bit distributed here, you can see there's no centralized entity, and what they require is that you can freely pass patches in between, so you'll see that the top developer has made a change that gets pulled in by the other one, but not by the one in the bottom, the version control system still maintains consistency across all of this. And this is opposed to centralized version control, which is CVS and SVN especially, where everything happens in a centralized repository, and every one of the developers has to check out and commit to that repository. Now interestingly, and that's part of the reason why I'm actually trying to avoid saying distributed version control system, because very few people actually use distributed version control systems in a distributed way. Who of you uses ADM with distributed version control systems? Okay, who of you uses distributed version control systems without any form of centralized server? Alright, that's about one-one, one-to-one mapping. Okay, I expected it to be somewhat different, because, well, Git is one of the modern version control systems, and it was developed by Linus, who has an immense amount of experience with the Linux kernel, and Git provides two workflow models, one of them is the completely distributed, everything like kind of flows here and there, there is no centralized entity, except people kind of accept that the Linus tree is the central Linux repository. But Git also supports having centralized repositories, Git.dev.oriminalia, for instance, gives us that ability. And so far, my experience with Git is very limited, but so far I haven't really seen a project, except the Linux kernel that does it in the distributed version, so far it's all in centralized. But let me get back to that example I promised a little earlier, to show why and how the CVS-like version control systems fail when you talk about branching. And what's happening here is that on the top we have a series of commits, that's basically the upstream branch, and then I branch two feature branches of that. F1 and F2, F1 does some development, and F2 starts, and in F2 I realize that in order to be able to implement the feature that I wanted to implement, I have to have one of the changes that has been implemented to part of F1. So I pull that dark blue change, I pull it into my F2 tree, and the light blue circles denote which of these commits actually have the changes that I just pulled in. And F2 is then done, I commit and I merge back into the main line. So actually this one should also be blue. And now F1, there's some further development being done on it, and eventually that feature branch is completed, and now we want to merge it back into the main branch. Now if you do this with CVS, your main yak, if you do this with SVM, then what's going to happen? Sorry? Well, I didn't hear it, but I'll just... It could merge it silently, but SVM unfortunately doesn't do that. Well, that depends. If nothing has changed, then it will give you that G state, like the merge has already been part, but that very rarely happens. I'm a member of a phone project, and we use SVM with about a thousand branches all over the place, and it's an absolute catastrophe, because SVM will actually say, sorry, the changes that you are trying to apply have already been applied. And especially if there's been a number of commits in between, then you get the conflict. Do you use SVM with merge tracking tool? Yeah, SVM is supposed to get merge tracking integrated in the main line 1.5 or something. So it will soon be able to deal with this a lot better, but so far it hasn't. And this is sort of an ecological niche that the distributed version control systems have picked up simply by tracking commits and not snapshots. So what are the features of a version control system that are actually relevant to Debian? And we think, Manus and I said down yesterday, and we isolated these couple of features. We think that branching and merging are trivial with the distributed version control systems. They're so easy, and you should just use them. Or once you understand how to use them, maybe become part of your workflow. The distributed version control systems can be centralized in an area, and if they are, then they're really accessible to contributors. So a contributor who on Friday night doesn't want to go out but stays home and then at 2 o'clock wants to start hacking an implemented feature and once the project hosted on India can actually with a distributed version control system just check out the entire repository without having any form of account and start hacking the entire weekend with a full featured version control system and commit and so on, and then in the end has something to present which may as well get them an account, but the time delay between flying for and getting the account hasn't really stopped them doing the work. Now I realize that SVK for instance is an extension to SVM or a different type of SVM that can do the same, but I'm not really arguing against SVK because it's going in the same direction here. And of course no need to track what has been merged where, the merge tracking which I've been parking about too much now. So it's time for me to actually get rid of these slides and have a little look in the command line and what I'm going to try to do now is show you how you can take any Debian package and easily convert it and I'm going to use Git and then show you what I mean with these branches and how easy it is to work with them. So of course I'm the kind of person that keeps all the valuable stuff for presentations in temp. So, yep? Can you increase the font size a little bit? Increase? Yeah. You're making it really difficult because this is FluxBox and RxVT. Can someone tell me how to do it? He's not at all. Ouch. So, I don't have those installed. Do you know any other fonts except for 10 and 20? We can try 12 by 24. 12 by 24. 12 by 24. I don't have to find that for the directory again. There we go. Okay, so here's a little software that I wrote for my book at one point in time with Hello World in Swiss German and actually gives you four languages and you can configure it and it's really nifty and has depth-configuration and translation to like a hundred and whatever languages, you know, with a very, very featureful but not quite new Hello World. So, I'm unpacking it as you can see. It's a very, very simple upstream tar ball here. So, what I want to do is really create a branch that has the upstream data in there so that I can just keep checking stuff in there whenever upstream makes any changes. If upstream uses git, I can just clone here. Git repository. But in this case, upstream doesn't even know what the written proposal system is. And the change log actually says that it was done like in the 13th century or something like that. So, I'll just simply quickly here. Initialize this package as a git repository and add all files. And then I can commit this. I'm just not going to do any commit messages for now. And do something like tag. This is upstream 1.1. I think that was enough. Okay, so now upstream is checked in and I can have a look at my list of branches and that's only master. I don't like master so let's go and rename that one because it is actually the upstream branch. Master turns into upstream. All right. Now I have one branch that is upstream and now I want to add the Debian directory which is in the git file. So, I just create a new branch which I can do with checkout which then also automatically switches into the new branch and say this is the branch Debian directory. And now in that branch Debian directory, if I call git branch then it will tell me with an asterisk that we're in the Debian directory. In that branch I'll just apply a patch and the entire Debian directory. Actually some people have pointed out to me that I don't need to be doing this tagging but I kind of like it and one of the features that Manos is going to show you he's going to do the same stuff that I'm doing with his workflow and he has an extensive set of scripts that help him. One thing that Arch does is you can seal branches so when a branch is sealed you can't commit to it anymore. You'll get an error message. And conceptually that's a very nice idea because I have previously, and I'm sure some of you have done the same been in a branch and committed stuff to that branch but that branch actually represented the latest stable security update and wasn't your trunk where you're supposed to do your work. So now you have to actually undo that change, port the commit into your trunk and it's just a lot of work. So if you had sealed that branch then nothing could have happened. Now with git tagging is sort of like the same because I tagged a branch at a certain point in time that means this is actually my released version and if I now commit to this branch after the tag I can very easily... I haven't changed the branch because the tag doesn't move but I can very easily take that commit and move it to the trunk even easier than with all the other versions of systems that I know. Okay. So now I checked in my divine directory and I can basically at this point in time build a thing and ship it but because we actually do have two different branches and if I look at the upstream branch I wish git had CO for checkout. It doesn't do that. If I look at the upstream branch then divine directory is not in there. So one of the things that Manage I think has come up with I'm not entirely sure whether it was your idea but I learned it from you is the concept of an integration branch and the integration branch is basically where you pull in all the changes that are supposed to be in the package that you release. So now I created this int branch and switched to it as well and I'll just simply pull in what I want. I'm going to... because this is already branched from upstream I don't need to check out or merge upstream into this I'll just simply merge my divine directory branch and then I find myself inside the integration branch with basically all patches that I've worked on so far and I can package this release tag and be done with it. Now if I actually wanted to add two more concepts that I want to show off and I hope I'm not actually using too much time two use cases of the ones that Manage identified earlier on one of which is upstream release is a new patch or it creates a new version and the other one is that I'm actually hacking on a local extension or I'm fixing a bug or something like that. So, for instance, the devian policy has changed and requires me to install the software in a different place than upstream has been doing. So far it's all worked okay but now the policy change is telling me I have to do something else. So what I can do is I can create yet another branch policy change branch and just notice, I mean, gravity David knows you're not the other day had a talk about it and he was all hyper about these branches because they're so easy to do, so fast to switch and you can throw them away and there's really no reason not to use them. So now I'm in this policy change branch and I'm just simply going to make this important policy change. I wish I could, because my legs are going to fall off very soon if I continue like this. Okay, let's make this important change here. This is the important change that policy requires. All shell scripts now have to end with a common X. All right. You have a new policy team. Come in. We have a new policy team. All right, all right. New policy, big things coming. And if you want it to be machine interpretable you have to talk to Mama. So minus A just simply says take all the things I've made. Usually Git has that staging area which it exposes but I don't really care about that right now. So now I've created a commit. I am actually in my policy change branch. If I now switch back to the integration branch for instance because I want to be publishing 1.1-2 so I can fix that gray block because of policy non-compliant. Of course that change is not in here. You see that in the policy non-conformant shell script because it doesn't end with an X. So now, preparing the next release all I can do, all I have to do is just simply say hey, merge the policy change in here. And so far I guess that's not magic to anyone of you. I mean you've done merges and branching and so on and so forth but the lovely thing is I can merge the policy change again. I can make changes to the policy change branch merge from that one into some other upstream related branch merge back at 15 new branches merge in between and then pull it all together and it will just say well, okay, those are the commits. You didn't need 32 branches for that because all I had to do was this but it just doesn't get confused. So now I have a policy compliant shell script in my integration branch and I can just package it and be done with it. Now let's see what happens when upstream releases a new version. What are we going to do? Well, my set of branches, every growing set of branches includes the upstream branch so I'm going to switch to that one and simulate a change, a very important one. That caused upstream to go to version 2.0 or something like that. There are actually tools for various version control systems which provide helpers to do what I'm about to do manually inject or whatever they're called so that you can take a tar ball and get all the changes into the directory which is rather easy to do, right? You just unpack the tar ball but the problem is that files that were in the last version which are not in the current version will stay in your repository because tar doesn't actually like delete files it didn't contain in the archive, fortunately. So these helpers scripts will actually make things easier for you. Now I haven't used the git build package here in this demonstration because you kind of want to know what's going on underneath but at least I do. I don't really like these high level helpers that do magic and when something doesn't work you don't actually know what it didn't do or what it did do. So upstream has provided a new tar ball and as a matter of fact they have adopted our convention that the changelog now also has to contain a comment with an X in it and it is 2.0 because we are also fixing those dreaded bashes. So I can have a quick look here at what upstream actually did I mean now it is almost as if I had just unpacked the upstream's tar ball over my existing data. So git status will tell me that the changelog and the file, the main file has changed I can use git dit and so on and so forth you can use this stuff. So I'll just accept the fact that upstream has done some major work in this package and I'll commit and I'll tag just so that I have a point of reference, an easy point of reference later oh yeah it said 2.0 Okay so what should I do now given that if I change to my integration branch then we're still shipping bashes and old not 2.0 type data. There are various ways in which one could go ahead now one of them is keeping this integration branch running that means now I merge all the changes that upstream has done into my integration branch I tag it after I release a new package and I just basically have one integration branch that runs alongside the upstream and simply pulls in all the changes that I need at all points in time. That's definitely one way of doing it and several people like that because the integration branch contains the entire history of the Debian package however sometimes it doesn't really make sense to merge upstream changes into the integration branch sort of over the stuff, over the changes that you have made for this package so in the integration branch are your fixes, your policy fixes and maybe fixes affecting the data that is not in the Debian directory now if you pull in from upstream that means you're sort of giving upstream priority wait, the other way around you're giving yourself priority because okay, so I'll just skip it then alright, just for completeness the other way, instead of keeping a long-living integration branch would be simply to say my integration branch is done now I'm going to create a new integration dash 2.0 branch and then you would have two separate branches it's nice when upstream changes a lot then you might actually want to have a separate branch because there is no real history between 1.1 and 2.0 but it's not always needed so in this case I'll just stay with a simple example and I'll just merge upstream's changes into my integration branch and I'm done with it so I guess I have many many... are you done? Are you ready to go on again? alright, I guess I have many many more things to show you and I spent at least four hours yesterday actually trying to work out the TypeScript that I could show up here of course I didn't actually use that TypeScript because I don't know where to put it but if I do find it one minute, if I do find it I shall make it public because it has a lot more examples and comments in it of why I do certain things and it also talks about rebasing which is a feature that Git users seem to like or hate and aren't really sure about and I haven't made up my mind but it's something you might want to think about and I'm not going to be the one to explain to you how it works but I hope I've shown pretty much everything that what I've been doing here maybe one last thing because so far it's just been punching in a couple of comments and maybe it isn't clear to each one of you what the benefit of all of this is especially because there are tools like D-Patch and Quilt which can also maintain patches one of the nice things I'm almost done let's go back to my policy change branch here and because that was actually created before upstream release that fancy new 2.0 release let's merge back all the upstream changes I could merge, I could rebase people find about that and then let's say I want to have that policy change actually integrated upstream well in that case I want to dip now Git actually makes it really easy to get dips between different commits because it doesn't really differentiate between branches and tags and commit IDs so when I say what's the difference between upstream and my policy change branch well it's not supposed to show everything I must have done something wrong I probably merged I probably branched the I think I branched the policy change of the integration branch or something like that or the one where yeah, try it sit down near the microphone and type but you can actually really easily create these dips between different branches between the upstream branch and a feature branch that creates it and yes, Dpatch and Quilt can be used to do exactly the same thing and as a matter of fact many people have argued that Dpatch is so much more accessible or Quilt is so much more accessible because you actually have the patches as separate files in the file system if you don't know the package you just go there you see a series of patches that are more accessible than having to understand Git and like trying to get it to list all the branches and then trying to figure out what these branches actually are there is something to be said for that but on the other hand how many of you have used Dpatch and or Quilt and have had a new upstream release that requires you to go and edit every single one of your patches because upstream has made a trivial change how many of you are using Dpatch and Quilt you guys must be doing something right because I'm not alright the benefit of using Git here is that Git can actually just merge across it has really really powerful merge abilities just like most of the other distributed version control systems because they are made for merging and they will actually track the differences between the current version and your feature branch without having to be statically encoded in a file that couldn't change because it's actually computed when you call Git Git and it's not static and fun alright, well that's it from me did you ask me a question, Stella? When you do want to change something like in a Debian directory, you usually want to test it do that like just write the changes in any recent branch that puts back the Debian branch commit and merge back or use something else in pilot you made a kind of elitic and a concept in my field for this one a sloppy branch a sloppy branch is a throwaway branch that I use for hacking and protesting and so if you needed to make a change you make it in a sloppy branch, make sure it works and then you take a change set the current change that works on and feed it back to the feature branch or the Debian packaging that you originally where it's supposed to go the sloppy branch is where if things go ugly you can destroy it away and rethink what you're doing that's one of my questions number one, number two what do you do about the Debian change slot what did I what do you do about the Debian change slot I mean, merging it works for most of these systems do you mean in terms of like how do I fill it out how do I merge it where do you create your change slot to be honest I'm not really using this workflow that much yet so I can't give you a I've thought about it a lot but I can't really give you a clear answer on that we didn't build this a lot because we had a teammate take stuff and get it right now it's pretty natural to merge stuff in terms of change options like if Julie will commit something and I can commit something on the branch I try to merge it from experimental to unstable I think I'm messing with myself I don't see if there's any current good way around it it does a reasonable job but it's on for a bit there's tools like that commit which I find very helpful and I've actually written my own even for back when I was trying to get in and use art written a little tool that would whenever there was a new upstream release then right into the change lock which branches would be obsoleted by that upstream release so these kind of integration that kind of integration is possible but on the other hand I think I keep pretty much everything in Debbie and change lock and then generate my commit messages that commit from there but it works for me I'm not sure about others it's horrible for lock check but other than that as we're talking about demo packages, how does the generated source package look like just the original table and then a very big and ugly image is that or do you have branches exported in any way so that the users or the interviewer or whatever can use the benefits yeah that's a very interesting question basically the way that things are done if you do it in this way and then use the standard TPKG dash source minus B stuff to build your source package then what you will get is the original tar ball if you made it and a div that merged every single feature branch into one it would be nice to keep these separate and there are various ways of doing it package 2 is supposed to give you the ability to have sub-directories and one of these sub-directors could be just your git directory and then it would have the entire history in the devian source package I actually like that a lot the other approach to this problem is HCT the canonical people they started to create something called the hypothetical change tool and it basically does exactly that they use the feature branches a lot I think they use pretty much exactly Manor's model and then generate separate patches for each one of them and actually then have them as deep patches in the source package so I'm sure there could be a much cleaner implementation out of this but so far we have what we have and that's what we have to work with I guess so I'm assuming that this will have it done yet right I have a question one problem that I always have when working with any kind of original control system and heavy packages is that the handling of the upstream power that should be an ethical cap around everywhere is somewhat not integrated you have to finally put the directory if you someone else pulls the posturing has to someone get the power to do so any suggestions? the problem is even if you tag the upstream and you don't make any changes the tar ball is going to be different because of all this meta information that it stores so the way that I've been doing it is that yes you need to have a central repository of your tar balls and I try to use the Debian Archive as much as possible for that so I just basically obtain the tar ball from the Debian Archive if it's there, if not then I'm about to upload it and then it can just stay in the parent directory that's one solution that's the problem HTE is actually supposed to hold this they have to really do the out of the magic for generating the same tar balls every time I've been told Blankstop HTE is supposed to be able to recreate the original saying be the exact tar balls every time I don't know how they do it with Blankstop how do you handle tar balls? how do you handle tar balls? I have nobody to ask I think I don't there's a question there can we do the question session later because there's so many questions do you mind if we add your question or reply to what he was saying let's make that the last one so it's really ugly but what the Kerberos packaging team what we do is we just check the upstream tar ball directly in the STK you blow your repository to pull the upstream tar balls but it means that everyone working from your reviews can blow a commentary at them some of the things that Martin touched upon I'll be exploring a little bit more detail in the next slide going with the structure so Martin kind of showed you the workflow kind of based on what I started off with I'm going to give you some context I'll do just what he did later with Arch but I don't know how interesting that is I worked a shell script that I essentially did line by line I can make it available online if you guys can do this so one of the things that is missing from the old version and whole systems and patch based systems is firstly decent handling of directory and filing if I have a file in one branch somebody makes a copy of this branch creates a new directory moves my file there and calls it by under name and make some improvements when I drag these changes back I want those improvements going to my file in the original directory under the original name I failed to see how patch and file can really handle that back and forth between different versions secondly I would also I like the fact that if I need changes to the permissions of the file make it executable or non executable and go back and forth I would like this change to be also represented in my version control system again CVS makes actually care whatever permissions your file used to have and if initially checked in was it you couldn't go back and forth and a useful feature that I find in having especially if you are using a real on-extra-goodness distributed workflow where there are a bunch of you in a team who are all collaborating on a package the ability to sign my branch or my repository and not have the signature be invalidated by a change in the future any version control system that relies on having one file in which all division information for one a file or a directory goes into like RCS and CVS used to have the form of e-file every time it was created in your version they added a jump on to the end of the form of e-file which means the signature was lost so how well how does mine workflow deal with the begin factory well first I like to have a separate category inside my report in my two terms Arch has a repository a repository is a collection of categories where a category is what you would think of the name of the software so a category is a package name a repository is a chunk of disk which contains a history of all the packages in there from the start of the repository until you decide to stream it and move on we have one repository for it to be in the link I had an edge repository now I am currently using the linear repository in the tutorial section I have created a brand new repository called DevCon 7 so I like to have one category for a package I create a bunch of branches like I can explain there is one branch from the only thing that goes in there stop that upstream injection there is one branch for every downstream distinct feature so if you are following somebody's practice there is a branch for it if you are hacking up something new there is a branch for it there are also things that we do for DevCon which are not really meant to be fed that they are usually small little things but changing part of it for the user local installed in Slack user there are changes that we make to make sure that everything configurable in Slack ETC stuff like that goes in the devian branch there is an integration branch where everything comes together so you have got a bunch of different packages the first time it hits your lobby branch or integration branch is when you know if these two patch sets actually are compatible or if something has to be changed in order to make them work together you can either make the changes back in one of the two patch sets or you can create a different branch that fixes up the fallout from integrating either one of those distinct patches into your development branch I have already talked about the Shopify branch as in testing one of the reasons I like doing it this way as opposed to having bunches of patches that you apply on the client every one of these branches can be built individually you can build straight out from the repository anybody can grab that build this and make sure that the upstream features that for the upstream behavior is just so every patch that is applied you can build that branch straight out of the report go there you can get the whole branch build it and see how the behavior changes this is not something that you get when you have got original source 50 different patches that are applied and the only thing anybody ever do like that is what you get in the end this way if I have to feed that patch it works great I can say look this is everything else that I have built here is this one feature I have implemented here is a clean patch against the most recent upload that we have done and this is what it does and it has been tested by me people like clean patches they don't want to get one patch that implements 30 different features all my dot search directory are branches of a common client this makes it really simple for me to do things that policy requires me to change for example even the policy doesn't quite require you to have 25 some sort of package I decided at some point to start doing that all I have to do was implement it in the parent of all these dot search directory run one command a little shell command that says for replay that change in every single directory of the 34 packages sort packages I maintained I went away came back from pocket and it was all signed, seized and uploaded to the policy case I didn't have to do anything manually that by itself is for me another justification for having composing my different packaging with upstream and dot flash there is the patch category I also found that I had a whole bunch of things which were common in my dot search directory stuck to how to what target dependencies are how to handle things like automates or translations and stuff so instead of replicating it all over the place I did what we all do I abstracted it away created a separate category and now it's there in common if I ever want to make changes there is only one place I have to go to okay here we get to the really part of this slide I'm going to show what my team did using IACS I'm assuming that since IACS is no longer fit and fashionable very few people here actually know how to get started using IACS since this is a workshop the most possibility that one or two people might want to play along so this is how you set up a local IACS repository that you don't need a network connection for just change my username for yours and ah the how my checkout tree lines have a special package which is what which is called packages it has a branch called Debian so in art parlance it is packages devian-1.0 that contains the directory structure which is how I build packages it contains one top level directory for every package I have it contains a config directory which has a sub directory for every package I have the config directory tells you how to grab all the pixel pieces to create a build buildable tree so it has my integration branch it has dot slash devian command dot slash devian command and the config directory makes you that no end user ever has to deal with all these things this is where I take to so what are the steps that I do for initial issue what is our wall from upstream I entire it it gets simply unpacked I change the directory name to be something that I like as opposed to something that upstream like I tell art to initialize this tree to become an art repository and then I import it I work my upstream branch I run something I just say tla tag upstream branch something else feature A, feature B, devian or whatever so I just these are things that take seconds to execute they are simple as tla tag source destination I create all my branches and I think they are very similar to what Martin showed you this is where the things start getting fun every single branch that I have right now is identical but I need to tell my integration branch that don't do whatever was just done to create this initial work on the branch because the integration branch already contains all the work so I do not need if I have 5 branches I don't want to merge 5 versions of the imported integration branch art has a nice little thing which is synchronized tree it basically says pretend that you want to it just grabs the metadata it grabs all the change log it grabs when it was done to it was done by putting it into the patch history everything is based on the patch history so now as far as art is concerned you had merged everything already talked about how you need to compose your building so the other question I am kind of galloping through this material and I see I am blazing over I can go back to my back part and I can show you but unlike Martin I have got all these things so I can just cut and paste so I don't have to type in all the command lines should we take a break the question when you say art which means I use TLA or Biles interchangeably it depends on what my fingers do there is some command in TLA that I use TLA automatically type in TLA there are other commands where my fingers type in Biles I have I can see no logical rationale that my fingers are using to choose either TLA or Biles so I am kind of at the mercy of the logic or the lack of logic of my fingers type in so far they are being worked with property and displacement for each other so you grab anything from it and it ought to work I think I will skip the TGM I think I will take a command line for a little bit longer I will tell you about why I selected art as opposed to going to the newfangled things like gifts or material or art is massively scriptable so for example every time somebody comes up with a new major origin art has I have three components there is category which is the name of the project there is branch and then there is a version so I like to change the version of my art every time a new upstream version comes out some upstream which don't release major upstreams more than once or twice a decade in action I go to the minor version I change whenever they change the minor version if you have got 15 branches it is quite tedious to make sure that you have done the made sure that you don't miss any branch got them all to the new version make sure that your integration branch knows that it shouldn't try to incorporate code changes because there will be no code changes all I need to do is I just say art new branch version package name, new version name that I go away it goes up by itself not only does it create a new set of branches for me it goes to where my working tree and re-names the direct re-names and make sure that I have got the latest version with the new version name and everything all sitting there waiting for it I also have a principle agent running I know weak security but here principle agent makes sure that I don't have to be present for it to sign each and every branch so I can just go away, come back 15 minutes later and it's all ready for me to start hacking if it was not a major version I don't have to do the art new version I should continue working do a strong laptop again then I unpack the new upstream tree and there is this neat little tool called Bars load DIR the equivalent one called TMA load DIR and I believe it has made it rain to Dyches and SVN and so on what the x underscore load DIR it will give it sit in the upstream branch, give it the name the new unpack tree it makes sure that the latest upstream version is what your branch now exactly mirrors the Bars load DIR has this neat thing of tracking file name changes so if it finds that a file name that used to be present and disappears and there is a new file name it may be in a different directory it puts it up and asks you is it a totally new file or is it just a remake so history is preserved even if upstream change the file name and I don't have to do anything about it it would and looks at sizes and makes heuristic guesses and so on and then you try to check if upstream has actually broken anything that you did so I create a new sloppy branch and copy my previous integration branch in there I recreate a change set that just happened with this new upstream branch in my experience for my packages 95% of the cases nothing breaks all the change set from one screen applies cleanly into my integration branch in cases where it doesn't know what to do and what to identify which particular patch it broke and then decide how I am to fix that so using this workflow doesn't actually work magic if there are incompatible changes of screen a human stem has to come in and decide how to fix that now assume that everything is okay because usually is there is no little script that calls what it does is it works into every single one of my branches finds out what are the patches missing from the upstream branch and it applies all the patches in sequence that have not been incorporated if in some cases I want to cherry pick and not pick something from the upstream I would already have told I am pretending to have that patch already so it goes and grabs upstream patch applies it all across my branches commits them goes to my integration branch applies the upstream change set directly and then tells I have to pretend that all the other branches have been synchronized and again I can just say I have to go away have coffee come back and upstream changes have all been incorporated and I didn't have to spend a second dealing with d-patch or twist or what have you even if there are points the patch would have a point if things have moved around a few lines I have had to work with packages that use d-patch and twist applying this in exchanges I can't for the life of me imagine how people manage to go through so much TV every time there is a little change can you say there is space between men and jay now I have you have got to update the change because the new york scheme version is coming you may need to do some other little things control file if the dependency has changed or whatever and then I commit the main thing about this is the commit hooks I have what seems like art create config which goes on figures out how many branches I have in my building it also looks at my upstream directly figures out how many branches I have and what is the latest facet for each of the branches grabs all this information puts it in file in the config directly so anybody just say art create replicate directory structure for this particular package at such and such a version the version that you just uploaded and I can happily look at this config file and reproduce my directory structure on your machine you don't have to ever worry about how many branches I had how they nest with each other all you need the config file but if you have a config file you can do that and actually you don't even need the config file which is even easier than that because the next thing it does is it creates a grab file a grab file is a something that can be dumped on the web any URI and you can give this URI to art you can just say art go to http artfordivian.org and there is a grab file and it will pull in all these things it will set up the build layout I talked about it will grab all the sources and mind you not all the sources that are mentioned in my config file might be from my repository it could be from Russell's repository from his tree of the divine policy order I can grab sources from wherever the original authors are if somebody else is creating maintaining that set I can point directly to this repository you just point art to it it will grab sources from all over the place and you are good to go finally there is a script called artford that goes in looks at the whole nested tree it walks down the tree and at the top of every art package it runs artford it cleans out the darkness art directly and art id directly and from that point on you run prebuilder or whatever build process you have for taking a clean divine source layout and working there I don't have a specific place where I keep upstream I just keep them in my build tree and I keep the local copy and just keep keep everything that I have ever built and uploaded over the last 5 years sitting on my development machine so I can always go back and check I think this is where my repair is right now I can go through and demonstrate and talk about a high level view and I can show you the command line that you need to use if there is any interest in this or actually I can open the group of questions we could also some people just wanted to try it and be around and answer questions if you are interested or if you are not sure how to do a certain concept if it is your first time is there a similar exporting tool for the gate to get the first directory? I'm almost sure there is good export it has good clean which cleans your entire directory except for the docket directory but I'm sure that the export command exists a bit as well so far I've seen an average version of the control system the main thing about the arch we're closed I didn't entirely understand just a bit about telling branches that are already been applied when they're not actually being applied to them could you go over that a little bit more because I think I got a little bit lost in the equation of that this is what the usage says so if I have got a new upstream version that came in and I applied it to say feature branches so the upstream changes are going to feature branches I applied it directly into my integration branch already because I did that in this copy branch and make sure it worked so now I've got all the code that came in in the new upstream version both in my feature set branch A and also in my integration branch but at some point if I had some changes into my feature branch I would like to import those changes into my integration branch so I just said that merged from my feature branch A it would try to add in this upstream code which is already in my feature branch again so you use TLA sync free and give it the feature branches patch number so that I already have that code would you please just go and get the patch log from that branch and add it in so my history is complete once you have your patch log in the integration branch any time you want to merge it in I'd know it that wouldn't exist because you have the patch log there you came to identify the fact that the revisions that you've merged independently in two different branches are in fact the same revisions and figured this out for itself this is how we tell it that they are the same revisions I don't think that you may not be understanding exactly what any git has exactly the same command as podgit patch and what happens is that in the meta directory arch and git keep a track of all the patches that have been applied all the commits and every single commit is a file an object and git but it doesn't really matter and what this sync archive I think of whatever sync tree does and what git fetch does is it simply pulls in all the changes but only the metadata of it so that you actually know about the changes you can dip against them you can look at the logs of someone else's changes but your local source tree hasn't changed at all so now what you can do if you wanted to be masochistic about it you could iterate through every single change that upstream has made look at the log file and say merge only that change without ever having to go to the upstream repository because you have already pulled all the data that you need to bring your local tree to the same revision as the upstream in your meta directory but it's not applied on the working directory and that's kept track of the version control system it knows that it has those patches available and it knows that it has to apply them to that range and it knows that they are available not entirely I think it knows that they have even been applied all you need to do is look at dot art directory at the top of your project tree inside that you will see that it has some directory that corresponds to each of the repositories you have pulled stuff from you follow the thing and you will see that it will patch log that it knows about so and that is how when you run arch it goes and looks in your dot art directory to figure out what you already have I'm particularly happy to go through and I have renamed the example that might have looked but I can't pronounce what package you are using in German it is not pronounced as it is written so I have this political word and I can read you to how I would set this up to understand this word alright can people hear me or does it need an algorithm so this is my that's where I am user local short arch this is the top of my tree I have this I have two tar balls sitting here packages is where I do my building so this I have created a directory for hello world because I knew that I was going to be working here but the directory is empty so I could have created it here I go back and look at my cheat sheet so the first thing that I am going to do is copy my upstream tar ball so now I have got the upstream tar ball sitting at the root of my hello world package I go to upstream and it and again it names something that I can't pronounce so this has been one of the major problems with getting hard to integrate it into the rest of the mindset or I might say this is the problem getting hard diffused into the developer community which has two darn mini dashes that you need to type so this is my upstream tar ball nothing will happen here the first thing that I need to do is I need to tell arch that I am going to be importing this package and set up the archive name is what I specified and at this point no code has been put into the repository it has just recorded the fact that a software whose fully qualified name in the arch universe is going to be you see my email address it is the repository for depthcon 7 which is going to be thrown away as soon as they get back home and this is the fully qualified name in this name space the next step is to tell arch that this particular directory tree that we are sitting in is actually the initial tree for my package so yes it goes and then if I do a link it gives me that you have got these bunches of files here which should normally be sourced but you haven't told me that they are sourced files yet so if I do ta link minus tree tell me only files which are un-tagged it will give me this which allows me to do things like tlay add I run the tlay link thing again and there are no problems because the tlay add stuff took the output of the other tlay command and added all those files so I would never do this in cvs and I have been using cvs since it was a c-share instructor how many people knew that cvs used to be a c-share instructor before a bunch of us jumped on the authors that they used next and made them change it to a c-share instructor okay, at least one person I am surprised part of the problem with cvs was I think that it started as a c-share instructor and it never actually lost that I talked about how tlay or add uses the logs as the primary way of keeping track the logs contain a whole lot of metadata so the first thing if you ever do before importing is to tell it to make a log okay in order not to shock people with e-maps I am going to use a more backward editor to edit this file people also hate the fact that art uses lessons and other strange symbols between filings the way that art does its logs is supposed to be like an email message it encourages you to write long emails like commitments so you people I have seen people write a couple of pages of rants and comments telling stories about their new feature writing commitments to it however since I am using e-i this is a log file now here and I can just say if I am not running it x because I can't run it x I can show you all these things my printable agent I am just going to flip into x done an aptitude update last night I could have shown you all these things between glory and x so I just imported it the field import is recorded here like that you need to type in all this stuff nobody actually shares as long as I assure you that these commands actually do work what you hear more about is what the commands are what it is like to do that so I just did this tlac import part the next thing I would do is go once I have got this little word upstream that's the only thing I have and so what I am going to do is there are these ones says of there are four task commands I am going to create four branches and I am going to check out all these four branches and I am going to do this in one go maybe you should make it explicit that tag in arch is the same as branch they are absolutely treated the same so when I was tagging previously I was not creating branches in git but when he does tlac tag that's actually a branch I am not interested I am not interested actually this case was confusing to me too I didn't know how to make branches and if you use bars you can actually say bars branch which is an area for tagging so my fingers unfortunately go to tlac because these are early things that I did go to the devo branch and then I can say I am using bars so at this point everything is identical I have made no commit on any of the branches if there are very many changes I could just have done replace here the bar get which is the same as git get and I now have this brand new directory here I create my devian directory here since I don't actually have a stilted devian directory I will just create one from scratch and I am in the proper device which is a dot flash devian told I would just give the devian directory 1, 3, 4, 3, 4, 2, greater than changing of the code bar greater than new new change log tlac commit find l something that I actually do hate about art and again I am not going to my little x thing in order to run the critical agent maybe it is better for you to see how you have these things when you are trying to do things that you don't want it to do then thankfully art has a local you can do a rman rf to print ti r on hello world and go away please thank you I will get you tired of this too the gif that we did out of this the gif.gc that is part of the source package that you can upload it doesn't actually have anything in it but it is in art it is just a straight gif from the art team that is true if you are using art export I used to not use art export and then it would be an honest to goodness checkout of an art tree and you could run your own art between whatever version you wanted assuming you had a network connection but right now if you use art export as recommended by a tiki master who latented me a couple of times it is one giant difference of thing or something why aren't some questions about the giant dumps for your gif suppose somebody comes along and needs to make a change to your package and then in you security you get hit by a bus while carrying your laptop and all of your art repositories disappear hold on a second my art repository the final repository has a mirror on art.debin.org every time you make a comment they go to my server which is in texas and they also go to art.debin.org so if I get hit by a bus while carrying my laptop you won't get using it but suppose you are all dickering of where you put all your backups and somebody goes and makes a patch anyway against this unified debt right well if you do I have tried to provide point if you go to the pt the practice tracking system you will see that there is a little link there which gives you different links one is for the browser you can browse every single thing in all my art repositories there is a second link that gives you a graph file if you download the resources for you don't need to know art there is a tla url and it will go around do things and it will give you all the different branches in a frame which are just full directory trees and all you need is less and cd and win or something better maybe and you can go and browse the sources sure one big dip is not itself useful if you want to make changes and you don't know where which feature set it belongs to or where it should go I had some thoughts about that giant dip to try to make it perhaps a little more useful quickly go through and instead of generating one giant dip use your history of commits to generate each small dip and you can make them all together with a comment between each one which is a richer past story which can actually come from there are a couple of problems with that first of all is the ordering in which the things are applied I don't actually have to worry about which order of things are done in my workflow because there are all accepted branches in 19156 if you actually wanted to create a series of patches then you want to figure out whether or not to have that screen as you and secondly there are some things that is my personal advice and it doesn't apply generally well I suspect that anybody who wants to take over my packages their first obstacle would be understanding my rules file understanding art is the least of the problems I would actually be more interested in discussing this workflow concept we have to pick up and I guess we can do the discussion afterwards is this going to occupy the order? yes Mako will be in here and find us there is one last announcement I want to make I did pull up a mailing list which is called bcs-pkg basically trying to unite all the people who are using version control for packaging so far it's a dead list there's not really much going on but I do have some people already on there from Gen2 and Fedora as well because they're basically facing if you're interested then I'm just going to say it out now it's too much trouble it's not yet on any Debian server so you will find it on my list server which is lists.mata.net and then slash actually you don't have to do anything that will get you to the index page and then you can find bcs-pkg in there and for conceptual questions like how do we do oryx.tar.gz and how can we better improve dibs I think that would be a good mailing list to get things started so if you're interested please subscribe and then I guess any other questions before we break off lists.mata.net and then just hit enter and it'll get you there at least you're used to alright well I thank you all for your attention thanks Manoj for having this talk and well if there's any more discussion then let's go with it out in the hallway track and other than that have a good evening actually anyone in what stock may I'll be up in the lounge hydrating up so if you guys have time right now we can just continue the question session one floor straight up