 Hello. I'm Peter Green, co-founder of the Razvian project, and I'm going to talk about a bit of infrastructure software we wrote to serve one of our needs that may be useful to other downstreams. It's called Auto Forward Portaget. Its job is to automate the really tedious job of keeping your slightly modified packages up to date with new stuff from Debian. So, ok. Often necessary to modify packages that we get from Debian for various reasons, changing compiler flags, disabling test suits and that sort of thing. Occasionally actual code changes, but mostly it's just changes to make files and things. Debian updates stuff quite a bit. And merging that manually, it gets tedious after about the fifth or sixth time you've updated the same package. So we aim to automate this process. We can't automate every single case, but we can automate most of the routine ones. So the first generation of the Auto Forward Portal was written in 2015. It was based on different patch. Had a lot of Razvian specific assumptions. Was difficult to extend it to some of the slightly more complicated cases, because of the way patch kind of either succeeds or fails, and when it fails, you don't really have a good working point for the next bit of automation to pick up on. And also patch tends to fail if your change has been merged upstream. It doesn't like that. So I had some discussions with Ian Jackson, creator of Deget. I decided to re-engineer the system around Deget. And in 2017 I decided to try and make this more useful for other people, moving the Razvian specific stuff out of the code itself and into a config file. And now I'm hoping to convince some other people to try it out and tell me what they think of it. So some terminology that we use. An upstream project here is a project that supplies dev packages that you modify. So for us that's Debian, for another one it might be Ubuntu. The downstream project is a project that's distributing the packages you modify. Again that could be anything from a handful of packages you use personally. To a major derivative. And a version marker is a string within the version number normally at the end, although not always for reasons of maintaining version ordering. Which is used to identify your downstream packages. So generally it starts with a plus and followed by a number. Something like plus rpi is what we use in Razvian, plus Ubuntu is what Ubuntu uses and so on. Now in order to work with stuff in Deget we've got to get it into Deget. We've got to get it into Deget with a history of a shape that makes sense for the merge. So the first thing we do is we pull both the upstream and the downstream packages into a local repository. We use rep-rep-row for this. An auto-forward port of Deget has some code to convert it to package whitelist into a rep-rep-row package whitelist. So that rep-rep-row can import the relevant packages. Once we've got them in that local repo that's when a tool I wrote called pull2get takes over. I say Deget supports converting between Debian source packages and Git package and Git commits. But it can't by itself build a sane history for this task. Pull2get takes DSCs in a pull-style structure, i.e. what rep-rep-row has, imports them into Git repositories, building a history. We're stricter about history for downstream packages than upstream packages. For downstream packages we insist on having the immediate parent version. Whereas for upstream packages we'll import history if we've got it, but ultimately we'll skip versions or we'll import with an orphan history. Now there's an option that if a package we need for our history is not in the pool we're importing from, we'll go and get it from snapshot.debian.org. This is quite important because when you're going to merge your downstream version, which is a bit out of date with your new up-to-date version from Debian, you need that common ancestor version, otherwise the merge won't go sanely. Especially when you first add a package to the white list, you likely won't have that common ancestor version either in your pool or in your Git repositories. Pull2get will go and get it from snapshot.debian.org, import it and include it in the history. So now we've got the stuffing Git. Git's a pretty good merge tool. So most of the files will usually merge okay, but there are some problem files. Debian slash changelog will always conflict. We have a tool that handles that and basically groups all the downstream changes into a single changelog entry that's on top. You could use merge changelogs instead. That's mostly a matter of style more than anything in terms of what you want your diffs from your downstream to upstream to look like. Debian slash patches slash series, that also fails quite often. Not always, but basically from emerging points of view we treat that like a weekly ordered set. We'd like to preserve order, but we don't fail if we can't preserve order. I spent quite a bit of effort writing a tool for merging Debian slash control, because that was a place we were getting quite a few conflicts. Particularly things like if we were removing a feature from a package, which we sometimes had to do, we'd remove build dependencies, but then Debian goes and bumps the version in those build dependencies we removed. Merge conflict. So we automated that. Symbols files are another big one for us. Symbols don't come out quite the same on all architectures. Then Debian goes and rough and something say architecture not army L. They change it to architecture not army L, not risk v64 or something. So those fix ups, and there's a few others I haven't mentioned, get the vast majority of merges to succeed automatically. But sometimes there are conflicts that we can't reasonably automate the solution of. Particularly conflicts in Debian slash rules tends to be a big one. We don't want to lose all the work the automation did. We want the manual work to be able to start where the automation left off. So if the admit edge fails, we do something slightly ugly. We take a list of files with conflicts in. We put it at the top of Debian slash changelog, which ensures that things can't be built from until someone delete it from Debian slash changelog, because it makes it a format violation. And then we push that to a special branch in a private getReposit tree. So we add files with conflict markers in them to a git commit, which is dirty as heck, but it works. And then someone can pick that up, fix things manually, and then do a git commit dash dash amend so no one sees the mess. Okay, so we've now got a merge that looks okay according to git. And by the way, for those who aren't familiar with digit, digit does patches applied git trees. So upstream changes will also have been merged by this process. But we've also got the quilt series, which needs to be consistent with the tree. As I say, it's a patches applied tree. And sometimes it's not. There's a couple of reasons for this. One is because of git merge can handle situations that patch can't. As I say, a big one is when your change has been accepted upstream. Or sometimes the patches won't apply. Sometimes they'll apply, but give the wrong results. We need to fix that up. And the way we do that, again, pretty hacky. I need, I could probably improve this, but it doesn't come up that often. Most of the time the patches do actually apply. But we go through, we go through all the patches, we try and apply them to the upstream source. If they're fuzzy, we defuzz them. Because DPKG dash source won't apply a patch with any fuzz in it. Whereas quilt will. And if we can't apply to tool, we just remove it from the patch series. And then after that, Digit generates a new patch with any remaining changes to make the git series consistent with the source tree. This tool has gone way quicker than I thought it would. So we use Digit to actually build the source packages. Optionally, we can then go on and call S build to turn those source packages into binaries. This is important for Raspian because Raspian doesn't have any arch-aul autobilders. So we need changes files that have at least the arch-aul packages in and may as well have the arch-specific ones in. We also have the option to use Digit push so that people who import our DSCs can get real git history. Reason that's not enabled by default is that Digit push needs a server setup which is a bit of a pain to do. I know it for Raspian, but I can't expect people just trying out the software to have that infrastructure setup. So that option is disabled by default. Okay, now I've talked you about, I'm going to try and do a live demo of it. Can everyone read that? So first of all, we need something for it to work on. So I'm going to run some commands that are going to grab an old version of a package from Deb. Before we do that, let's just run the clean script I've got in here. So we're going to download an old version of a package from Debian and we're going to apply a patch to it to basically turn it into a surrogate for what would be an old downstream package. Okay, he's working now. I just had an internet connection problem. So now we've told it to get an old version of the hello package from snapshot.debian.org. Can you hear me okay? Yep. Oh, there we go. It's got it now. So we've got our rep-rep-pro setup. Pretty simple. This isn't a rep-rep-pro talk, so we're not going to details about that, but it's. And see we've got something to pull our packages from our upstream distro, which for this demo is Debian said. And we've got a config file for auto-forward port-a-get itself, which is in tilt slash dot auto-forward port-a-get. So. Mostly file paths this, so path to the working rep. Path to where we want to put the get repos. Path to the auto-forward port-a-temporary directory. Path to the output directory. What our local version marker is for this test, it's plus test. For Raspberry and it's plus RPIME, so on. What we call a revert marker. Sometimes we modify packages and we revert those modifications. Occasionally, due to versioning order, we need to forward port-a-reversion. I mean, I know that sounds odd, but it happens with particularly when you're doing patches to things in stable. Whether we want to invoke S build on our results. Whether we want to invoke do you get push on our results. Names and emails to use. We've got another option to push some of our get repos to get hub. Again, that's disabled by default, but that's configuration for that. Whether we want pull to get, which used to be known as DSC-der to get. Things got restructured slightly. Whether we want that to pull packages from snapshot.debian.org or not, you usually do. And the names of the suite. So there's the option to have a split main and staging suite in your downstream. We're not using that for the demo, but that's why the option is called staging suite. And what our upstream suite is called in our local repo. And finally, the name of, I mentioned the working branch before where we push stuff so that we can work on failed. So we can continue where the auto forward port or fail that. Okay, so that's our config. That made it in for time. Time is not working. We've still got plenty of time. So the next thing I'm going to do is that test package built, which represents an old downstream package. I need to import it into the repository. So that's the next thing I'm going to do. All these commands I'm pasting are coming from a tutorial file that's included with auto forward port or get. We also have a whitelist file that lists the packages we want auto forward port or get to work on. So that's in tilt slash dot auto forward port or get slash whitelist dot. And then the suite name. So whitelist dot seed. And there's also whitelist dot import, which is a list of stuff to import. And that's automatically updated from the other whitelists. You can see we've got one package in here at the moment, which is hello. Now we've got everything in place for the auto forward porter to run. I should be able to actually run it. Where's my mouse? You see rep rep pro running, then pull to get running, downloading the base version, common ancestor version. It's failed for some reason. It worked when I tested this earlier. I know what step I produced. I forgot the temporary directory actually has to exist. Okay, so it succeeded this time. Okay, so it succeeded in the actual auto forward porting, but hasn't managed to copy it to the output directory. It works this time. So as you can see, it's pulled the sources that changes forward from the old version of the hello package to the latest version of the hello package from set. Any questions? You're all being very quiet. Hi, thanks for the talk. How widespread is the use of this tool? I'm not aware of anyone else using it yet. This is partly why I've come to give a talk about it, because I want to convince other people to use it and tell me what I'm doing that's good or bad with it. It serves raspians needs. As I say, I'm not aware of anyone else using it yet, which I quite like to change. I'm also musing whether or not I should package it in an upload it to Debian, and if so, how exactly I should structure that packaging. Should I go with doing what Git does, this idea of having a master command that then runs all the different scripts of sub commands? Because at the moment it basically just runs from a Git checkout, so you just check it out and you run it from there. I don't think I want to put all my commands on the path that will pollute path quite a bit. So not exactly sure what direction to go in with that, and whether it's worth, as I say, packaging it and uploading it or not. I already spoke to you yesterday about this, but I'm interested in this tool, if it could be announced to also merge with upstream sources, because Debian is an upstream, but Debian has its own upstreams. One of my main use cases is automating the update of dependencies. Sometimes I package something for Kali, and it has dozens of dependencies, which are easy packages. The manual work to update them is tedious, just like yours, and if it could either use directly the upstream Git branch to update packages or possibly download troubles and build a fake upstream and use that for merging, it would be interesting. I think that could be made to work. I'd have to have possibly a bit more of a look at how do you get imports work. The thing is to make a merge work right, you need to get the right common ancestor, which for a new upstream version is the old upstream source. There are quite a few tools with a Git build package to create fake merge points between what you get from a trouble and what you get from a Git repository. It would just be a matter of tweaking, I think that would be the main thing, tweaking the tooling so it gets the right common ancestor version in place, so that then Git can do a sensible merge with whatever new upstream you've got. I think most of the rest of the stuff would more or less just work as long as the right common ancestor was in place. You might also need to think about what you're going to put in Demian slash changelog, but again that's as much an issue of style as substance. Those would be the two main issues, changelog generation and getting the right common ancestor point, I think. Other than that, I think a lot of the scripts I've written wouldn't actually be doing anything in that case because a lot of the stuff I've, a lot of the scripting is about dealing with merge conflicts in the Debian tree and if you don't have a Debian tree on one side there's going to be no merge conflicts there. The other thing that I wanted to mention is, well, obviously prepare a new source package for, well in your case, a new version for your derivative, but the other time consuming part is ensuring that package works. Well, you hope it will, but we have tools nowadays with auto package test to ensure that it at least does that to some point and it would be interesting as well to have your infrastructure runs those tests and just to ensure it and a flag package which might have an issue. Even if they build, even if they could be installed directly in the repository, maybe it's interesting to flag them saying the test no longer work, so maybe you want to double check before I upload it, that kind of thing. Yeah, I'll be brutally honest, Raspbian has run on it. The way I've managed to run a more or less full rebuild derivative on one person part time is by running on the philosophy of if it compiles, ship it. It may or may not be the best way to run a project, but... I understand. I just wanted to share it with others to give ideas of where you could go and where they could help you to make it more usable to other persons. Because you're looking for the use cases. The issue I always run into with test suite is unless you're intimately familiar with the software, it's difficult to tell what a test suite failure actually means. Does it mean that some bit of corner case functionality we don't care about is broken? Does it mean the package is totally broken? Those are generic problems. You can always keep your philosophy, but at least you have a supplementary data point to make your decisions. The tests are built by Debian, in your case the tests are written by Debian developers, so at least you don't have the work to create the test just to run them. When we run it in raspion, we do run it with a do s build option, which means that any build time test will have been run before we upload the results. So if there's a build time test that fails unless we've disabled said build time test, that will result in a build failure and the modified package won't ever get uploaded. Yes, but I don't have any questions, just to announce the build. Anyone else? You've got five minutes left? How usable is it without a separate instance of Digit? It'll all work, you just won't get nice get histories. It's basically the position, so it'll mean that every time your package is updated, then it'll go from Auto Forward Portagit to your repository, whatever that is, and then it'll be re-imported by Auto Forward Portagit to produce the next one. And if you don't have a Digit infrastructure, that import will be an import from DSC building, which works fine, but it just means you get a very messy get history. But the DSCs that come out will still be fine, the merge process doesn't really care what the get history looks like. So yeah, it will work fine without Digit infrastructure, you'll just get nice looking get histories if you have Digit infrastructure. Anyone else? You've got three minutes left? Okay, thank you Peter.