 Okay. Welcome again. The next talk will be about Britain and the talk, the speaker is Anthony Towns. Howdy, thanks for coming. This was originally a full length talk and I got asked to turn it down and when I did my slides yesterday that didn't turn out so well so I might be going a little bit fast. I guess most people here are non-native English speakers. Can you stick your hand up if you don't speak English natively? Yeah, so the first section will be not that relevant so hopefully that'll get you used to my accent and whatever and you can cope with when we get to the complicated stuff. So I'm Anthony Towns. I'm to blame for a lot of stuff from about ten years ago. Most of that bit's not relevant so I was DPL of TP Master blah blah blah. But more particularly I was the original author of Britney so it was originally in Pearl and then Pearl and C and then Python and C and now it's thankfully been taken over by other people so there's a sat-solver version that I don't think actually gets used and there's a new version that's in pure Python so it doesn't have the horrible C stuff and I think there's a guy in the back there that's responsible for that mostly. More recently I was a release engineer at Red Heart so similar job, less political stuff, more technical things and obviously all RPM stuff instead of Dev stuff. I was in the Australian office and I worked with the docs team and the storage team and pretty much everyone else. If you're looking for a job and are interested in working with things that aren't Debian, I totally recommend Red Heart. The first day I joined there was a huge flame war about licensing so I thought right at home. So one of the things about working at Red Heart is that unlike Debian it's a big company so they actually care about all the boring legal side of things and not the copy left licensing stuff but are we going to be risking millions of dollars of money if we have bugs and whatever else? So one of Red Heart's customers is people running stock markets for instance. And so they care about audits a fair bit so if Debian releases a package and there's a bug then we fix the bug and that's fine, everyone gets on with life. At Red Heart if you have a bug then someone might come back years later saying hey there's this bug and we hold you responsible, we need you to fix it and Red Heart needs to be able to go back and kind of work out okay yeah they're telling the truth who really did ship that bug or no they're just making crap up. So and one of the things that comes with that is a kind of reproducible builds version so if you've got everything you ever used and released you want to also be able to reproduce the build environment and be able to kind of get at the same thing again. So if you look at Red Heart reproducible builds you'll get something that's not quite as cool as the reproducible builds that's going on in Debian at the minute. Okay and so what that's all done with is the Koji build system. It's kind of a combination of tools from the Debian universe so it's like DAC in that it manages the archive that gets mirrored out or sent up to RHN and mirrored out whatever. It's also like S build in that it handles all the auto builders. It's kind of like P builder in that it's used for test builds from developers. It's like Alioth in that it kind of controls a distro wide get of the source code and it's kind of like snapshot.debian in that it's got a whole lot of history going back for years and years. So Koji was one of the things I worked on at Red Heart. Yeah, that's links to various information, whatever. So the sorts of things that you can do with Koji is commands like that that you'd run on your laptop to the first one is kind of like promoting a build into testing. The second one is doing an official build and uploading it and the third one is just doing a test build like you're doing P builder. So when I say it's like DAC, I mean it really is kind of the same. So take a DAC command line. You can generally have a pretty similar Koji command line. The example there is just listing the source packages that are currently in testing or the way yesterday and the source RPMs that are currently in Fedora 22. So where Koji gets interesting and does things that Debian can't do is in history. So with DACLS or R Madison you can see what's currently in the archive and you can do the same thing with Koji but Koji also lets you see what things were like what back in February the 8th. And you can do that just with a simple command line from your laptop at home, whatever. And you can also get an audit trail. So when did the packages get uploaded? When did they change? Who did them? And a simple command will just give you a history of that that you probably can't actually read. Is that any better? So an audit trail like that makes things easier to kind of understand when you're not just hacking on some code and you end up with a bug. So if you end up a few weeks later with something doesn't work on my system but it works fine on mine, is that because you're running a different version because I'm out of date because you're out of date. And it's basically similar to some of the sorts of things you get in Git. So you can do a kind of Git bisect except with repositories instead of just a single code base. You can obviously do a Git blame. So back here we had who tagged this broken version of Firefox into Fedora 22. And the other kind of useful thing that's, the other thing that's kind of useful for is that you can just look up the history of, if you were doing lots of build system stuff, you can look up the history of what you were doing three months ago. So I kind of think all that was pretty cool. And so the question is can I redo that somewhere else that's now relevant now that I'm not at Red Hat? And okay, it'd be cool to have DAC do it, but DAC's pretty complicated, so screw that. Be kind of cool to have devotee or DebBugs have kind of historical information. So DebBugs in its mail logs has a history of who changed the severity and whatever else, but it's not really queryable. That'd be cool, I think, but both of those are pretty complicated. So I figured what could be simpler than Brittany. Hey, I wrote it and it's still pretty similar to that. So that's kind of what I figured. So if you want to kind of try something new, keeping it simple is a good idea. You could go haywire with Brittany and try to record exactly when it tagged what into testing. That's kind of getting to the point of doing everything with DAC as well so that gets complicated. And I figured the hints file is kind of how the release team interacts with Brittany. Who here is familiar with the hints files? Who here is not familiar with the hints files? Great, next slide. So the hints files are how people control Brittany. The only other way is really changing the code, and that's not really very much fun when there's more than one person using it. So back in the day, that's how I did it. Okay, so not next slide. Sorry, we've got a break. As you might guess from the title of the talk, how you do this is with temporal tables. And temporal tables are relatively simple in that you're really just adding some time stamps to the database that you've already got. So there's two kind of different ways to do that. One is with a time stamp of when an entry is valid and when it's invalid, and the other is a history table as compared to the current table. So either adding columns to your existing table or adding a whole separate, this is how stuff used to be table. So the way I chose is basically the same way Koji does it. I couldn't find a better way. You store the times when what's in the database changed, you store when it's becoming valid, and I've also got an extra column that says that's just a boolean to say, okay, this is an active row because that makes indexing a little bit easier. I like that because there's only one table to query, and when you're doing updates, it can be fairly atomic. So that's what a table looks like. Is that readable? So a hints file might have a couple of packages frozen. So d-package apps in aptitude in this case. And the corresponding table to store that, it would have just originally, wow, that's exciting. It would have originally just had a single column saying the package to be blocked. To make it a temporal table, we add some timestamps, which are these two, and the active record. To make that a little bit more complicated, we've got the indexes and uniqueness constraints. So uniqueness constraint on active and package. If we set active to be true or null, then that means we've only got one active package at a time. And I've also got a primary key on create and package because I think that's useful, but I'm not sure. The other constraints are fairly straightforward, I think. So queries on that become a little bit more complicated, but not a whole heap. So if you want to insert something into a temporal table, you have to make sure that the old version is obsolete if there was one, and then you're just inserting one. So if there was an old active version, then where is it? If there was an old active version of GCC, then we set it no longer active and marked the time that we deleted it, and then we can just insert the new one. And that's a fairly simple formula that you can just basically pattern match to whatever your old insert clause was. Similarly for delete, we're not actually deleting anything from the table anymore, we're just setting the delete column, and updates get a bit more complicated because if you're doing more than one row, you need to get a copy of the data, possibly modified, and insert new rows, and that's not a trivial standard SQL command. And yeah, let's just skip that. Sorry for the video. Boop, boop. Okay, so Britain. I chose to do it in Python because everything's in Python, except if it's not in Haskell. SQL alchemy is awesome because it lets you do database agnostic, not obnoxious, database agnostic SQL. You kind of construct it as code, and it'll do the dialect stuff for you. I haven't been able to find anything other languages that really do that. And I'm assuming it's run locally, so I'm not doing client service stuff, and I'm doing it as subcommands because subcommands are cool. Okay, and the goals make something useful. Hopefully it'll be generalizable so I can reuse some of this code later for like devotee or debugs or whatever comes to mind, but make it useful first, and hopefully make it easy for the release team to try it out. Okay, so this is the next slide that I was referring to 100 slides ago. The hint types that the release team can stick in the hint files, and if you want to be a release assistant sometime and help them out, this is kind of useful knowledge. Basically, they're just a bunch of one-line commands that tell Britain that it should try and remove a package, it should block or freeze a package, that a particular version of a package is worth bypassing the freeze, that a particular version of a package should go in in spite of new bugs or broken compiles or whatever else, and it also has hints for the complicated transitions where one package breaks another, so they all have to go in at once, and the other one is just overriding the urgency, so instead of waiting five days or 10 days or two days, it can be forced to wait 20 days or zero days. Okay. So, because I want to make this kind of temporal now, so I want to have you be able to introduce a block on a package, remove a block on a package, but also have the block on the package be still there, but let this particular version of the package go through, I kind of decided that unblock was a little bit ambiguous, so I'm using the block or are you just going straight through it? So I've kind of renamed them, so the hints that I'm using are now remove, block, approve, exempt instead of force and transition instead of hint. And so the idea then is that for each of these hint types will be basically three subcommands to BritHint that let you list the hints that are current, and add a hint of that type or that you remove a hint of that type. Okay. So implementation, creating tables in SQL Alchemy, SQL Alchemy Core is what it is. My implementation of subcommands is pretty obscure and horrible, but I didn't get to repeat myself too much, so does that. If anyone wants to come and grab me later and tell me how to do subcommands nicely, that would be great because I really don't like any of the ways that actually work, let alone the way I've done it. Okay. So the main way that the code is structured is with a generic temporal table class. So instead of the ORM model of having an object per row, I've just got an object per table. So that object then implements query and insert and update and remove sort of methods. And my idea here is that once if I can make this generalizable to other applications, then this temporal table will just be able to pick up what's in the database and say, okay, sure, you've got a temporal table of bug severities, fine. Here's where you query functionality for that without having to write, without having to subclass or do a new code. And that's kind of working. So it offers a query method. So you can put in envis just to link to the database session. So the transaction lets you choose the time stamp to query on. And then the keyword arguments is all the columns for the database. So you can query on package or query on version or query on the transition name is pretty much all that these ones have. And then that just translates to a temporal table query, which is complicated enough, but not terribly straightforward unlike this sentence. Okay. So it also has an update method and also has a removal method that seemed to disappear somewhere. Oh well. Yep. So there's a Malt class because Malt is the name that I'm going to use if I can actually generalize this into other and to other uses that provides the access to the database. And obviously it's got a, I've obviously implemented a history option because I think history is cool. It's really complicated. The idea is that history calls are generalizable. So you can query the database and get a bunch of things back and then that needs to get translated into human sentences. So the question is, does it work? Is anyone from working on the Tanglou stuff here? Okay, cool. Tanglou has another Brittany instance and they're kind enough to have actually checked all their hints into Git over the past couple of years. So that's been a great test case. So I've gone through the Git and imported that into Britain and voila, that's probably completely illegible. Better? Is that readable at the back? That's probably the best I can do. So it'll basically let you query who did something and tell you from 2014 to 2015 Noscage made these changes to the hints files and likewise for a particular package it'll tell you who did what to it and that's all working fine just imported straight from hints files and likewise obviously it can tell you what the current state of the Tanglou hints are. So they've only got removals active and these are the packages they're trying to remove at the minute. So ideas for the future, I'd obviously like to actually publish the source code. I've started synchronizing with the Brittany hints but that's only been for a few days so it's only been a few auto removals that I've got there which isn't terribly interesting. Obviously this is a pitch to the release team to see if they want to try and use it. Currently missing is easy and forced transitions, UDEB block UDEB and unblock UDEB variants which I don't think actually do anything different to regular block and unblock and urgent and aged days. Ideally I'd love to generalize this more obviously and there we go. Yes. So I've actually got one question for you guys what license should I use? What's a good license these days? Hands up for GPL2? Okay, that's pretty convincing. GPL3 interesting and non-copy left really more non-copy left than GPLV2 in this crowd. I'm shocked. Okay, non-copy left disease. So if anyone wants to ask me questions while I see if I can publish this anyone? Bueller? Bueller? How do I yes that. That's now freely licensed and up on GitHub. Hopefully. I assume that worked. Yeah. Okay, thank you very much then. So you go to all this effort to make the database time aware I'm kind of surprised that there is no ready-made support from let's say PostgreSQL that you can sustain well keep all history of all tables in the past and here are special SQL commands that query the state from some earlier point. Did you look into existing databases that provide that out of the box? So there are add-ons to PostgreSQL that there is a temporal add-on to PostgreSQL that does things along these lines. I didn't want to use database specific stuff because I like developing on SQLite and deploying on PostgreSQL so hence the SQL alchemy stuff. I'm not sure the PostgreSQL stuff is great but there's definitely Oracle and Windows stuff for doing this because I mean they do tax stuff with temporal table sort of things, right? So you could have used wine and used the database? I'm sure someone else could have, yeah. Okay, that must make it about lunchtime.