 Well, what I am going to talk about is adventures, dependencies and deployment management because this is perl. Perl5 is just a virtual machine. Cbat is the language and Cbat is a wonderful thing. Right up until you realise that you're now trying to maintain a compatible set of versions across a couple of hundred dependencies. ac mae'n fydd meddwl y benderfyn, ac mae'n mynd i'w ddweud y ddeddiadol y sgiliau awrthyr mwyfaswyllt a chaelothau, mae'r ddigon a rheiddiadolio'r filanhau ac erbyn. Dydaw i fynd i ffyrdd i 저. Mae'r ddigon yw'r automatic sylfaen o'r dyfwyllwch o'u syniadau. Mae'n ffrifoedd oedd, a'n mynd i'w ffrifio. Dyma'r ffin yw meddwl mewn bosibl, business to business email where various other stuff attach. So the initial approach to that was, well okay well don't worry about it. When I first got there, the mail service worked by you generated the master.house.wd file on the primary mail server and then clicked to reload the .dv files because it was BSDi. mor hwn yn rôl i ddim yn oed o'r cyfnodd o'r SDI-draeth. ym mwy o gweithio'r HDTP, D.com, a wnaeth y ddweud. Felly mae'n gweithio'n gweithio, ond mae'n gweithio, felly mae'n gweithio'n gwahanol gyda'r rhan o'r Llyfrgell. Felly, mae gennym dda chi'n gwirio'n gwirio, mae'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio'n gwirio. ac, ydych chi'n ymwneud hynny, rhaid i'n gweld y bod yn cyfrofiad gweithio mae'n dim yn uchiddio gyda erbyn. Diynau, rydych chi'n uchddio gydaえる cynllun y fferm ofail o'r ysgol yn y dwfond. Llyfridd i bwysig y fferm o unrhyw yn y Ffnerd Gwlad Gwlad Gwlad Gwlad Gwlad Gwlad Gwlad Gwlad. Yn gyfrofiad gyda fferm o unrhyw ffwrn i'r ffferm. Y patchy 1.3 gynhyrchu'r definitio virtual host by copying the struct for the entire server definition and then pokes in the virtual host settings. So your memory usage over having no virtual house basically increases linearly. If your manual slides, which will not overcommit memory, this means that when you're a patchy boot, it tries to allocate 4 gigabytes of swap all at once and the machine wedges for 45 minutes. This was not amazingly useful, let us say. So the solution to this was, ooh, what about this tilde foothan? So break out more of a rewrite because what could possibly go wrong and what you do is when you get a request for domain.com that's going to need to hit the CGI then, you rewrite it internally to slash tilde domain.com and then aliasdomain.com that's why we're a T.P user in our lab. And then the CGI is the one that's the right user. This was in production for several years and I'm not sure whether I'm proud or scared of it, probably both. One of the things that I got into at the time was a model of automation which, rather than having an update and delete, always work on the principle of ensuring revoke. So you're either saying this should be there in exactly this state or this should not be there at all. And working on that basis means basically you're trying to converge towards a correct configuration and ensuring revoke for both the model to be eigenpotent, which gives you a lot of freedom than revoke. But it requires carefully designed mental structure. And no, I do not consider that a patchy conflict to count as careful or designed, but I did figure it out in the last 48 hours before going live because my relevant systems team hadn't actually tested the web servers. Anyway, it took me several years to recover my faith in the existence of systems administrators that I actually like, but that's a different story. So I've been wondering for years about how do you generalise this stuff? And the answer is, it's sort of possible. If you look at things like CF engine for all its faults, it tries to converge towards a desired state. But then I got distracted because we're a consultancy. We get brought on project when a code base in a deployment is two or three years old, and it's basically... We made a complete mess of running to the point where we made enough money to be able to afford to fix it, please understand. So what you get is what you get. So the hostile environment has become interesting. So in order to be able to do anything useful, you need zero install bootstrap. Because nothing else is going to be said. And at this point you're going, but I want to be able to use Cpan. So, okay. Step zero towards being able to always use Cpan is local load, which was actually me throwing a tantrum after about the 12th person that we come freeload hash thumbs saying, I can't use Cpan, I'm not rude. You can install it to your directory. Here's how to configure Cpan.pn. No, that's not what I said. Here's how to configure Cpan.pn. No, not... I've actually got tired trying to explain it to people. Spent two days reading full chain source code and discovered that actually if you set these four environment variables, it all works. And that's local load. To be entirely honest, at the moment, local load with... there's a way to make sure that you always have the same versions you expect in there is not a bad least-worth solution available. There were a bunch of things wrong with it, but it exists and it works and the caveats are obvious. So, consider this to be a... Cartman annoys me because I know all of the things that can go wrong, but everything else annoys me more, which, you know, from a systems point of view is the closest to a recommendation you're ever going to get from anything, right? But, hang on, my continuing question. So, step one, at Fat Packer. Fat Packer is fun. Basically, I know incredibly annoying trying to get single file versions of scripts. Par does not count. I can never talk anybody through configuring parser if it actually works. I had trouble configuring parser if it actually worked. So, alternative approach. Pull minus thing is correct. That gives you the same ink. So, OK, now you've got a list of modules loaded. That's not sufficient. Because distributions delay loan parts of themselves. I mean, even up until relatively recently, Carp did it. You had Carp.pn and Carp-slash-heavy.pn. So, if you had Carp.pn and Carp-slash-heavy, and if you only watched %ink, you ended up with one and not the other. And, lo and behold, nothing worked. So, well, distributions install were.catlist file. Yes, this does scream if you're using distro packages because distro packages go, we already have all our mechanisms for doing this. Let not shit the standard one that all the builders work with. Thank you very much, Darren String. Well, you can go from %ink, find the backlist for that file, generate a file list from that, and then you've got a complete list of files that you're going to need. A which point? Well, I mean, your first thought is, well, let's just bolt them on to the front of the script in the order that they're going to be required. But then, you know, again, the downloading problem, depending on what code parts you're going to find, you're going to need hot files in different orders. So, instead, what we do is you create a hash of here documents that embeds all of the code, and then you stick a code graph in a dink. And your code graph, there should be a backslash on the front of that because what it's doing is opening a scalar graph from the file handle to it, a which point, when you require a through-slash-bar.pn, it gets the content of the here document. A which point, all I have to do is bolt the code graph and the hash onto the front of the script, and you have a complete self-contained script. And, in fact, that fat packer is why C pan minus comes as a single file. Because me and Gaw had his own code for it, and then went, hey, if I use maths, I don't have to maintain mine anymore and switch. And then I gave him a commit bit and he cried, but, you know, these things happen. So that gets me as far as anything pure, fixed set of dependencies I can bootstrap. That's nice. Doesn't work for access, but generally, excess is either an optimisation rebinding to a C library. If you need a binding to a C library, you already have the problem of getting the C library back. So it turns out not to be that big, it is. Has resulted in re-besyncing myself and a few other people maintaining pure pool versions of disks that would otherwise be only access. And the pool is actually scarier than the C code, but it works. And it's still much less painful than part. So, coming back to it, you still need a dependency train to feed in. The packlist approach is fine, but it broke false, and it provides that you already installed the dependencies. And the thing that aggravates me is every single set of tooling on the planet seems to rely on the gold version. You either have, I mean, you either have getLatestFromSeaPan or get whatever the versions in the Debian repository are. Or, OK, can't really get whatever versions are in my carton.wok file, which at least gives you pure application but it's still not that interesting. And then you start exploring why it's like this. And I came across a fascinating thing, the Paul Graves tube algorithm which is in, I think, chapter 14 of our list. Even if you don't write common list one, we recommend to schedule the book because it's got some fascinating ideas in it. And you run into this problem. OK, X needs greater than 1.1 of that. So you go, what's the latest version? 1.12 of that. And then Y needs greater than 1.1. Oh, well, that's fine. And then you find out that there's a bug in 1.1.12 that makes Y not work. But you already allocated 1.1.12 for X so you need to get a backtrack. Right? And the choose algorithm does this by basically faking up continuations in common list. Which is not an unreasonable way of doing that. But I thought it would be more fun to have a list with continuations so I didn't have to do that. So, there is an experimental list called kernel. Yes, K, E, R, N, E, L. Yes, that was a stupid name for a programming language. Yes, I'm sorry. But kernel is what's known as an operative list. The fascinating thing about an operative list is that you don't get handy evaluated arguments. What you get handy instead is the expressions for your arguments and the lexical environment in which you can evaluate them if you want to. So you effectively get the full power of list macros at one time as well as compile time. This is very cool and very good fun. And very useful for prototype it. Hence why when I discovered somebody who equaled them to the fairly simple version of it to Perl. And then demonstrated that you can actually my job as scripting Perl if you so decide. What's going on there, by the way, is creates a derived environment for the evaluation and binds in the name and I think E, P is the original environment and then pushed to the evaluation section. Doesn't matter. What you can write with that, oh yes, because I'm strange I've used JSONi, which is basically JSON without the syntax that E and I came up with as the syntax for those because it saved me writing an asex for a person. So you can build something like this because it's an operative system those symbols for being the R, the E are actually names they're not evaluated through to values. So you can basically rewrite this into something that passes an environment then step by step. And the thing is if you're passing an environment then step by step you can capture a continuation. You can capture a continuation what you do is to E is short for execution environment when you're inside dependency solving you're always trying to produce a working execution environment and typing that out. No, if I wanted to be that for those I'd write Java. So the idea is you try to see if there's already a clash. If you have a clash then you merge it with the current you merge the requirements with the current choice which gets you to then resolve to the correct 1.11 instead of the bogey 12 and clash restart is invoking a delimited continuation that jumps you back up to the point in the algorithm where you made the incorrect choice. At which point the code continues on as if it made the right choice in the first place. If you're in a fresh if you're in a fresh choice situation with restart corrects the continuation and then with build choice is actually calling back through to an embedded pull object but updates the collection for the execution environment because I wasn't doing the entire data structure stuff in less boring. So that works. As it's traditional with operative lists however because of the fact that they let you do everything at runtime they have to do everything at runtime this does not perform particularly well. You can watch the CPU make for about a second every time in backtracks. So I consider this to be an interesting point of concept but not particularly useful. Of course there's bonus fun here as well because of course meta.json is not normative. The requirements in there are guaranteed to be probably sort of something kind of like the final requirement but if you rely on them for anything except generating an informational web page you are screwed because dependencies will vary depending on operating system, per version and so on so you have to have my matter which means your dependencies depend on the configuring part of it means the only way to do this is to actually have a build system baked into the dependency resolver. So I went through the cfam.presources they actually managed to patch it to the point more emotively than what I wanted but cfam.presources is very much oriented around having a queue of things to build and if you're in this situation I've got basically a tree right? Cfam plus nicer architecture in a lot of ways but it insists on recursing internally which works fine with a tree but then it's fundamentally not parallelizable and very hard to control so okay well there is a third option and then Mia Garry decided that supporting plugins for cfam miners was too much work deleted the entire plugin interface and declared it a black box so that stopped being an option there I had to say my current conclusion to that is nooks to out, I'll deal with that when everything else works so in the meantime this is over the course of like several years I ended up with this fascinating problem we had a customer whose infrastructure was a bunch of random amazon web services ECG machines deployed by a sole developer who could not see that means way out of a paper bag who was no longer with the company so they had no idea what their production platform was actually running a which point I'm in a situation where I want to be able to run lots of different brands having the fatpack every time it's just going to get too much like hard work right? fatpack is fine for one script you're going to run lots of times and play work beautifully for cfam miners but it's not so good for ad hoc systems too effectively what I want is automatic fatpack but I don't want to have to think about it how do I end up not thinking about it? I want to think called object remote object remote you can say new call on call on what that does is actually dispatches to the method on in the namespace new at which point object remote takes over opens up a connection to the remote host and you get a proxy back so what it actually does is that ssh is to the host runs pull dash at which point pull is waiting for the script on stdini so at that point you shut your fatpack version of yourself across the wire and then send it on the score on the score n because n sends to pull script is done you can stop arzing now at which point it gives me spin in back so I can reuse spin in and stdiniow as my communications channel so at that point we go okay we do that door is a proxy object it has an auto-load method it takes the method thing and it's got a reference to the connection internally collapses all of the arguments to JSON shifts them over the wire to the other end all based around Paul Evans' future.pm which is if you're trying to do aced event driven code look at IOAsync IOAsync with futures is so much less painful than any other approach um so it creates a future assigns an idea to it sends the method call and the future idea at which point the other end can turn for as long as it lights and then sends the line of JSON back with the results of the computation so you can run these multiply and power up but if you're only doing something simple then I'm not going to wait to also routine that basically goes block until this is done um that of course has to be backed by a spin loop the thing is because I'm trying to hide the async stuff under the hood when you're not caring about async it has to be a re-entrant spin loop that's when it gets fun because what you have to do basically is create a list that you're waiting for um and then only when the top future um on the stack is finished at that point you can stop loop um you can't just go over another one because the point is it has you know if you have loop waiting loop waiting inside that loop waiting inside that this one has to return first so you do so which is why you need the local and that makes it operate successfully re-entrant way um we we managed to break this quite hard um doing massively power up under there at one point um the current version actually works um and then you can just return the value from the future all fine um so the goal of all of this is now I'm putting an object in that because you know having an object and a code in that ink is just fun right doing that you can then have a hook object you have to fully qualify the method now because ink is a special symbol in Perl if it sees it on its own it's always forced into package mail even if it's sub-ink um so a which point you ask for some code and then do the open file handle to it and then sort it question now is where do we get the code from we get the code by making an object a remote call back to the host because we're re-entrant and the protocol's too bad a which one the host scans through its ink and if it finds the file ships the source code across which means that I can create any pure module that's installed on the master host that I'm SSHing from and if it's installed on all of the remote systems which means all I need is code 581 plus um and an SSH connection to any of these machines um and I can then whenever problems I want to and manage to gather data and pull it back to an operations box to figure out what's going on um so at that point you put things like using endless packages processes, net stat, information um um the first week live was a huge success um we found two missing services the production was dependent on that were not documented anywhere and one bomb at hahahaha hahahaha hmm hahahaha hmm The Bobnet is now gone happily. We still track things like process lists, package lists to make sure everything's kept in sync across all of the machines, so if any of the system's automation tuning that's updating stuff has a bad name it doesn't do the right thing. The Interestaction Code, which is completely independent, pick it up and tell us it went wrong. Anyway, so, go back to the choose algorithm. Choose was inspired by Prolog. Prolog is interesting. The thing about Prolog is it's all about predicates. It's a logic programming language which is close to that to a tip and not quite. Prolog is a beautiful design in that it's just procedural enough that you can work out what it's doing as opposed to most sort of sacks old that's in the similar where they're a black box, you feed it to the input and you either get an output or nothing. But the thing about Prolog is, apparently why that can be interpreted in multiple words, so if you run it with server then to a value, but package friendly, that will return one per package installed on the server, so package list for a server. You run the same thing with package bound, but not server. You get a list of all servers that package is installed, but with both nodes you get sure or false is this installed on this machine. So that model will look kind of interesting because of the idea of expressing truth so you can basically talk about is this my desired state and what is the current state. Prolog does have a kind of odd syntax in one respect. I'm saying odd syntax from the point of view of being a phone programmer, right? So obviously I want to be able to build something that's a reasonable, decorative interface that's happening to work with and 1970s French logic programming is probably not the optimal new eyes. So obviously the only thing to do is to call in even more strange and ugly languages. Fina, TCL is awesome because it basically makes and made out of strings. The point of which I realised I was going to like TCL was the point of which I got half an hour into using it. I got annoyed that it didn't have an unless keyword and managed to input it more in pure TCL in about 20 minutes because TCL basically braces in TCL are a single quote character. So what happens is the body just gets passed through as an argument to your subrative. So you mud the test and either read part of the body or don't. I mean under the hood once you've treated a particular string as code it caches the code representation so it's not the horribly swell it sounds like. But that basically allows you to met a program using strings. It's actually quite a fun language descriptive. I found myself more and more breaking out TCL for simple things simply because while I could write it in perl I spent so much time doing live scale object-oriented applications perl. It's quite hard to shift my brain to do good scripting perl whereas if I switch to a different language I'm not quite as incompetent. Anyway, so I had this idea of basically a problem used by a solar. So the idea of a fact is that you express things by saying okay an environment satisfies a requirement if there exists some build such that that build can satisfy the requirement and the build is available in the environment. So you basically think this allows you to do the two-step resolution because it's this road-packaging event where you work on a package name and you've got a C-pound that has a requirement independent of distribution so you need the difference. And then transitive dependencies become relatively easy to handle because all you have to do is say for each transitive requirement basically what the prologue system will do is it will recur the small possibilities of the requirement of and then recur store looking to satisfy them. At which point you get the world forwards down the dependency trade branch and because the prologue is logic programming the backtracking concept is already built in so the whole old action which was bad I need to go back is just a property of the language. Okay, so the end result from a prologue is you either get here is a set of values for which your original request was true or it says no. That's insufficient. The software I'm working on could also respond with maybe. This is systems, the answer's always made, right? More precisely it's not so much false as false but the idea being this is currently false but if you take these actions it will be true. So for example if the package installed on the server your choice of your responses are either going to be true or false but if I install it then it will be true. So you can actually produce possible solutions so what you effectively get out is not just a set of values but possibly a list of actions to be taken to make those values valid. Going from that you then end up with the values for the result but a projected result after the actions are taken. So if you say do I have a patchy on this machine it's going to say the answer will be yes after I get that install patchy. The problem you then get into is how to model that because you obviously don't want this thing going away and starting installing things when you're just checking. Well what's the obvious place to deal with state management? Let's steal some ideas from Haskell because the basic principle of a model is that you have a world state and so you say the new world is the result of applying some function to the old world and that allows you to interleave state through without compromising functional purity. In this case what we need to do is basically make sure the system always backs on to a view of the world we control. A wet point we can model actions as expected new world is the result of applying the function. So it says if I want that to install a patchy I expect the world to be the same except the patchy will be installed on this machine. And then after if you actually tell it to do it it can go back and reject. So the idea is you basically solve, run, re-solve each time you get a list of actions some of them are dependent on others. Like if you're creating a file in a directory and the directory isn't there yet. The file creation action is dependent on the directory so you have to run the directory creation and then you loop back through. This is meant to be a sys admin tool, I'm not having it trust itself, I know who wrote that code because me, that's not good. So you basically keep re-munning the solver until you end up with the last admin action. So in order to test this out I wrote myself an SSH key manager. So you basically describe rules. This is actually a copy and paste from running code. So .ssh on account it finds the home directory, it declares that a directory should exist in it called .ssh and declares them up. Right, um, whom they're on? Okay, whom they're on is that's a random remote again. Well, I already solved that one. All I have to do is write something that works locally and then ship it across the wire with object remote. The other key trick is the VM form of this is pure continuation housing style. So the state of the computation at any point is always encapsulated in a continuation. That then means that rather than prologs traditionally use a thing called the choice point which basically records how to go back a variable assignment. If you don't mind burning some RAM and in this sort of situation memory is not a great experience. It's a lot easier to just carry a version of the world with the modifications or where you applied. But again that fits in nicely with the Manado style. The other trick is that means that when you get halfway through and find you don't have a piece of information you can invoke a wonderful thing called return multi-level. Basically, if you've only used exceptions for flow control return multi-level is like that except not horrible. Return multi-level takes advantage of the fact that perl labels are dynamically sculpt. Gensins are the label name, stashes that away by the code graph and then basically to the point of the return multi-level. If your full function constate when you invoke that code graph it doesn't go to that label. Because the label is dynamically sculpt it always finds the right one. Fiddles with the return value returns from the original spot. So from four layers deep inside the solver I can basically do return to top. Here's the continuation. Here's the thing you need to run that affects the world before the continuation will be run. So that then gives you the ability to basically break out of the solver loop and be able to do multiple parallel actions and then go back into solver. So you never have your set of values and actions. So initially that's going to return a create direct reaction if the SSH isn't there. If it's there but the permission is wrong you'll get a turn off direct reaction. Because it always has to return the actions to the top label you can always see what it's going to do before it does it. So then authorised keys becomes a fairly similar thing and I can say is a key installed on this? Does the authorised keys file exist? Does it contain the right lock? But at the same time that code not only represents testing if the key is installed but because there are compensation actions attached to if any of those fail it can produce the actions for make, directory and byte file. So you effectively get SSH copy ID style of behaviour built in. The phone bar is that I can build the configuration out of the same thing. Just say find me the appropriate comfort file, what lines are in the comfort file. So you don't know not the system with three modes. Query which is pure true fault. Sol which is allowed to go fault but. And ensure which is going back to my original thing which is basically to figure out the actions to take to make this correct. Run the actions and then double check it all actually worked. So at that point you basically go to test all of the SSH keys installed on a remote machine. You just write in that and no key key meets for the local file. Built on top of this one set of rules I have status which just reports all keys installed. Sync which goes and installs whatever keys are missing from whatever machines they are missing from. But notably because of the action model I don't sync minus M. I'll make minus M. The system can always produce a driver which is something that really annoys me about a lot of tools. I really hate it when I tweak a comfort file and the only way to figure out if it's going to deploy the byte file is to run a test deployment on a staging machine. That's kind of slow. I want to figure out that I've screwed up much quicker than that, thank you. So sync is pretty much literally solved for all keys installed. Well minus M is and then you switch it over to ensure to actually do the work. Which brings us back to this. So at this point the idea of a build that can satisfy a requirement might be a pre-existing binary build or we might have to go away and build one. Transitive requirement problem at that tracking is already built in and the result is going to be to install an existing package or install the result of building a package and because things are just actions that each have the projected result of the package is available none of the rest of the logic needs to care about the fact that you're supporting source and binary building power. Which I'm hoping is going to allow me to have multiple additional sources for builds like saying there's already a package in a deviant repository and you've rather used that. Advocating to install that package is just a different action that makes the information available. So the current state of this is the DCL-BSL works. The CDS solar core is not showing me much of that because I really, really, really, really want to rewrite it once more before the rest of the world looks at the code. But they basically work. What remains comes into the simple matter of programming category. If you want to have a look, that's a timing I know. Dkit.get on git.shadowcap.co.uk has my current working progress including the key manager. I am actually using the key manager because I figured the sooner I started dogfooding the better things would be. Hopefully that contains some interesting ideas and next year I will have something actually deployed. Thank you very much. Given that I believe we are out of time as I propose to do questions in the form of I am going to go through into the canteen and sit down with a coffee, but if you want to ask me things, please feel free.