 All right, I'm going to start. My name is Michael Johnson. I work at RPATH. I'm going to talk about some of the open source technology that we've been creating at RPATH specifically for packaging. And I've heard a lot of pronunciations. We call it canary. I want to talk about how their packaging process is simply powerful. We have a simple process with powerful features. The way we make that work is that whenever canary does the wrong thing by default, we'd make it easy to override and make it do the right thing. I'm going to talk about how we fight bit rot in packaging by enabling just enough packaging. I'm going to talk about what I like to call dependency heaven, just enough operating system. So I'm only going to talk about packaging here. I'm not going to talk about the process of using canary to manage your system, to install packages on your system. Only the building packages with canary. I'll only talk about system management as it relates to the process of building packages. I'm not going to talk about canary repository management. I'll just mention that canary stores everything in a web-accessible repository. And I'm not going to talk about building system images with our builder. Let me go back to my initial slide for a moment and mention that this URL on the bottom is currently live. And if you have your laptop, you can follow that. Many of the items on these slides are actually links, so you can follow them to more information. So the first question is, why canary? There are already several packaging systems in the world. And people were a little bit annoyed that those were not unified. So we'd have to have a pretty good reason for creating yet another packaging system. The answer is multi-fold. But one of our key ideas was that we would enable system management modes that are inspired by concepts in source code control. Particularly, the new lessons we've been learning as distributed source code control has become common. We also have extensive experience in what I like to call legacy packaging systems, like RPM and depackage and so forth. What I mean by that are package systems where the main definition of a package is an archive file as opposed to our concept of a package, which is something that lives in a repository and there's a lot more internal structure to it. So here are some of the things about legacy packaging that we found are good and bad. And this is not at all a complete list. One of the things that we really liked, particularly those of us who started this who had done a lot of RPM work, was the habit of using pristine upstream sources plus patches when you're building a package. So that when you go to a new upstream version, you don't have to recreate an archive. Another really important thing that happened relatively early on in today's modern legacy packaging was automatically discovering some dependencies. The typical example of this is shared library dependencies. But we also, based on these, this decade of experience or so had some very strong ideas about what was wrong in legacy packaging. In particular, we found that the control files, whether it's a spec file in RPM or anything else of that type, tended to be very verbose and have contents that were copied from spec file to spec file. And that introduced bit rot in a variety of ways that I'll talk about more. It's very much a two-manual process. We just had to do a lot of things manually that could be done automatically. And I would say a classic example of this would be percent files in the RPM spec file. It was also too complicated. There were just too many things you had to know how to do to get started. You had to read a book practically. There were two interestingly linked problems. The dependencies were insufficiently complete. Just because the dependencies were satisfied didn't mean that a package was really likely to work. And the other interesting thing was that dependencies, even though they weren't complete enough, were too broad. They were insufficiently specific. And so you end up with what gets called dependency help, where you drag in lots of packages that are unrelated to what you're trying to do in an attempt to get a working system. The last thing I'll mention is that scripts are often buggy. That's the post install, pre install, uninstall scripts that get run while you're installing or uninstalling packages. And we looked at reasons why they were buggy. And I found two real big reasons why they were buggy. One was that people would copy a piece of script from one place to another because they didn't need to write everything by hand to start with. And they would copy things they didn't understand. And so either a piece of script would work in its original context but not work correctly in the context into which it was copied. Or it might be a bug in the original that someone didn't understand when they copied it. Or when they're copying it, they might make a mistake in the copying. So there are really three kinds of errors that are introduced by having each of these scripts be specific to all the individual packages. So we went through and investigated a bunch of build systems when we were starting working at Canary. And we came up with some goals for the packaging process based on our investigation and our experience. One of the things that we noticed that was bad that I didn't even list was that information was commonly duplicated. Typical example of this is in an RPM spec file you have two mentions of every patch. You list it as a source and then you have where you apply it. And very often people who could apply in a patch but still reference it, so it's still part of the package but not applied. It made it unclear when patches were applied and not applied. So we want to provide only once and only necessary information. You shouldn't have to copy a lot of boilerplate to make things work. We wanted to do the best thing by default. One of the problems, RPM has something that they call policy, RPM policy. But it can't be very powerful because it's very hard to override what it does. So it can only do a little bit. We wanted to extend the concept of policy to be able to fix lots of things, find lots of errors. And if we make it very powerful, it might not always do quite what you want. It will do the right thing in the 80% case or the 90% case. But you need to be able to override it. So we needed to make it very, whatever decisions we were making at any level of packaging, we want to do a very powerful, sane, consistent thing by default so the package doesn't normally have to specify what to do, but make it very easy to override when something goes wrong. We wanted to avoid copying errors. And when I say copying errors, I've just mentioned three kinds of errors you can have in copying. We wanted to avoid all of them. We also had a very important goal that came from looking at distributed source code control. And that was to branch without forking. To do a branch that tracks what's being done upstream with some change applied. And we have this concept in multiple places in Canary. Finally, we wanted to enable reproducible builds. So our source packages are stored in the repository with everything you need to build them. Our control files specify clearly and completely what to do. And then, because I say clearly and completely, if there are no bugs, it's not software, right? So sometimes something wasn't reproducible that should have been, we record what was really done, and then you can go look back and figure out what changed. So we called our control files recipes. We wanted to have a different name, so it was clear what we were talking about. Eric Tran, one of the other co-inventors of Canary, is a gourmet cook. So we called our control files recipes and the process of building we call cooking. And again, we want to do the right thing 90% of the time and make it easy to override for the 10% case where we can't do quite the right thing by default. These recipes are implemented in Python, but that's not really very important because we have restricted what you do in a recipe to turn it into something of a domain-specific language. What you're really implementing is a subclass of a set of a class that brings in other classes that are pulling in thousands of lines of Python code, but what you're implementing is basically a class that sets up lists of things to do, and it's not actually doing the processing from code, so it's more declarative than you'd expect from saying they're written in Python. So we have infrastructure that we've created to intentionally inhibit complex code. When you're reading a recipe, you don't see a lot of conditionals. You can't say if file foo is in my build directory because at the time the recipe is being processed, nothing has been unpacked yet. And so instead, our conditionals are expressed in terms of configuration items, and those configuration items are stored as part of the packaging, and we call that a flavor. So if you build your package once for x86 and once for x8664, your conditional is based on am I building for x86 or x8664? You could also say I'm building for having SSL enabled in this package or not enabled in this package, and you could build both of those flavors and store them both in the repository. And you can do arbitrary combinations of these, but the fact that there's a conditional is recorded in the way it's stored, which makes it possible to reproduce what you built the last time and know how it was built and why. The stages of processing these recipes is a little bit different from other packaging systems. Yeah, this is a little bit familiar. You start with preparing sources. Pretty much everyone has a stage in which they prepare the sources. So if you've used RPM, RPM-BP, RPM-Build-BP, it's essentially equivalent. But then we do not at all separate the build and the install phase. So if you have a source file that doesn't need to be processed or modified in any way, say a config file, in the preparation stage, you can install it directly into the tester, the proxy for the system where it will be installed later, and you never have to touch it again. So even the preparing sources phase can already start to do the install phase. But other than that, the build side, the make and make install will live in the same part of the process. And then the last stage is checking the results with policy. And I'm making sure that, I say checking, it's checking and possibly fixing the results of the build. So here's a little bit more detail about how to prepare, how Kanari prepares the sources. You can provide a partial URL and it will guess the name of the archive. So all these examples here are directly pulled out of existing real recipes. In our SQLite recipe, it turns out that SQLite just lives at HTTP colon slash slash SQLite dot org slash SQLite dash version dot tar dot bz2 or tar dot gz. And all we have to do is give the base name of the URL, tell it that the name is SQLite and the version is whichever version we're building and it knows how to go fetch it. Similarly, we have a mirror system. If you've got something that's stored in the whole source forage mirror system, we will just go through and iterate over a whole bunch of source forage mirrors looking for all these different archives until we get the first good match. There's a patch, you can reference things by URL or you can check them directly in. So we have an example here of a patch that solves a security issue. You check that patch into the source component and reference it directly. So that ad archive can add all sorts of archives and automatically know what to do with it. Not just tar and CPIO, it can bring in RPMs and optionally if you're bringing in a binary RPM it can even honor user names and modes and stuff directly out of the RPM. We can even unpack ISO images directly. We can also add patches which are automatically applied. You don't have to say first add a patch and then apply it, it's all one thing. You can add individual source files and you can even pull directly from source control systems. So we have a place where we build snapshots of the latest canary out of our mercurial source code control system. We just say add mercurial snapshot from this URL. If we wanted to we could say tag is such and such a version and build from a tag instead of from the tip. But all of these things including when you're building from a source code control system, when you've prepared the sources and you're committing your source code to the repository, it looks all this up and stores it all in the repository. So even if SQLite someday deletes a version of an archive, you'll still be able to rebuild that specific version of SQLite because all the sources are stored in the repository. However, the things that are referenced by URLs when you check out a source package don't automatically get brought down to your machines. You only get the files that you've checked in directly like the CVE patch, the example here. So that means that a checkout of a recipe is very fast. Then it doesn't matter. You can mix and match the commands that add archives and the commands that do builds. The first it does all of the commands in order that work with source files, archives, source files, patches, so on. And then it does all the things that are build actions like configure and make and make install. And we provide sane automatic defaults that's the, as appropriate for the command, arguments and environment, so that builds are consistent between packages. All the packages will by default have the same C flags and CXX flags and so on, LD flags, so on and so forth. So configure that this is a method you can call that will invoke configure with a consistent set of default command line arguments. It'll set prefix, it will set datadir and libdir and all of the standard configure command line arguments so that all of your packages are configured the same way. And if you need to, you can override them. And then when you call make, it will set environment variables like C flags. And when you call make install, it will have all those same environment variables, but in addition it will set destor. And to the proxy root so that when you're doing your make install, all the files will be installed in the right place. One of the things you don't see here is anything like a file list. Whatever gets installed in that proxy root is part of the package. Anything that's not there is not part of the package. So we've encapsulated most of the common build and install commands. And if you were here for the last talk, one of those is CMake. We add them as we get reasons to add new commands. So it's not that hard for us to add a new command. It's a few lines of code. But then that makes sure that builds with that tool continue to be consistent. The question is why can't you just say r.build and it will do configure make and make install? We'll get there. I'm trying to give... So let me step back for just a moment and mention that I'm trying to give here the concepts that you need to understand Kanari. And if I talk fast, I'll actually be able to show you some slides from another presentation that has a lot more specific details. But I will get to that specific item. After all the build install phase is done, then we go through policy. And we have over 100 basically automatic operations that get run across every single package. And there's... It replaces a lot of manual work that's done in spec files. An example of this would be... There are many distributions which by default have a policy of removing the libtool.la files because they're not needed for most things. They're just extra files on the file system. And so they remove them. That can be done as a matter of policy. A lot of recipes go in. Make install has happened to have installed twiddle files, backup files from Emacs or something. And so they have to work around that by removed, you know, find Pypex, argsrm, or something like that. We just have a policy which knows a whole bunch of file types that should normally not be on the system. So examples of these policies, other examples, generally transparent multi-architecture support. We've got multiple policies which investigate your build and we'll notice things. If you're building an x8664 and you have some things installed in the lib and some into lib64, it will try to move everything from lib into lib64. And if that doesn't work because there's a file conflict, then it will create an error. Otherwise, it warns you. Another example of a very trivial kind of policy which has noticed several bugs, non-binaries and binders. We just have this list of common directories that should hold binaries. Bin, sbin, userbin, user sbin, userlibexec. And if you find a file in there without some execute bit set, that's probably a bug. And so we raise an error. Now, if you know that you have a reason that you need to have a file in there that is not executable, you just say, the error comes out saying non-binaries and binders found this non-executable file. You can say non-binaries and binders exception for this file or for these files. So, as I said, they're easily overridden with information that Canary provides at build time. So, not only are they easy to override, they're easy to enhance. So I'll get a little bit ahead of myself and say that Canary has a very interesting dependency handling. One of the cases that sometimes shows up when you have a launcher script for an application that sets a library path, well, you can't resolve the dependencies without knowing what that library path is. So you can tell Canary, build up the libraries based on this LD library path existing. That's an additional... Computing dependencies is a policy, and that's additional information that you provided the policy to help it do its job right. It's not usually needed, but for things like Firefox it is. It's also easily augmented. I say there's over 100 operations. That's what we distribute. But anyone else who wants to build an operating system on top of Canary can implement their own policies and make those part of the system on which you're building, and then those will also apply to your packages. So, here's an example of what our recipe looks like. And this is a made-up example I just typed out to show the structure, right? Class, my application. OK, you can see this is Python a little bit. It is an auto-package recipe. This means this is a package that uses the standard configure make-make-install process. The name is my application. Version is 1.0, and it requires a development library and a runtime from a few random packages in order to build. In order to unpack this, I need to go... I'm sorry, I'm missing a trailing slash on my example there, but you need to go to this URL, find my application-1.0.tar.dz2 and unpack it, and then because it's an auto-package recipe, the default is that it runs configure make-and-make-install. Now, if you need to pass extra configure arguments, you say that the configure stage is to call configure with these particular arguments. Most of the time you don't need that. So, this is kind of an idealized recipe, right? That I just typed up to show how things look in the general sense. You probably want to see a real recipe just cut and pasted from a real source component. That's incredibly more complicated. It has three build-requires instead of two, but that is actually the exact content of version 222 of the Xyle recipe that I pulled out of the repository, and many of our recipes are that simple. Some of them are even more simple. So, I see that this is an auto-package recipe. You're not left limited to the superclasses that we've defined for Canary. You can implement your own, and so an example of this is that there is a GNOME package recipe. Well, GNOME package recipe knows the GNOME standards for where archives are kept, and you say this is a GNOME package recipe, this is its name, this is its version, and the GNOME package recipe lists common build requirements for GNOME packages, and so many GNOME packages don't need anything beyond that common set. You may not even need to list build requirements. Just name and version, and that's the entire recipe. Three lines plus any copyright statement you want to put on a three-line file. Now I want to move on to some of the characteristics of the process, and I don't know that this gets into a little bit of how you install... how Canary installs packages on a system, but only in as much as it influences the process of packaging. So, it's transparent. It's easy to see what's happening. It's repeatable. You can go back and rebuild something you built before and be confident that you're going to get the same thing out of it. We automatically subdivide the packages into components. Now, if we go back to the recipe, you'll notice libtermcap develop and techinfo runtime. We'll see a little bit more about that. We have very rich file-level dependencies. Everything in Canary is done on the basis of individual files. That's the core object. And we have a robust system script design. And I say system script, it's the thing that would replace post-install, pre-install, uninstall scripts and things like that. Triggers. So, how do we achieve transparent and repeatable builds? First of all, as I mentioned, all the sources that are needed to build a package are stored in the Canary repository. If that doesn't happen, it's a bug in Canary. I can give you an example of that. We have the ability to authenticate via a GPG key. We don't currently store the actual key that we've downloaded from the key servers. That's a bug we're going to fix in the next few weeks. We just realized it when a key server went down and we had trouble building something. Build requirements are automatically discovered. So, you remember that list of build requirements? I built without that list on a system, and Canary told me that I needed to add those items to the list. Also, when I say packages that satisfy build requirements are recorded, what I mean is when you're building the package, it actually looks at the system database and sees exactly which versions of all the packages needed that satisfy the build requirements. And it's not just the build requirements that you have explicitly listed in the recipe. It's all the build requirements that's inherited. So, for example, the auto package recipe is going to have all the build requirements needed to run, you know, configure, make, make, install, you know, the C compiler and make and shell and awk and set and grep and tools like that. Then, the entire log of the process of doing the build, including adding the sources, adding the patches, doing all the build actions, exactly what shell command was called for each of the build actions, all the output from policy is stored in a build blog, and then that build blog at the very end of the process is put into the package. And so the build logs display all the actions that are taken during the build. We also store debugging information on the packages. So, if you're familiar with debug info packages, we have debug info components. So, speaking of components, files are automatically assigned to components. You don't have to decide whether to create food develop or not. You have a package which has one binary, one library, one header file, and you think, oh, it's too much to split it up, so I'm just going to create the food package. The problem with that is that if you're building against the library, you pull in the runtime, you pull in all the libraries that runtime depends on and so on and so forth, that's what creates the dependency-hell problem of cascading dependencies. So, the problem with having all of those lots of little components, if you had the concept of a package being one file on an FTP or HTTP server would be unmanageable. You'd have 10,000 files that you'd be downloading via FTP or HTTP or something. The fact that we have this repository model that's more like a source code control repository means that it's not a problem. So, the way we assign files to packages is regular expressions and file modes. So, you can say, this goes into this component only if it's executable or if it's not executable. And it's easy to override. You can say, no, I know that even though this file is not under user share, it still belongs in the data component as an example. So, I say that we have six or eight components per package and we have, so we have about 800 packages in a core distribution that turns into close to 8,000 components actually. That sounds like a nightmare for the user because they would see this huge amount of information about updates. In reality, it's not because we have something called a group that I'll talk about later. Normally, when a user is updating their system, they see one thing. I'm updating from state A to state B. And the groups that I'll talk about later turn that from what would be an update, a nightmare to understand your update to being very simple to understand what's happening. Canary's dependencies are semantically very rich. So, if you're used to a shared library dependency, you can use the library to implement the system 5.0.2. Canary has a very different view. We say it's l32 or l64, lib4.so.2 that implements the system 5.0. And the x86 or x8664 or ppc or what have you instruction set and on the provided side declare all the symbol versions that are included in that library. On the required side declare all the symbol versions that are included in that library. Used by the linkage to that library. And we can then match those up. The dependencies are per file. So, for every single file, we record that file's provides and requires all its dependency information. And then when we're creating packages of files, we roll that information up into the package. And the fact that we create these components avoids the dependency creep. The main handlers are our answer to system scripts. When you install a package, or you're making some packaging change, and you need to do something on the system. In order to avoid having to copy information from one script to another, we have just one script for each kind of thing that needs to happen, whether that's installing an info file or installing a kernel. Any of these kinds of things is one script. So, each file has zero or more tags. A file is an info file. A file is a kernel or kernel module. A file is a...you name it. And then you have one script on the system, which runs and knows how to handle installation and uninstallation. And this means you never copy yours. Now, could you have a bug in that script? Yes. In practice, there haven't been a whole lot. But because you're able to focus your attention on making...doing a good job of one script, the good news is if you fix a bug in the script, you fix it for all packages, even old versions of other packages. So, for example, let's say we founded a bug in the script that installs an info file. Maybe there's a missing dependency or something. When you install the new version of techinfo, which includes the script across those info files, it can go fix up all the mistakes it made before, even for packages that were created before that fix was created. You don't have to propagate the fix across 50 or 80 or 100 packages that contain info files. So this is robust. You don't just transcribe errors between control files because they never show up in the control files. In the first place, there's no transcription to happen. It only runs when there's a relevant change. So if you're updating from version one to version two of your package and the info file hasn't actually changed between those two versions, the fact that Canary does relative update, something I haven't mentioned yet, means that that file is just not included in the update at all. And because that file is not included in the update at all, that script never runs. So there's no chance that it's going to break anything by running when it doesn't need to run at all. It also, because we've made the concepts of installing and uninstalling a system-wide thing with a system script, it means that we have robust rollbacks. And an example of how robust our rollbacks are, Matt Wilson, another one of the inventors of Canary was on his system and typed in the wrong window a command that started erasing his entire system. And we noticed this when on IRC, he said, oh, no, and then we saw MSW has quit. And he caught it with control C halfway through erasing his entire system. Canary rolled back one, and about one minute later, his system was back and his back on IRC. And I can give more examples of that if you're interested, but not right now. Finally, I want to go through the process of building at a little bit higher level, the process you go through when you're building. First of all, we have a tool called Armake. It's part of our Canary suite and show you how package building works in Armake. I want to talk about shadows, which is our concept of branching without forking. I want to talk a little bit about groups, which is how we define consistent sets of packages. I want to talk about some of the capabilities we have for inspecting what's in packages that you've built and also the process of testing. First of all, the Armake tool. The Armake tool looks at the build requirements that you have listed in your recipe, plus all the build requirements that you've inherited from your superclass recipes, and the full dependency closure of all those explicitly listed dependencies and creates a clean change of environment. But it only has the things that are listed as required or needed to have a dependency closed set of packages installed. Then it builds it and commits it to a local Canary repository. It's actually running the Canary repository code on the system, and it commits it there. From there, you can install it on your system and test it. You can inspect it with all the tools that I'll mention for package inspection. And then when you're satisfied that the build is good, you can commit it to a permanent repository from the temporary working space on your system. You can also build from multiple architectures from one console. So if you have an x8664 system, you say Armake build foo for x86 and x8664. It'll start up both builds. It'll do them serially if you've told it to do one at once. It'll do them in parallel if you've told it to do more than one at once. And build them all. And then you can commit them together as one transaction. So the repository sees one transaction that is both x86 and x8664 builds. And you can do that for any kind of configuration changes you're trying to make. Building the SSL and not SSL versions for x86 and x8664, that would be cross product as four. You'd be committing four different configurations or flavors of that one version. Shadows are how we branch without forking. And they track changes relative to an upstream version. So a typical use is to add a patch or a configuration file or to build separate binaries against different base OS versions. Examples of this. We have customers who want to use a library that we provide as part of our operating system, but they need to have a configuration or maybe say PHP. They want to change PHP.ini to allow more memory for their particular systems that they're building. And they could create a shadow of PHP and modify that one file. We have a facility, a special kind of shadow that's called a derived package where you don't even have to rebuild it. You just say, install the upstream version, make changes to it, and repackage it. Well, it's useful for building separate binaries for different architectures or different base OS versions. The same recipe can build differently in different contexts. R-path Linux or foresight. Now that it was just announced someone ported Canary to open Solaris, maybe you'd take the same auto package recipe because you can look at Xile at recipe. You see, there's nothing in there specific to R-path Linux or anything. They could build that, make a shadow in their repository, cross repository shadow, and build for open Solaris on that shadow but track the changes we're making to the source code. Groups are collections of packages. And this is one of the key distinctions in Canary, is that our way of collecting packages is source and binary. Most other instances I'm aware of are, here's a list of names of packages. And maybe you'd even track changes to names of packages, but the client system is going and processing this list and deciding what to do for an individual system. What we do instead is we say, you are building a group. Here's a source. It's a recipe just like for a package. It says, I want to add this package and that package and another package. And it goes and looks in the repository and adds the packages and records the exact versions of those packages and components. And so Group My OS, version one, is going to say not just a few names of packages, but internally in the repository, that resolves down to the exact contents of every file installed on the system. So let's say that your operating system is derived from our PathLinux. And you've only added three packages and a plus our core basic operating system. Then we come up with a security update that affects you. You build that group again, you cook it again. It references new versions. Now you've got version one and version two in your repository. When someone wants to update, they say, I want to update to the latest version of Group My Appliance. And in our Group My Operating System, Canary then provides only the changes that are actually... It calculates the full changes between those two versions and applies those changes to your system. So it's a lot more like source code control. If you are tracking TIP and you do whatever your source code control system is, you merge up to TIP. It's the same kind of operation on your local file system. You're applying the changes, not just pulling down a whole load of new packages in your local file system, regardless of current system state. So a group defines a consistent set of packages, and this allows you to have a very nice QA and release process. You can build a package, the developer can look at it, and say it's good and commit it. Now you can have testing, or QA, and take a group and we have a way to move a group through the process of doing test and QA. And you can define whatever your process is, and you can move it from stage to stage. It's the exact same contents each stage along the way. So when QA is done and says, this is a good slice of the operating system, we run whatever tests we want on it, which are going to be different depending on what you need, then you make that available to the end user. Now, when I'm building an appliance that only I am using, it's something very simple that includes, say, a mercurial that I'm going to run as a DOMU on a Zen server. All my QA for this for myself is to compare the differences between them and just look and say, was this the change I wanted to make? But if I'm building a commercial product where you really need to run a whole test suite against it, that would fit into that place in the process. So it's a flexible construct. Package inspection, there are a few core things that I want to talk about. We have the ability to look at essentially all the elements of the package and look and see what's in it and what's different between different versions. One of the very useful things is file versions. When you build a binary package, you build the second version, first the first version, then the second version, the versions of the files that are stored, if the file hasn't changed, the file version in version two of the package is still version one of the package. So it's not just that the contents are the same, we actually say it is the same file. And so if you've rebuilt it and the binary has changed with a man page hasn't, the man page will actually point to the old version. And this keeps going. So you can look at a package that's part of a, you know, long-running maintenance stream and there's been some development behind it. You can see all the history of the development where things changed by looking at the exact version of each file in the package. File dependencies are another critical thing that we can display. When you build a package and it has a dependency you didn't expect, you want to know why this new version of this package dragged in a new dependency. You can easily find the file by asking, can I just show you the dependencies for each file? You know, pipe that to less and search for the exact dependency you want to know about and see what file it's in. And then you can determine, is this something I want or something I want to fix? We have a Canary RDIF command, which just displays what's different between this version and that version of the package. And the default output is fairly concise. You know, files changed, dependencies changed. Then you can ask it to give you details about exactly what changed, and it's a diff-like output. And the last thing I want to talk about for package inspection is the repository browser. Because the Canary repository is essentially a web service, it's almost free for us to provide a web front-end so you can click through with a web browser and inspect the package by browsing it. Click into the component, look at the file, download the file. That's a very convenient way to get things like our build logs. If you just want to see the build log for a package, you just go look in the repository, click on it. You can either install it on your system with the Canary command line, or you can look at it in the web browser. There are a couple things I want to say about testing, because I believe that testing is part of the process of building a package. The first is I've mentioned that Canary makes rollbacks reliable. And they also make testing easy, because it's not a problem to install the new version of the package out of your RMake repository or off of your development label on your computer. You try it out. If it doesn't work, it's very easy. It's one command to rollback your latest change and you're back at what worked before. The other thing is I've mentioned several times that Canary is when it's applying an update to your system, it's essentially a merge. It's not changing files that haven't changed. What you can't do is go into the file system and start just changing files, and then expect when you do an update to a new version that is just going to blow away your changes. So that's the one thing that might be different in Canary testing. You can't make random changes to files, and just assume that the next update operation is going to revert whatever changes you made. It's still possible to fix that if you do it. It's just not a good testing practice with Canary. So these are places for more information. I don't know how we're doing for time. I'm right on at 45 minutes. I have time for questions. One thing I want to mention is if you want that two-hour-long presentation thing is a link, I have that presentation up, and I want to show you more technical examples of what Canary recipes look like and what the process looks like. So I'll start with questions. I think I'm supposed to do this. Oh, well, I'll just repeat the question. Do we support soft dependencies or alternatives? No, I've tried to make... This is very much my influence, but I've tried to make dependencies be very specific and have a meaning that can't change. So we don't have virtual dependencies, like provides generic capability foo. We've had a few requests for things like that, like provides SMTP, but to my mind that's really provides something that listens on port 25. And so we haven't yet added something, a dependency class for port listener, but that is something that would make sense for me from my standpoint to add. We haven't... One of the reasons not to do dependencies that you would expect to be resolved by multiple different applications is that when you're building these groups, we often do automatic dependency resolution. And so if one time you cook the group, it was resolved by one package, and another time a different package, that would make for unstable groups. So we have... You could actually package multiple libraries that provide the same soname and the same symbol version, and then it would have to pick between them. We had a case of this before I put in all that handling for scripts that used LD library path, where OpenOffice was providing libstandard C++. And so when we were trying to build something that required libstandard C++, sometimes it was trying to install the version from OpenOffice, and that wasn't working very well. So we have chosen not to implement that in order to have a stable update string. There's two of you. I'm sorry, stay again? Sure. Okay, how do you... Is it really asking how you bootstrap working with this? We built an operating system called our PathLinux that is built with the Kanari technology. We don't try to support mix and match between different package managers because dependency resolution doesn't work in that environment. So you couldn't have Fedora Core for your operating system and Kanari for your applications. That said, I'm very interested in seeing other operating systems running on top of Kanari. And as I said, it's just imported to OpenSolaris. I believe there are people who are starting the work of actually building OpenSolaris with it. The other half of the bootstrapping problem, another thing I didn't get into here is the fact that Kanari has built-in support for cross-compiling. And in fact, when we build our operating system, we do a native cross followed by a bootstrap phase all driven by Armake. Armake is able to look at how the packages are designed. It automatically breaks, build loops, and does builds until it gets to the point where it can start solving dependencies cleanly and then builds the whole operating system. And that's a fully automated process. And that tool is OpenSource. It's available to anyone. People have used it to try out PowerPC builds and I think we're going to see more other architectures built with Kanari now that there's getting to be a little bit more interest. We tested the cross-compile by doing a fully working charot in PowerPC. It just hasn't been the focus of the company at this point to do anything other than X86 and X8664 because that's what our customers are using. So we focused our attention there. And you had a question, too. What kind of packages can it build and for which distributions? The packages it builds are Kanari packages that are in a Kanari repository. Right now, you're building basically for our path Linux or foresight. But there are... The other way of looking at this is every time... I said I wasn't going to talk about building system images with our builder. We have this web site that you can go to where you can create your own repository and build your application in there. We have something like 10,000 of these repositories that people set up in which they're building their own operating system. And then some of them do mash-ups that combine aspects of multiple repositories. Because Kanari is a distributed repository system, your operating system can contain components from multiple repositories. And there are... So from that standpoint, your packages can work in asterisk now, which is built on top of our path Linux. They can work in OpenFiler, also built on top of our path Linux, like I said, I'd love to see other base operating systems besides our path Linux as a core built with Kanari. So if folks are interested, please do talk to me afterwards. So as a proof of a concept, you mainly implemented that on this restricted set of applications so far, or do you have some other experiences with other platforms so far? How much time does it actually take to recall the new spec files or how do you call them? And another question, based on your experience, what share of typical projects can be actually covered by these means and methods you describe like automatically configuring, making store and stuff like that? So how much time does it take me as a maintainer to convert to Kanari? So the fundamental question is here is how much time does it take to package arbitrary stuff? The answer is not a whole lot. We're maintaining two streams, the R path Linux 1 and the R path Linux 2 beta. We have basically four and a half people doing that while implementing new features and supporting customers. When we were just starting, part of my time was developing the build side of Kanari. Part of the time was actually building the OS, and then we got two young folks' interns and they basically finished the OS in six weeks. Now Ken van Dijen is sitting up front here. You should talk to him afterwards because he has experience building whole OSes on an almost entirely automated basis. The GNOME developer kit is an excellent example of that. I say I'm out of time, so if you have more questions, please come talk to me. I'm here for the next talk. I'd like to plug it, package kit, but then after that I'd love to talk to folks more about Kanari. Thank you very much.