 Felly we're passing on to you lot. I'm going to go over some of the consequences, and some of the approaches that we've used to mitigate the difficulties, and I'm hoping also that some of you here might have some useful suggestions either for the Zen packaging in Debian or for us as upstream. So now, let me show you an overview of a typical Zen system with the key components broken out. Don't worry, there's not going to be 46 slides like this one. So firstly, I have to make a note about terminology. We use the term DOMnaught to refer to the first privileged guest operating system, which in most systems does all the back-end work of managing the system and its real hardware. Many of you who won't, sorry, right, yes. Many of you, even if you're Zen users, probably won't be familiar with several of the boxes on this diagram. Particularly there's some internal pieces which are part of our Zen upstream code. And you'll see I've drawn this blue box around the code for which we're upstream, and it's a rather unfortunate shape, and that's where many of our difficulties come from. So the code for which we're upstream comes in two big pieces. The first is the Zen hypervisor itself right here at the bottom, which you will have all heard of. And the second is this collection of libraries, scripts, demons, supporting tools, et cetera, which allow the host to do the work of managing and supporting the system and all the guests. Upstream, we call this part the Zen tools. Unfortunately, this is a bit confusing because there's also the Debian Zen Tools package. I'm going to keep using the upstream terminology here, so you'll just have to bear with me on that. The Debian Zen Tools package is a little thing up here, which is a sort of provisioning utility, which uses all the Zen tools stack code. So together with these higher layers like Zen Tools package and OpenStack and whatever, Zappi maybe, we call everything above the hypervisor and the domino kernel, the tool stack. So as you can see from this diagram, there are some components of the system for which we're not upstream, and there are three really important parts of that. The first of these is the guest operating system. So obviously this is the whole point of the exercise, and so in some ways it's the most important, but it's also the easiest from a compatibility point of view, at least for a downstream like Debian. So the guest might be another install of the same Debian version as running in Domnord, or it might be some other free operating system, which has been specially adapted, paravirtualised, as we call it, for running in Zen, so we call that a PV guest, or it might be an unmodified operating system for which we're providing a full PC emulation, so like an HVM guest, and that might even be a proprietary system such as Windows. Now the point of Zen is to provide a reliable and stable platform, and to decouple guest operating systems from hardware and sport software, so I'm going all this stuff here in blue in the lower layers. And often the guest will be run by a different personal organisation to the host, either in some kind of cloud facility, or because you're hosting internal sysadmin providers are different to the guest admins. So we've traditionally had a very strong compatibility guarantee. Guests which are written against old versions of Zen, and work correctly there, are supposed to work on future versions of Zen. Currently upstream supports guests written to run on previous Zen versions going back to Zen 3.0, which was released in 2005. So that means we retain the ability to service all the old guest-facing hypercalls, all the IO facilities at memory management interfaces, even if they've been replaced by something newer, better, faster, less pain to support. And this compatibility guarantee goes even further. For guests and Zen versions which support live migration, we ain't support live migration of your guest from an old version of Zen to a newer one. So if everything's working properly, you should be able to upgrade your Zen host without even needing to reboot your guests. And this migration facility means if you have multiple hosts and enough spare capacity, you can do what Zappi calls a rolling upgrade, where you can upgrade your entire hosting platform one host at a time by migrating the guests off each host as it's to be upgraded, and then migrating them back again afterwards. So actually meeting this compatibility promise from upstream point of view in every detail is quite tough. And the most difficult part of it, or one of the most difficult parts, is we don't always know exactly how the old guests behave, and sadly not all of the interface documentation was always perfect. But almost all these guest interfaces have pretty good extension mechanisms, so there are spare fields in instructs, there's key value namespaces where we can add new keys, there's a fair amount of capability negotiation mechanisms. There's not always sufficient because guests may be buggy, and even quite subtle changes can expose new bugs in guests, but even when that happens, we try to cope. So for example, a particular common kind of guest bug is that it gets partial or broken support for a shiny new Zen feature. So it then works fine on the old versions of Zen without the feature, but when you upgrade your platform, the guest tries to use the new feature and stabs itself in the eyes. So we have knobs that Zen administrator can twiddle for each guest to say, well, just don't advertise that feature, so to work around that lets you sidestep those kind of guest bugs. So what this promise means in practice is that when we're doing upstream work, we give forward compatibility for guests a lot of attention, both in design implementation, and of course it means we treat compatibility bugs as regressions which deserve a fix and back ports and all those kind of things. Now there's one other difficulty with providing this guarantee which I'll be getting on to, which is particular to HVM guests, which causes pain for downstreams such as Debian, and I'll get on to that later. So the other big part of the system for which we're not upstream is the dominoct kernel. Most in installations use Linux. There are other platforms that support dominoct kernels, but certainly in Debian the kernel available is Linux. So for many years we maintained a fork of Linux, this is the Xeno Linux that Ian was talking about, which had intrusive changes to the memory management system in truck handling, hardware access and so forth, and this became quite unsustainable, as you might have predicted. So nowadays, since around Linux 3.0, all or almost all of this necessary code is in the Linux upstream tree. So now the interface between the dominoct kernel and the hypervisor here is much broader than that provides to guest kernels, and also the interface between the tool stack and the dominoct kernel is relatively broad. This means that dominoct needs to be able to support all the facilities for direct access to guests needed by the management tools, and it also needs to cooperate with Xen to manage all the hardware, but because of the way Linux is developed and the fact that users would constantly need new versions of the dominoct kernel for all sorts of reasons not really related to Xen, like running on new hardware or fixing security bugs, trying to maintain our own stable branch of Linux is a non-starter. So instead, we have a kind of reasonably loose coupling here between the dominoct kernel and the other components of the Xen system. So we expect Xen and its tools to work with a range of dominoct kernels. On the other hand, this does mean that life is a bit... on the other hand, life is a little bit easier because we don't try to be bug compatible with bugs in old dominoct kernels, that is. If a user is running a Xen system, we expect them to be willing to run a kernel which has Xen-related bug fixes, even for bugs which were exposed by a new version of Xen well after the kernel was released upstream. So obviously we don't expect users to roll these kernels themselves. We aimed to push the necessary changes between mainstream Linux stable branches and out-to-distros that way, but that does mean that you can rely on users getting backported kernel bug fixes. So this tolerably loose coupling and all the upstreaming of the mainstream Linux, the code into mainstream Linux, means you can use the same kernel for your Xen dominoct as will boot on bare metal, and as Ian was saying, in Debian nowadays, there's no longer any special Xen kernels. The standard 686 Big Mem or AMD64 kernel is good to use as a Xen dominoct or as a guest as you like. So another important component of the traditional Xen system for which we're not upstream is QMU here. When we run unmodified guest operating systems, we need to provide them with something that looks like a PC. In particular, we need to provide something that looks like a screen, a keyboard, ID, a network controller, a simulation of the underlying bus hardware, like PCI and so forth. For a long time, the QMU project has been trying to provide this same thing in a different context, so it was an obvious choice for reuse. But when Xen first wanted to do something like this job, the QMU project was quite difficult to contribute to, and the code base itself made a lot of assumptions about the architecture of what the whole system was going to be and what the purpose of the QMU code was. So the Xen developers at the time took a copy of the QMU source code into the Xen tree and just hacked it up until it did what they wanted. So that, as you can imagine, has been a bit of a problem. So over the subsequent years, that version of QMU, which we now know upstream as QMU Xen traditional, got important performance improvements, important bug fixes, security fixes were made directly to the Xen version since upstream didn't have a security response process, and I think they still don't. We merged from upstream a couple of times, and we did try to upstream some or all of the Xen changes, but the number of factors made that meant that this wasn't entirely successful. Inertia and unwillingness to expand the scope of the QMU project on QMU's part were factors. Poor communications really didn't help, and we had a lack of effort and thought there were more important things to do, so ultimately that didn't really come to very much, at least not until a few years ago. But things have changed now. The QMU project has been utterly revolutionised over the past few years and is incredibly fast-moving and dynamic nowadays, and we've been aggressively upstreaming the changes necessary for Xen. So QMU Xen traditional is now in the deep freeze, but why do we need it at all? Well, one reason is the new QMU Xen based on upstream is not quite ready. The Xen 4.2 release will still use QMU Xen traditional by default, but we do expect the new upstream-based QMU Xen to be the default in Xen 4.3. But a big of a reason is that we promised old guests would continue to run. The emulated PC provided by modern QMU Xen is quite a different beast to that provided by QMU Xen traditional. It's got different device IDs on many of the devices, the emulated chipset has been upgraded, the new upstream QMU can't read the save files generated by QMU Xen traditional, so guests can't be saved, restored or migrated to the new one. Even worse, many guest operating systems react very poorly to a complete motherboard and chassis swap, which is what it would look like if you just upgraded to the new QMU and reboot it during a reboot. Even if they don't mind that in theory, in practice the new setup might expose rafts of guest bugs, which would be very difficult to work around, especially given that we're now much more closely coupled to the upstream QMU codebase. And while in principle it might be theoretically possible for upstream-based QMU to emulate the old one with sufficient if-defs and run-time flags, the resulting code would be very ugly, very intrusive, and not at all suitable for upstreaming. It would be totally unsuitable for long-term maintenance. So we as the upstream intend to main QMU Xen traditional indefinitely. That means that there are two versions of QMU in the Xen system, the old one and the new one. It's even worse, because of the speed of development to the QMU project and the lack of Xen-focused QMU committers, getting important features and changes into upstream can be rather slow. So if I just give an example, PCI passed through an important Xen feature for many years in QMU Xen traditional, took six to eight months to get into QMU upstream. So we as the Xen upstream provide a version of well upstream QMU, which is actually an upstream QMU with a lot of Xen-related changes. We have a policy of not committing anything into QMU Xen, which we think wouldn't be suitable for upstream. So in theory QMU Xen, as we call it, would be entirely suitable for use as a replacement for QMU. But in practice we don't expect people to want to use our slightly funny branch. For one reason, we don't do any non-Xen-related testing on it, and it might be based on a different version of QMU than the one you'd use for other purposes. So the net result is that to get full Xen functionality, a distro like Debian needs to include two additional copies of QMU, besides the normal QMU, and perhaps a different version for KVM as well. And one of the versions used by Xen is ancient. This is of course not the kind of thing that makes the Debian security team happy. Of course, it's by far from the only occurrence of this kind of thing in the Debian archive. And at least in this case, Debian can work with an upstream who still supports all the versions that Debian needs to ship. We as upstream do security and stability patches and compatibility fixes, even for the solidly frozen QMU Xen traditional. So given all that, it's good to see there are some efforts being made to help Debian manage code duplication and forking more effectively. For example, by searching the archive for similar code to fer it out copies that we didn't know about. But ultimately, in this case, the real work of doing support, particularly security support, for multiple versions is necessary. There's no other way to avoid breaking those old, perhaps hard to modify, guest operating systems. So here we can see one of the key ways that we do with the problem of interfacing compatibility. Just provide both versions of the software and use whichever one is appropriate somehow. And in principle, this is the same approach as done in a more formal systematic way with any seed library with a reasonably stable API. We're all familiar with Debian's well-established and mature approach to this, shipping perhaps two or three versions of any library in one release. I think Debian could really benefit from better tools, not just for spotting embedded copies and packages which are forks, but also for cross-porting changes, tracking which embedded copies came from which versions of upstream, which had which essential security fixes and all that kind of thing. So enough about all that. There's one other difficulty which shows up in this diagram. The Zen project upstream doesn't promise that you can run one version of the hypervisor with different versions of the tools. That is the user-spacesport management software will only work with a corresponding hypervisor, and that's this thick version line all the way down there. We don't break compatibility within a stable release, but in general there's no compatibility from one release to the next. The reason for this is that the management interface of the hypervisor is complicated and closely dependent on the features available in a particular Zen release. This means that both we want to be able to add new interfaces, such as new management hypercalls, and we want to be able to discard old ones. This whole situation leads to a similar problem to the one with Linux and Udev. If you're upgrading, you want to upgrade the tools on disk in the file system and the hypervisor in slash boot. If you do this in the naive way, there will be a time when the two versions on disk don't match and the system won't be able to work if it's rebooted. Luckily this doesn't make the system completely unbootable and impede you fixing it, but it might well be unable to run guests at all or the guests might break. It also makes it difficult just to try out the new version. If the new version doesn't work, you have to fish out some old packages from somewhere and install them. So the Debian Zen maintainers have arranged to allow co-installation of different versions of the Zen tools. Now our upstream build system doesn't support this particularly well and I applaud the efforts of the Zen maintainers in Debian to sort this out. So the Debian packaging for Zen edits all the paths to contain the Zen version number and there's wrappers for all of the Zen entry points which version of the hypervisor was booted and run the right set of binaries. So this compatibility rule seems quite restrictive and you might say, well, we should be more relaxed about it. But even if we upstream spent more effort on this area, it's not clear how much would help. The most obvious improved compatibility guarantee would be using the tools from Zen version N with the hypervisor from Zen version N plus one. So relaxing this line here on the diagram. But I wouldn't do very much good because firstly, distro like Debian which releases less often than Zen upstream will still have the problem because versions of Zen in two subsequent Debian releases would often differ by more than one Zen release. And even for a distro which releases less more often than Zen or if we offer the longer compatibility period, there'd still be a complicated upgrade path. You'd have to make sure to install and boot the new hypervisor first before upgrading the tools. So in practice, Zen will probably anyway have to use the capability to co-install different tools versions and use the appropriate one. There's one other big transition which doesn't show up so clearly in this diagram. So the old Zendy demon here which formed the core of many of the old Zen tool stacks is being phased out. Mostly this is because it's a terrible mess internally to the point where not only does nobody understand it, but it's probably not even possible to understand while remaining sane. Also it implements in a non-reusable way a good deal of the functionality wanted by other tool stacks. So one particular such tool stack is Zappi over there, you can see, which is a sophisticated API forming the core of what's known as XCP, the Zen Cloud platform, which is the free software release of Citrix Zensource's Zen server product. And now in Debian as Project Kronos. Zappi's used by orchestration layers such as OpenStack and CloudStack. And it's going to be shipped in Weezy. Another intended user of... This isn't the wrong place. So we're replacing Zendy with a C library known as LibZenLight or LibXL. The command line tool which is used by many Zen users to manage domains XM is being replaced by a mostly compatible rewrite called XL. And the intent here you see is that users such as Debian's ZenTools package will find that XL and XM are suitably compatible. So this is coming along nicely in the upstream tree. We've got a preview of it in Zen 4.1 which is going to be in Weezy. And in Zen 4.2 LibXL will have an API which we intend to support in the future in a backwards compatible way. Sadly, ABI compatibility is still beyond us. So users of the Debian's ZenTools package and its utilities like ZenCreativeImage should find that the generated guest config files are all compatible with the new XL utility. And you can use XL pretty much like XM. So your ad hoc high-level machinery shouldn't need too much adjustment if any at all. And we spent considerable effort trying to figure out what ZenD does in reproducing its behaviour so far as it's comprehensible and reasonable. The results aren't perfect from a compatibility point of view but we hope that in any particular system the changes need to use XL rather than XL will be minor or hopefully completely absent. One exception to this is the ZenD Managed Domain's facility which you invoke by saying XM new rather than XM create. We don't think many Debian users are using this and need many users at all. The functionality like this is better provided for in other ways. The most obvious is the init script-based automatic starting setup which is shipped in Debian for some time. And if that's not sufficiently sophisticated we suggest using a more sophisticated management layer such as Zappi or OpenStack. So if you're a ZenD and XM user we'd really like to encourage you to take a look at the new XL tool stack in Weezy. If you use ZenD directly rather than via XM or use XM in some way which you think might be unusual you should check right now that LibXL and XL meet your needs. ZenD is pretty much dead now as far as development goes and we're probably going to remove it in Zen 4.3 or maybe 4.4 probably therefore it won't be in Weezy plus 1. So if there's important features missing from LibXL and XL we need to know about it really soon. So one of the other users of LibXL intended is LibVert which is a bit of a more complicated problem. I know LibVert isn't very big in Debian more of a Red Hat thing but I will talk about it here briefly. I'm not much of an expert in it so if anyone is do speak up. Anyway our intent is that the LibVert driver the ZenLibVert driver will be linked against LibXL so the LibVert demon itself will be linked transitively against a particular Zen version and we may need to provide multiple versions of that too. So to try to make all of this somewhat less painful we've been working quite hard to ensure that at least in the 4.2's release we've got API compatibility in LibXL and indeed the command line compatibility with XM. So I hope I've given a flavour of some of the difficulties we face where the upstream for quite a mature and sophisticated system and so we've had to take some different approaches for some of the interfaces and some of the layers. So if anybody's got any suggestions for how either Debian or upstream can do things better and make our or other people's lives easier please do speak up. So any questions? Or you're all brains imploded by my diagram? Anyone on IRC? OK, well thank you very much.