 Hello John. My name is Anil Madhavapethi from the University of Cambridge and John is from Citric Systems. So this is a slightly unusual presentation because this is a project that's been going on for about four years and we're just about to come to our first 1.0 release so I wanted to just give you an update. This entire slide deck is actually self-hosted on a little tiny microkernel that's entirely typesafe and written in language called a camel and it's also running on an open source version of ZenServer which is also something we want to update you about and everything is running online on the internet. So there really is a dramatic kind of self-hosted aspect to all this that might or might not have been working about an hour ago but it's definitely working now. So before I get started, because there's quite a lot of content and we only have half an hour, how many people here are familiar with Mirage and have heard of it before in a previous presentation? Great. How many have never heard of it and have no idea what's going on? Alright, a few. So what I'll do is I'll give you a very quick update on what Mirage is just to give you a feel for it and then we'll move on to how it all works. So the basic idea is that right now we have the cloud and we have lots and lots of bad traffic coming in from the internet, you know, from sands, internet, NASs and SDNs, wherever it is. And typically we're going through a lot of C codes, there's a lot of potential unsafety there and then it takes a lot of effort to try to secure those domains. So what we want to do is to try to figure out how do we make everything type safe so that we can get past this whole problem of buffer workflows and general memory on safety. The problem is we tend to, for example, write our applications in a type safe language but we're not really protecting the kernel, which is what sees the traffic first. We're not protecting our backend device drivers and it's a bit like protecting your generals in a battle but not really giving your frontline troops the defenses because once your kernel gets owned, user space is out of luck. So given that most of the attacks never come from within the VM, you tend to trust the code running within the VM, we're trying to figure out how can we rearrange and rethink the way that we construct these guest operating systems in such a way that everything is type safe. And the reason we can pull this off, the reason we can build a new operating system is because two things have changed in the last five years. The first thing is that the Hypervisor Zen gives us a very stable hardware interface which means that we don't have to keep working in device drivers. And the last few research operating systems built in Cambridge, there's one called Nemesis that Ian Pratt and co-worked on before Zen, all bit-rotted because graduate students could no longer write device drivers. And also now in the cloud, we tend to care about protocol compatibility, not so much ABIs anymore. I want to deal with HTTP interconnects and so on. So given this, what's our solution? The solution is very simple. It's something called Unicernals. I want to write high-level source code, and this high-level source code compiles directly into a virtual machine that crunches out all of the layers. And so you can really view virtual machines as kind of Unix processes done right on the cloud. Instead of having multiple layers of software, I will just have my compiler output a complete Zen kernel that's standalone that can be scheduled and run independently on the cloud. The problem, of course, the devil's in the detail. How do you go from this whole idea of building an embedded system to actually having a usable programming environment for actually building products? And that's what John will talk about because a Zen server, the Citrix cloud tool stack is adopting a lot of these principles in its use. So the contributions, the things we've been working really hard on is trying to figure out how do we go from high-level source code and a high-level programming interface to outputting these usable things. And this involves a technique called library operating systems, which is turn your entire operating system, all 15 million lines of code in Linux, into a set of reusable libraries that you can repurpose for outside of the kernel space. And also the final outputs are all in a little single address space VMs that are very efficient. They only do one thing. They run the OCaml runtime. How many people here have ever done any functional programming or statically type programming in general, playing with high-level languages? So about half the room. Anyone OCaml in particular? Haskell perhaps? Yeah, okay, yeah. So we've got about Haskell and OCaml. So the difference really between a lot of the techniques we espouse is a focus on static typing. And the reason for this is that we really want to be able to reject bad code at compile time. So unlike the use of Scheme or PHP or Ruby or even nice languages like Erlang, which is a very functional language, we really want to be able to type and make and have the entire set of source code in the application compile down and rejected if there's some inconsistency. So it's a combination of static typing and module systems that make this stuff possible. And what a module system is, is just a very, very scalable way of building very large systems where code can be reused. And I'll show you some examples of this. So currently what happens right now when you build an appliance is that you go through lots and lots of tools to build an appliance, right? So I go to some source code, some object files. This goes to a linker and the linker goes to a user on the library which is then executed and talks to the kernel. Merage is completely different. In Merage what happens is that everything is a library written in OCaml and we tend to go through one tool chain that links everything such as device drivers and network stacks and so on. And then it links a boot library which makes the device bootable and then you give it all the configuration files, also more OCaml code. So you're satisfying all of your dependencies and in the end the compiler just gets an enormous number of source files. For example, around 10 million lines of code for a typical kernel and then it outputs a standalone kernel. So this means that it has a complete view on what's going on. So how does this actually work in practice? The problem is, and this is why the project took four years, is that building a single quick hack of an embedded microcontroller is quite easy. The problem is when you're actually writing code, no one wants to debug micro kernels because you don't have any of the useful, big, large utility of large tool stack. So we started propounding this workflow of developing code under Unix, which is nice OCaml code in a nice Emax or Vi editor, and then you want the ability to progressively remove bits of Unix by replacing them with types of equivalents. And then the final step is just to recompile to kind of eject yourself from Unix and end up with a micro kernel. So the majority of your logic in your programming can happen in user space where you won't go insane trying to debug kernels and you have the benefits of Unix management stack, but in the end the production should all be detached and be the minimal possible thing that goes out there. So how does this work? Well, the reason this is hard is because it's all about build systems and package management, right? Because if we're pushing everything into our compiler, you need to be able to figure out how do I specify that I'm running under Unix, how do I specify I'm running under Zen, and just deal with this enormous dependency problem that results from turning your entire 15 million lines of kernel code into libraries. So this year specifically for those who have heard about Mirage before, there's been a huge amount of progress because we've been focusing on, we set ourselves a goal of releasing Mirage 1.0 this year. So of course that means December is when we release, just the very last minute. And what we've done this year is to take all of the source code we've written, which is entire operating systems from kernels, TCP IP stacks in ML, a full set of Zen device drivers from block front, from net front, from ring drivers, and all the protocols. And we've also glued it together by building a new package manager called OPAM. And we've also hooked it into the Travis continuous integration system. One of the interesting things is how popular this package manager became. So OPAM, the OCaml community is generally considered quite small. Since we released OPAM in March, over 1,800 packages have been, sorry, 1,800 packages of which 530 unique have been uploaded to OPAM due to our kind of GitHub network effect thing that's been going on. And we've also released an O'Reilly book called Real World OCaml. So a lot of the pieces are rapidly coming together to make all this stuff work. I wanted to show you quickly just a screencast to give you an idea of how this stuff works. So I'm starting on Mac OS X, which is what this is hosted on. And I can just use Homebrew to install OPAM. So this is a very normal development environment, nothing fancy required, certainly no use of Xen. And when you initialize OPAM, it connects to GitHub and it just retrieves a latest set of Mirage packages, more generally OCaml packages that are available. And this is quite nice because it gives you a nice workflow where you can decide which packages to pin and which packages to use. For example, this is saying OPAM and for Mirage and it's telling me the full set of packages that are available. Now at this point, I can OPAM install Mirage Unix and it will go and download a whole bunch of libraries. And so what this is doing is setting up a development environment for me, such that I can just say I want to build this presentation using Unix on my Mac. And the fact that it downloads everything means that I now have an environment where all of my libraries from my operating system are available in Unix mode and I can just clone my website again of GitHub and I can just type in make. So when this is built, this just gives you a Unix binary. This Unix binary isn't using sockets as normal. It's actually using Tuntap and Mac OS X to go off and link in the OCaml network stack and then redirect packets by Tuntap so everything is being handled as if it were an operating system except running within Unix. So when I build this web server, I can just connect to my local host. I've set up my networking so it's 10.002 and I'm now running one of these Unikernels on Mac OS X. And you can see here there's all of the debug code. Now the cool thing, I can then switch to Ubuntu and say, well, I've now debugged my website and I want to run this on Ubuntu so I can just add a PPA and the PPA sets up the OCaml environment except in Ubuntu and at this point, you know, it can do the install again to get the right version of OCaml but a new target becomes available and the new target here is the fact that it can discard the Linux dependency, set up a cross compilation environment and ensure that it works, it outputs a Zen kernel instead. So at this point, the only thing I have to do differently apart from the app it installs for OCaml, we've also packaged up everything on Ubuntu, is to, let me just speed this one up, this is just doing the same thing as before, we're just cloning the same website but when I do the installation, I simply tell my package manager, OCaml install Mirage Zen instead. So the package manager is getting a target. I'm telling it, I want you to target Zen or target Unix and it goes off and it figures out exactly what set of constraints and libraries to pick out and it just gives you the right set and then it makes sure all the cross compilation is set up. So you can imagine this is a source code version of Ubuntu's AppGet package manager. AppGet has a sophisticated constraint solver to deal with binary packages and OCaml actually reuses that constraint solver to deal with source code dependencies and so a typical kernel involves hundreds of libraries, therefore this thing just scales and it just works. So that was just a whirlwind tour, it's a little bit online to show you what's going on. So just to summarize that, all we have to do is we just write a normal bit of OCaml code and this is a slightly misformatted but this is a DNS server and then you write a configuration file that gives it the configuration and this is translated into OCaml as well so everything is homogenously put together and compiled. Then I simply type in OCaml install Mirage Unix and I can run it under Unix and then once I've debugged it, I can recompile the same source code to Zen by OCaml and install Mirage Zen and I can just use Excel to construct it as normal and then when all that's done, the magic happens by the OCaml module system and everything is type safe, any errors or inconsistencies will be rejected. So how does this work? The whole point of doing this in OCaml and not in a language with a less powerful module system is that we had to convert the entire operating system so imagine everything you have in a typical Unix environment into the OCaml module system. So I'm obviously not going to give a full tutorial on the OCaml module system but this should be something that a beginner can understand. So let's say I want to build a home page and when I build a home page, I typically need to have an HTTP service so I have a module dependency on that. Now HTTP requires some notion of TCP so therefore the most obvious way to do TCP is to map it to Unix sockets so we use a kernel and then that depends on a Unix module so this is a very normal programming environment. The thing we did next was to say well you actually have a choice here. If I want to compile into Unix, I have two ways of doing it. My TCP module can either be sockets or I can actually implement a complete networking stack and then satisfy that by a Tuntap which is an ethernet driver into Unix. I have a branching point where when I'm installing under Unix I can say well go the Tuntap root and eventually you'll find a module graph that will satisfy TCP or go down the Unix root and dependent sockets. But the Tuntap root has less dependency on Unix because it's not using the kernel networking stack and you can take this to an extreme. So then I can say well I don't want to depend on the Unix module at all can you give me a module graph that will just give me a Zen root instead and suddenly my network interface driver NetIF there instead of using Tuntap will go to a ring module which will go into an event channel module and a Zen store module and then voila you suddenly have a Zen micro kernel. So this is literally just taking the entirety of Unix converting into a structured module graph and breaking up your conceptual notion of Unix into a set of libraries. And these set of libraries can be repurposed and retargeted for many many uses. So this is a fragment of the graph. You can also compile many of the Zen modules under Unix to use the user space device drivers Grand Dev, Grand Dev and event channel so that you can often compile Zen apps either as micro kernels or in domain zero. So it gives us an immense amount of flexibility in how you build and retarget and reuse the source code and really this is the purpose is to build up a reusable library stack that we can depend on for a long time. So there's lots and lots of repositories online on GitHub Mirage and so in particular we have support for an increasingly exotic number of targets. So I've talked about Zen and Unix we also have a JavaScript backend we haven't quite figured out what that's for but you can use exactly the same principles of retaking a module and compiling it into JavaScript. This summer we had an intern who built a free BST kernel module backend as well. So instead of having a notion of an operating system of syscalls he expressed the intracernel APIs as OCaml modules and suddenly you can target the source code and just load a kernel module instead. So we've actually got a version of the Mirage website that has implementation being run directly from a kernel module without ever hitting user space and so many of the repositories for example ZenStore, the Mirage platform have implementations that map to each of these things but expose a consistent module interface. So for example the ZenStore one runs on Unix and Zen and very recently under free BST we have a block driver that compiles to block front or block back either in Unix or in Zen etc etc and Vchan and so on and so forth. So this really is why Mirage is called Mirage. It is not really an operating system it's just a collection of libraries that you can glue together. We're kind of arms dealers and you can just assemble them for whatever problem you have at any given time. To give you an idea of what it looks like it's actually a very natural encoding of the devices. This is block front. It's just got a record which is a set of features and then it's got a state descriptor T which is just an abstract SID descriptor and you can create and enumerate and read and write pages that you might do and see. The modules are relatively simple and should be recognisable to anyone who's done any kind of coding in this space. So what are the cool uses of this stuff? I said we could do anything with this and the students in Cambridge have been going crazy so they've been we're working on moving our home pages into little micro kernels so these micro kernels tend to be able to megabyte in size so it's very nice for just quickly launching devices on EC2 or Rackspace and because they're all written in a camel you can introspect the values we can actually self scale so if a queue gets too overloaded you just do a VM create and you spin up and fork another VM so these things can have a notion of scaling built into them and it all compiles to ARM as well so there's a native code version in ARM and we're working on the Zen ARM backend because we control the entire set of stack the TCP stack is just a library you don't have any artificial so we've been using this to build lots of middle boxes that need to do very low level protocol manipulation so specialized TCP stacks that are data centered TCP for example we can build custom block back ends that do encryption and storage so we have several demos of those coming out and one of the most interesting ones is because we know everything going into the virtual machine we can actually do this crazy thing so let me explain this diagram so you have a programmer who's green and you have a cloud which is the enemy and the cloud is basically where all the bad guys are so what we're doing is we're putting a compiler right in the middle of this so that everything going to the cloud is going through a statically type compiler so whenever I compile some OCaml code I put it through a SAP solver that is resolving my dependencies and then I have a precise notion of what Git trees are going into my code so then it goes into a compiler which links it and outputs a Zen kernel and then we push this to a binary repository on GitHub because it's just one megabyte so that's pretty well at this point whenever you push it to the cloud the cloud can watch GitHub because it knows the source code it depended on and anytime anyone pushes to that source code it just recompiles itself so in other words you've got a direct relationship between your GitHub repositories and the micro kernels that are actually being deployed so if you contrast this to Linux right now there's a very kind of loose dependency graph you don't really know what's gone into your image because you have a set of binaries a set of source code and you might be running ad hoc things like Puppet and Chef in this case we have a very, very precise dependency graph there's obviously an extreme position and often go into closing the loop so we really do plan this future where the compiler and a computer can react faster than humans and just know exactly what you've put in the cloud and just react and kind of get up there so I'm not going to talk about the microbench but yeah, good we have lots of time so the slides are online there's a lot more information I'm happy to talk about this later on but there's lots and lots of information and general up here but really the neat use of Zappi of Mirage is its application Zappi project so I'm going to have it with John and he's going to tell you exactly how the how the rest of the stack fits together so right, so Zappi project I'm going to start by introducing a bit because to everyone what this is so Zappi project is the core of a ZenTurbo it's not just the Zappi daemon that runs within your ZenTurbo it's all these surrounding bits and pieces that are required to make it into a functioning system it's a mix of lots and lots of the chemical code a fair bit of Python and a bit of scripting glue around the edges so what does that enable the ZenTurbo to do so we do a lot of stuff, we handle life cycle operations we've got a metadata database for all your VMs we expose this database and the operations over at XMLRPC API we've got the statistics we do networking setup we do manage bonds your VLANs, you're using opensvc which are all bridge mode we've got storage manipulation via plugins so all sorts of different plugins so NFS or LVM based things or various others which manage the life cycle of your disks memory ballooning we do disaster recovery, higher value piece live storage, there's a lot of stuff that it does so it's a bit of a beast that's grown over many many years so that's what a ZenTurbo does so this is the kind of a vague architecture of what's going on inside the ZenTurbo so there's lots of pieces Zen, the kernel, ZenStore there's other bits that I haven't written down, QMU there's PV drivers which I haven't written down now of this this is the ZAHU project it's your ZenOps D, the thing that drives Zen it's the squeeze D that does the ballooning the network daemon which is the statistics gathering storage API server some of the plugins, not all of them most of them the CLI, not the ZenCenter, not the kernel not Dom's area, not those the other bits and pieces so it's first appeared in ZenTurbo 4.0 in 2007 then we open sourced it in 2009 so it's been on Git, however since in 2012 we did this project Cronos which was trying to get Zappy and the other daemons into Debian and Ubuntu which I'll talk a little bit more in a second in 2013 then we transferred this to the Linux foundation and we also open sourced the rest of ZenServer okay, so hang on, there we go so Cronos project, this was the first effort really to get Zappy working in an environment that wasn't ZenServer or XCP as it was then and it was a long way from being distro-agnostic, it took a lot of work a lot of patches to get this actually working on a Debian system and in fact it was a success, we got it in there, it works but we learned a lot of lessons as a result of this so we did have an embedded mentality when we were developing Zappy and Co so if there's a problem in LVM we could have fixed it in the storage plugins or in LVM and we're in an embedded distro, let's go and fix it in the easiest possible place in LVM this is not really good when you're trying to actually then put this thing into a Ubuntu or Debian where there's a normal LVM, it doesn't work so that was bad secondly what got into Debian partly because of this was really a fork of the Zappy project we had to make a lot of changes lots of patches and it was really a dead end really because we didn't get those patches there were very specific patches to make this work and didn't get back into the mainline not really a sustainable model so since joining the Linux Foundation we've got this goal to become a lot more distro friendly and a lot of this is helped in fact by sharing with the Mirage project because the idea of turning a lot of things into libraries is a goal that benefits both projects so what we did was we originally had two monolithic repositories, we chopped them up into much smaller bits so our two became 50 odd or something like this a lot of libraries for example the Zen Store one which is now shared, shared memory rings and then other ones which aren't directly shared yet NBD and LVM bindings, things like this and we also split up the demons as well so the Zen API repository became ZenOpsD became XEP NetworkD XEP RDD various other repositories, so it's all been split up this has been really useful, it means that the job of getting these things into more upstreamable downstreamable distro friendly type repositories becomes a lot easier because you only have to make like SqueezeD look nice and then job done you would then go and make NetworkD look nice and it's a much easier thing than making the whole thing work properly so we've been doing it sort of bit by bit we've been putting as much as possible of this into OPAM so we've got an OPAM repository that's got all of these libraries in that you can get from GitHub which is useful what else has happened so Zen Server Core we took all these all these split apart things they're now much easier to get into packages for distributions so we did that in fact Ewan is going to be giving a talk next in this room I think talking about Zen Server Core but it's really partly the fruits of this effort so if we put things up we can now package them up much more easily so we are producing packages to install primarily on CentOS 64 but also Ubuntu and Debian are also important targets for this and importantly it's the same code base as Zen Server so we're not using a fork now all the patches are being upstreamed everything is now we're running from the same set of Git repositories so it's really good for that so the idea is we make the Zappi projects as a whole usable on any normal Linux so in order to make this happen we've had to put a few extra components in the ones that aren't specifically part of Zen Server for example the storage management that we had before was very based on taking over the storage completely and not being very friendly not coexisting with other things going on on your storage so we had to make FFS the flat file system SR which just puts VHDs down on your local drive makes it much easier to test this stuff on a Debian or CentOS box we've got the Zen Server install wizard because we don't have a host installer now so we can't set the thing up exactly as Zen Server or XCPUs to so this install wizard just puts some things in place makes Zen your default grub target sets up bridging or whatever those sorts of bits and pieces and message rich and libvert I'm not going to talk too much about that because I don't have much time so what's going on next so immediate next in Q4 2013 we're working really hard on the libzen light port so this is really important libzen light is what the Zen project are using to manage their VMs it's the library that's designed to be supported by them to do these sorts of things and so what we had been using was binding directly to Zen control and doing everything ourselves manually every time we upgraded Zen we had to fix all the things that broke the Zen light port is going to fix all of these things I think this is the first time that libzen light has been used outside of the context of Excel so we've had... there is a libvert one as well so but we found that we need to make a fair few patches so a lot of these patches have been sent to the ZenDevList and hopefully going to Zen44 so it's a little bit tough to compile this back into the moment so it's not built by default but we're working on that so what else we're going to add so we've got Zenostats this is a project to more efficiently gather statistics from your VMs so at the moment a lot of the statistics go via Zenstore and this is a terrible idea because your Zenstore is important for control operations and if you're trying to do write your memory utilization from all your hundreds of VMs into Zenstore at the same time it makes the whole thing grind to a halt so the idea is for Zenostats is to use some shared memory a basically lock free thing so that your guests will write their stats into there and something in Domzero will read it out and if it sees corrupt data it can throw that up data away because it's not important for the continued running of the system it's important for the just for the stats what else we've got ZenTestVM project so this is where again using Mirage this is really good so Mirage gives us a unicernal that can boot in microseconds so we can use this for really fast efficient testing of all sorts of bits of functionality one of the difficulties we have with the Zappi project as an open source project is that it's so broad and expansive that if you try and put a patch in there's a reasonably good chance you'll break something somewhere and we have ZenRT to do this sort of running nightly now this is coming out to be used as a service as Alex was talking about yesterday but it's still a very latent way of finding out what things you've broken so getting something that works really quickly and easily to give you feedback on a lot of the functionality of the Zappi project is going to be really useful so we're working on that we're also working on the message switch so this is again for partly motivated with Mirage in mind what Mirage allows us to do is that we've got all these libraries now aspects of of our control stack and what we can do is now package them up into smaller units and run them in VMs so not everything is running in DOM zero so for example ZenStore is an obvious target here we can make a ZenStore link it with the Mirage kernel and have that running as a separate domain so this is something that's going to give us a way of communicating with all sorts of different types of VMs that we'll be spitting off from the Zappi project for that next year we've got another Zen server release coming out at some point so there's going to be lots of bug fixes which is going to be good news we're going to be working on some improvements to the database so currently this is a bespoke kernel implementation in Zappi the process we're going to be needing to do some improvements there to improve a few problems that there are with that implementation again this is something that could be targeted as to something we can hoist into a VM if we want to we need to continue the cleansing that we have been doing of the Zen API repository so we took out the network demo and the already demo all some bits in there that require a favor of work to clean that up then we're going to be doing some work to support the Windsor architecture which is the architecture where we haven't drive domains and so forth that's been talked about several times before Zen Store Mirage VM as I mentioned so please come join in so both projects we have a Zen server core releases where you can test this bleeding edge stuff we've also got nightly releases of a Zen server all the development is open and on GitHub under the Zappi project or Mirage spaces pull requests and welcome we're about four minutes over but we could probably squeeze in a couple of quick questions so this is a randomly specific question about Mirage the test VM have you got SSH working at Mirage? yeah so I actually run a presentation for my PhD in 2006 actually so by the December release we're going to have an interactive console and SSH going into it so it's just a matter of adapting the old SSH code it's eight years old and it should be fine what we're currently using for the test VM is VCHEN so we're talking over VCHEN from Dell Zero so the reason I was thinking just for context is there's a whole load of bow arch open stack tests and we'd like to run Zen on Zen and it's good but sucks so if we can have a small VM that does SSH that just passes all the other tests this is a good thing this is exactly what we're doing as well we're running Zen server VMs so currently nested HVMs not terribly good but for running PV guests like these Mirage guests it's absolutely perfect so one of the challenges here is just the enormity of this project there's hundreds of repositories so we have a challenge getting to December so what I'd really like to do is to find a few focus use cases and just satisfy those so this sounds like a great one so the test VM is probably the first the goal here is getting Zen server and stacking projects as a tier A hypervisor and open stack there's more way of doing that it has to be tier one so this is kind of a build verification test where you just have to pass it quickly too exactly, it actually starts a whole cloud and starts connecting volumes and SSH again but still you can't you have to pass those tests to get into control got it one last quick question for someone alright let's give them a hand thanks for your time