 coming. My name is Avi Kiviti, some of you may be familiar with my previous project KVM which I was the maintainer for the last, for seven years, but now I'm doing a new project OSV which Gabor and I will present. And for those of you who doesn't know me or for those that do, I'm Glauber working together with Avi on OSV and previously working on the past few years with containers. So I'll let Avi start. Hey, thanks. So this is the team that is doing OSV at Claudius. I have a full head in real life. That can change. And it's completely frozen. There it is. Okay, that's not expected. The first one was fine. Yeah. Okay, so this is going to be a little awkward, but probably resolution difference. I don't know. Yeah, we'll manage. Okay, so let's look at the slide to come up. Is there any other PDF here that I might have in Fedora? I don't know. Okay. So let's look at the story so far in operating system evolution. First people coded directly to the hardware. There was no operating systems back in the good old days. But then they found out there was a lot of common code to application and common tasks they needed to perform. So people wrote operating systems for those things. And then with bigger machines, they wanted to run multiple systems, multiple instances on a single piece of hardware, especially with the mainframe. So they invented hypervisors way back in the 70s. And again, applications still had a lot of common code, and so they abstracted that into runtimes. A good example of a run-manage runtime is the Java virtual machine, the JVM, and also things like application servers. So there is a pattern here where people keep adding more and more layers in order to abstract common functionality. And this is how it looks like. The hardware on top of that almost always to date these days is very prevalent. The hypervisor in clouds. Then you have the operating system presenting an API. And more often than not, there is some runtime, the JVM running on top of that, and then the application server and the app. And then we have to switch to a slide and it takes five minutes. This one is fast. No, it doesn't move. It's also cutting here. I'm sorry about that. Maybe if you take it out of presentation mode, it's going to get better. Maybe. Okay, so let's focus on three of the layers in the stack, the hypervisors, the OS and the JVM. And what we see is that they all have some duplicate functionality. So each of them is trying to protect the layer below it from the layer above it. For example, the hypervisor protects the hardware from an abstract. They have the hardware from the operating system and it presents a unified set of hardware in KVM, that's Virio Block and Virionette. The operating system does the same thing. It abstracts the hardware via device drivers and presents it as files or processes. And the JVM, again, does the same thing. It takes the various operating system APIs and abstracts them again into the JVM APIs, which are then used by the application. And the same thing happens in reverse, where you have protection. Each layer does the protection. And this is all duplicated. And we all know what duplication means. It means enlarged footprint and inefficiency and more things to manage. So that would all be okay, as Avi said, if it came for free. And what we can realize, when we look at the job that the operating system is doing, so here you can meet more or less what kind of tasks each of those layers are doing, turns out that the, historically speaking, the operating system is not really good at isolation. It can isolate processes well enough in terms of security, in terms of accesses to resources, but when it boils down to actual multiplexing the resources in terms of resource usage, it doesn't go that well. So there has been attempts in the Linux kernel to change that through C groups and the whole container thing, which was what I was working on. But the historical fact that virtualization, that the operating system is traditionally not very good with isolating things, is that what led to the wide adoption of hypervisors. So that's one of the reasons. If you look at how things are, it makes sense for like back then, in the day when it had like VMware start getting virtualization to get serious, that is in the x86 market. I mean, virtualization itself was invented by remsas II, already working for IBM to build the pyramids. But if you look at the x86 world, it comes with VMware pretty much. And what you get is that you get a virtual server. So if you're going to isolate a workload, then each virtual machine isolates whatever is inside the virtual machine pretty well. When we have the cloud things change even further, because now there is another pressure which is pretty much the flexibility that you want from cloud appointments. So we get Amazon, for example, which is an extremely large cloud provider. And they're not even talking about machines anymore. They're talking about the compute nodes. So you just have something that compute, and it's getting more and more commoditized. You just get on some of those compute nodes, and when you need them, you bring them, when you don't need them, you send them back. And things have been truly, have been turning truly, truly massive. Your workload is usually not entirely tight together. So for maximum flexibility, so this is the kind of things you have in cloud. I mean, a bunch of buzzwords, but it's basically many kinds of way, many types of way in which you can deploy things in the cloud. But traditionally speaking, there is an extra pressure to have things completely isolated. So you don't really want many applications running on the same virtual machine. Because aside from the fact that the operating system inside won't isolate them that well, you have, you would like your workload to scale independently. So if one part of that scales, the other one does not necessarily so. So you just bring up a machine. And really, if you look at how those things are being deployed, you really have commodity tiles being deployed. So you don't really think anymore. People don't really update their softwares in their virtual machines. So the OS is basically immutable. If you want to run a new version of the operating system, you basically spend out a new VM. And it's very important from my point of view to point out that this is not really a proposal that we're making. We're not coming here to say this is how people should run cloud and VMs. This is pretty much, we look into the cloud and we look at the Amazon workloads, for example, how they're doing their deployments. And that is how it is working. So not only there is duplication and functionality, but that duplication is pretty much useless. You're not using all that isolation. When you go look at how does the typical virtual server looks like in the cloud. So you have the traditional cloud infrastructure and each of those virtual machines, they run a virtual server, Linux most of the time. Linux supports, it's a multi user operating system, but most of the time you're not using it because of things that are isolated among VMs. You can run many applications, but turns out that in practice, you're usually running just one of them. And that's pretty much it. I mean, so you get a lot of complexity for no reason. So the whole idea that we get for OSV is that we should really try to diminish this complexity by making things behave in a way there is more natural giving boundaries that we have today. So we truly believe that less is more. And that's why we're not giving shirts this year, but we're having boxer shorts for you. Right. So because less is more. So during the talk or even after the talk, come talk to us and collect your boxer shorts. And don't forget less is more. Okay, so so what are we trying to do? We're trying to build the very best operating system for running cloud workloads. And that means the easiest to manage providing the best performance being the best overall. And I like to just say that we are often asked, is it a new distribution of Linux? Is it a modification? No, this is something that is written from scratch, something that we've written that cloud use and are now open sourcing. And it's completely new. So let's look at how OSV is built. So this is basically a core sort of kernel that runs in the same address space as the application. It runs just one application. So there is just one process instead of a wide range of process which have to be managed and switched around. There is no concept of user space and kernel space. There is just one space. So there are no privilege level transitions which are which cost time and also require validation of parameters. So that's on what we change. What we don't change are the APIs that that's application sees. So there is no need to port your application to OSV, it will just work, but it will be faster and it will be easier to manage. So these are some of the changes that we are making. In terms of memory, we are using huge pages to reduce memory access latency. We have sharing between the JVM heap and the system memory so that you don't need to configure each of them separately, but it dynamically changes according to requirements. We have improvements to IO and since we're in a single address space, that's very easy to do. We are making scheduling improvements. The operating system knows that it is running on virtual hardware and it is optimized to run on top of a hypervisor. So we use this knowledge to improve the scheduling algorithms. Since it's smaller and simpler, then it's already tuned for the cloud workloads. There are no thousands of configuration files and knobs that you have to twiddle to get the exact performance. It comes pre-tuned. And we're also doing improved communication between threads. So if you look at that, we're very confident that by writing it from scratch, we can get things done better because we have designed decisions that a traditional operating system that runs many kinds of things cannot take and we do because we're basically writing everything over. The very fact that we have pretty much no drivers help us out with this task because there is no 10,000 network cards to support. There is only Zen, KVM, VM, or pretty much. And there is very unlikely that we're going to have 10 hypervisors in the very future. I mean, we can have an extra one who knows but not too many. So right now, up to now, I've told you things that we've been doing that everybody does and we believe we do better due to the decisions we made. But there are also nice opportunities to be done once you get rid of the user space, kernel space separation. I'm going to detail some of them now. So when you get a network package, for example, what you get, as you can see here, is packet processing is quite serial in the way it behaves. So you get an interrupt coming and it tells you that a package is available. So while an interrupt handler, you allocate SKBuff, for example, in the Linux world and then you keep processing that until it finally gets in the application. And one of the reasons it is like this is that somewhere there, like you have a very clear kernel user space boundary that we do not have. If you look at the, and this is the traditional socket model, right? If you look at Van Jacobson's proposal, you cannot see all of it but most of it, it's well enough to understand. Van Jacobson has been speaking about that since 2006 and that's not something that it's impossible to do with Linux. It's something that is more natural to do with our model, which he calls network channels. So instead of sockets, we're moving towards network channels. So what network channels does is that in interrupt handling, you actually don't do anything. You don't even allocate memory. You just get the package and relate to the correct location. So whatever the network car put the package is fine. And then you process that very, very close to the application. So you're already processing this. So first of all, outside interrupt context. And also in the same CPU, in the same cache line, very, very close to where the data is going to be consumed. The results are actually quite good. Those are not our results. Those are experimental results from Van Jacobson on Linux. So it's actually very easy to do that in Linux experimentally. But in practice, it is a little bit harder. You need somehow to get your network stock running in the same context as your application to get as close as possible. So one way of doing this is to moving more of the stack into the space. But the other way of doing this is what we're doing, move more, I mean, move the application into kernel space. But what matters is, once you're out of the boundary, we can try, we can do those kind of things. Also as Avi said, but I'm going to detail Java machine virtual, the Java virtual machine memory ballooning. So if you're familiar with the ballooning drivers in KVM, for example, that's pretty much the same thing. People who run the JVM usually have to statically set the size of the heap and say, yeah, you need to know in advance how much memory you believe you're going to need. And what we do is that we just allocate all the memory to the Java virtual machine, and we have a balloon driver inside. This is still actually work in progress, but it's currently being done. So it's not for the future or mid future. When you need memory, that balloon drivers get memory back to you. And when the Java virtual machines need that memory back, instead of garbage collection, for example, we can try to ask the operating system first, can you give me back some of that memory? So it's the traditional ballooning driver, same thing between guest and host. And this is also made not only possible, but a lot easier by the fact that you have no trust issues here. So the operating system and the Java virtual machine are actually all operating tandem. There is nothing to protect against. Okay, let's look at the timeline of what we've accomplished since we started. We started around December and quite quickly we have a Java Hello Award running. We have SMP support quite soon after that. And then we enable the various features, support for networking and storage. We're using a ZFS, our file system, and we gradually optimize the networking stack to get really good performance. And we're not running real-world applications like Cassandra and MemcashD and we're getting very good results out of both of them. And we're very happy with where we are and we're not continuing this, optimizing more, enabling more functionality, and testing more applications. Part of the reason we've been able to proceed so quickly, first of all it's a smaller application domain. We're not trying to do every possible workload, like Linux does. We're focusing on a smaller set of workloads. And the other reason is we're using modern techniques. For example, we're using the C++ languages, our implementation language, and not C, which reduces the amount of coding we have to do. We're able to reuse more libraries, more data structures, and this way it reduces our development and debugging overhead. So the status right now, we're running Java, but not only Java code, pretty much any JVM-based language like Ruby, which runs on the JRuby project, and the newer cooler languages like Scala, Ruby, Clojure, they all run on OSV. In addition, we also run selected C applications. We're not supporting the entire POSIX API, so we can't just run any C application. But we do support a large enough subset that important applications can be ported with relative ease to OSV, and we run Memcache D, and others are also possible. In certain benchmarks, we already get very good results. So right now it's not tremendously better than Linux, but they're sufficiently better to show the promise, and we plan to improve it even further. In micro benchmarks, it's very easy to show the potential because you have things like a context switch, which are relatively expensive on Linux. They run very, very fast on OSV. Boot time, which is particularly important in the cloud because you scale by spinning up newer instances on demand. So boot time is very short, which allows it to respond very quickly to loads that become greater and lower in time. It runs the same thing. Of course, you can just pull the plug. It's a lot easier. We're using the ZFS file system, which is a great choice for the cloud because it's self-healing, so it reduces the administration overhead. And like we mentioned before, we're using huge pages. So why is OSV open source? So first of all, we all come from an open source background and we love open source, but it's more than that today. To be credible in the infrastructure space, you have to be open source. Otherwise, no one is looking at you. And we're looking for a cooperation. We're looking for developers who are interested in the kernel space and doing an operating system from scratch. We're looking for people who are interested in a management stack versus the operating system and people who know the DevOps workflow, how you can deploy your application dynamically, push a new version every day, or sometimes even multiple times per day. And we would love to cooperate with the open source community on that. We already have multiple contributors to OSV, including from two companies and also individuals. We have a very friendly BSD style license and no contributor license agreement, so just sign off and that's it. And we really would love to cooperate with these people. In terms of architecture ports, so we started with x86, of course. We're only 64 bits and no 32-bit port. Out of familiarity, we started with KVM and it's running very fast on that. There is also a Zen port that runs on Amazon and you can use our image on Amazon and it just works. It's still not as fast as it could be. We're still optimizing it, but it does run. And we plan to do a VMware port eventually for the enterprise customers who are mostly on VMware. We also want to do an ARM port since ARM is such a hot topic in virtualization and in data centers. And if you have another architecture that you want to see supported, then patches are very welcome. All right, so this is what we do. Not everything that we told you is completely ready, but it's everything at least working progress. I would like to tell you a little bit more about things that we plan to do that shows the potential that we have. So those are all still ideas, but those are ideas that make sense and they're actually implemented in other scenarios in other circumstances that we just believe we're actually capable of implementing them in a nice and supportable fashion. Most of them revolve around changing the JVM somehow. So this is one of the main differences that I see, particularly between what we're doing and the containers approach, is that we now have your application having access to all the facilities of the hardware. Because, I mean, KVM or Zen or whatever the hypervisor is protecting you, your application, the JVM in that case, and it is the JVM only to provide a single entry point. So you don't have to change your application and every change is behind it. But the JVM now can see the hardware. You can see everything and there are many interesting things that can spend out of it. The one I'm going to detail is this. I mean, Garbage Collection is by far one of the most annoying things of the Java world. I mean, it's nice when you code, but it's not really nice when it just stops your machine to Garbage Collect and you can't really do anything. One of the techniques that the Java Virtual Machine already employs is to keep a dirty bitmap or any kind of dirty tracking, really, to know if a particular region of memory has changed since the last Garbage Collection run. And if it hasn't changed, there is no reason to go through it again. You can just reuse information from the previous run. Now, the JVM does that, but it spends a lot of time by tracking those memory because it needs to be tracked in the just-in-time compiler level. I mean, every memory access needs to be audited for that. But it doesn't have to be this way if we give JVM access to the page tables. So, if you can use the hardware dirty bits, for example, whenever you have a change, you're just going to see that represented in the dirty bits by the hardware. So, you don't really need to do anything in software. We can proceed a lot, a lot faster in that. So, in this case, it's not really only the Garbage Collection that goes faster, but the whole execution can actually proceed faster. Okay. Let's do a little technical deep dive into some of the features of OSV. So, the first feature is not really a feature, but it's an attribute of how we develop it. And it's using C++ for a kernel, which is pretty rare. But let's look at two pieces of code. On the left side, we have a typical kernel code that locks a particular data structure with two locks that are nested and does some operation and has a few checks. And as most kernel developers will be familiar with, there is a lot of codes that checks for errors and does the appropriate rollbacks. And it becomes very complicated and very long. So, this is a very typical code. And on the right side, you can see how it transforms when you use scoped locking. You no longer need to do all those rollbacks manually, all the error handling. It just works. You want to return something, then you return the value. And the language already takes care of unlocking everything in the correct order in its correct place. Something else we do is we have a strong focus on performance and tracing. So, the code here shows how you can declare with just one line a trace point and then use it in just one other line. And on the screenshot below, we have an OSV shell command that allows you to enable trace points dynamically. So, usually a trace point is compiled to a single knob instruction. But when you enable it, it compiles the code that dumps all the parameters to a log file or is able to a statistics counter. And we can count statistics or generate call stacks. And then we can use that to analyze application performance and understand it and all of that without changing the code since those trace points are placed statically into the system. Another feature, this is a more advanced feature which will be a little in the future, are data plan applications. So, a certain class of application has performance requirements that are very high and they don't want to go through the entire stack. So, for this kind of application, we enable direct access to the virtual ring buffer. And there's an application that can sit on top of the ring and just consume packets as they come in, do the transformations you need to do and send output data very, very quickly. And this way you can obtain very high performance and very low latency. Of course, these are applications that need to be written specifically for that API. But there are some classes of applications that will benefit greatly from it. Now, OSV can save a lot of resources as we've told you. We believe we're faster, we're more performant in many ways, and we're reducing the application. But CPU and memory are not the only expensive resources you have in your organization. I mean, another very expensive resource are people. It's not that we want to fire people, we want to give people vacations. So, what you typically have is it's not that we're proposing to get rid of people. I mean, it's in the same thing. It's all about the application. So, when you look at the whole stack, you have someone managing the hypervisor to offer you some cloud offering of any kind. But once you run your application, which is your domain, the thing you understand about, you also need someone to manage all those operating system instances. And that's a sysadmin. And it's not that sysadmin is in general should not exist, is that sysadmin is misplaced. I mean, you shouldn't be there. It's just cost. If you look at what people are doing in the cloud today, you always have an operating system as your guest. So, you need to configure your operating system. You need to set up the users. You need to set up the configuration file. You need to tune it to the particular use case that you have. All of that needs to be done. It's either done manually by a human being or it's automated away. But even this automated away process is error prone and it's particularly tricky because if you make an error in the automation, of course, you're just going to replicate that. So, automating away the problem, it's a way of looking into this. So, the OSV way is slightly different. Instead of automating away, you just get rid of the problem completely. So, there is no tuning to be done. There is no tuning to be done. Like as Avi said, we're only doing one thing. So, you run virtual machines and only one application. There is no choose between workloads. That's your workload and end of story. There is no state to be saved. I mean, there is no configuration files. There is no way to do that. Basically, you need a new VM. You come up with a new VM and you're good to go. There is no patching. I mean, there is especially patching. I mean, of course, we can release new versions, but we're not doing something which you're going to try to update your operating system. In particular, if you're using any kind of Linux distribution for that, you have unrelated packages that are just there because the system needs it to run. And you need to update them eventually. You need to follow the security advisors and all that. We're all doing a single binary thing. And again, we just expect that to be a lot easier from the management point of view. The actual management of the runtime of the virtual machine during the lifetime. We do have management software. So, we have a REST API and you can connect either Shell or SSH or web browser is that the example we're showing, which is basically aimed at deploying things at OSV. So, it's in our interest to make this as easy as possible. You basically, if you have a Java application, for example, you drop the jar in there and you're good to go. It's basically a way to install things. You get things installed in OSV and once you prepare the image, there is no maintenance to be done. That's the goal. So, and again, I mean, we're aiming this to be completely stateless. And this is something that fits the cloud model very, very well. You don't ever have to keep state of things. If you need a new VM, you just come up with a new VM and that's already in the state that you need. We're planning on integrating all of that with Chef, Puppet and all the tooling, as Avi said, of the DevOps workflow. So, that's not anymore in the kernel, but it's a very important thing, a very important aspect of what we're doing. And it stuck again. So, while the next slide loads, you have an application. So, what do you need to run on OSV? If you're running Java application, you need to breathe and you're done. There's nothing else. So, that's one of the reasons we're being so focused on the JVM. So, it's not that you're, we're not JVM exclusive, but focusing on the JVM allows us to allow you to run your application without any single change. You can just drop your jar and you're good to go. Because any eventual modifications that we may have to make to the JVM are made to the JVM itself. Your application is just fine. Even if you use JNI, and that's why. Because we also try to support the Linux POSIX API. We cannot support all of it. So, some of it is obvious. For example, if you run only a single address space, it's impossible to fork. I mean, threading is fine. We do have pthread create and pthread destroy and all kinds of pthread callings. But fork, xx, something that you cannot do. And we need position independent code. So, at the very least, you need to recompile your application. And to instead of generate an elf binary, you generate a shared library. So, there is some effort. That's not for free. That's why we focus on Java. Because if you have a runtime to isolate it from that, then it's a lot easier. But we don't expect that to be a lot harder either. So, please. So, the question is, I need to repeat that, sorry. So, the question is that we need to make a position independent. Sorry. Why? Yeah. That is because when you run, when you traditionally run an elf binary, it expects the entry point to be at a definite location. And it's very, it's fairly easy to guarantee because they've just created the virtual mapping anyway, right? So, I can just, that's free. I mean, you can just go and load your application in there. But since we only have one virtual mapping, again, we don't ever switch virtual mappings. We don't ever flush the TLB explicitly. That's why our context switches are so much faster, by the way. So, we cannot guarantee that the address in which your program needs to be loaded is available. So, and that's pretty much what position independent code does. Position independent code is code that you can load anywhere. So, chair libraries because, and that's pretty much the reason. So, some applications may not go well with OSV. We're not too much concerned about that because, again, we are very focused. So, if you're, if OSV cannot run your application, we're fine. So, the question is if either the first restriction that it must be a single process application is temporary restriction. And the answer is no. It's part of our architecture. But again, single process does not mean single processing. I mean, we support threads just fine. We just cannot have different processes. And what does that mean is that we cannot isolate things using memory mappings. So, when you fork, for example, you generate a completely different memory mapping. And that we cannot do because we only have one address space. Now, that address space can be running as many threads as you want because they share the address space. The process in which you do isolation through memory mappings that we cannot do. That's part of the OSV architecture. That's why you cannot fork, for example. And that's why it doesn't make any sense to talk about exec. Exec basically just replaces the virtual mapping, which we don't have. There are other small cavits as well. But they're mostly, I mean, it shouldn't be too hard, but we're not going to run anything. That's not our goal. So the question is whether we have any performance number for the Java workloads. I mean, we have some. We're not necessarily disclosing all of them. You want to comment on that? So we saw some performance gains with Cassandra, which is a particularly important workload because it's also a no SQL database. I don't remember that the numbers are not very high. They're not huge improvements right now. But we've only just started with optimizing the JVM. But they are better than Linux guests already. So we're ready better. But I'm not going to tell you how much better because it's a little bit unscientific so far. Hyper V. Yeah, so you're from Microsoft, right? So the question is whether you have any plans for other hypervisors and we would like to run everywhere. So as far as hypervisors are concerned, we would like to run everywhere. There are not that many hypervisors out there. What is in our plans is we have KVM running very well. We have Zen that the Zen support is still getting better. Yeah, the drivers. It's not just the drivers. It's also the timekeeping. But it's really not very difficult to port OSV to a hypervisor. It's just a few thousand lines for the drivers and just some glue code for detection. It's pretty simple. So if you happen to have a different hypervisor than send patches and we'll be happy to merge them. All right. So that's basically what we have. If you have any more questions now that we're done, I can go around with the microphone. So you don't have to repeat your questions. We'd like to thank you before we make a question. Don't forget to collect your boxer shorts and he has the first question. Hold on. And by the way, we are hiring. So if you like what you see, feel free to drop us a line and we'll be happy. So I know C++ but don't know Java. Am I still able to contribute something valuable to you? So yes, of course, we have places in all positions in the stack. If you know C++, you can work on the kernel code. If you know Java, you can work on the management stack. And if you know all of them, then you can contribute on the changes to the JVM that we're making that will make Java run Java and JVM based languages run faster on OSV. So the application would be a different part. So we have APIs that allow you to upload a JAR file or a war file or an ear file, whatever they're called. And they just get deployed on the instance and then you can start and stop them. It's pretty simple. I think there was a slide with, well, it would be pretty difficult to get to the slide with, we need an operating system just for showing slides. What happens if you have a bug in the JVM or in the application? So the same thing that happens if you have a bug in the application or JVM on a normal operating system, it crashes. Since it's just one application, all of the value is there. You don't lose anything. So with OSV, everything crashes. It just stops. It aborts and asks for help. So if you have a typical, if you have a multi-user system and the entire system crashes, it's horrible because all of your workloads crashed. But the way we see applications deployed today is just one application per server. So if the application crashes, the system that remains has no value. All of the value is in the application. So you basically restart the VM? Yeah. What about long-running processes? What do you do? Do you do live patching and stuff like that? Do you want to do that? No. The way we see it, that's not what people do on Cloud Workload. They run their instance and if they have a new version, then they just throw it on a new VM, points a load balancer to the new instance. If it's good, they kill the old one. If not, they kill the new one and try again. It's a lot more robust than the old methods. Is OSV capable of taking advantage of kernel same-page merging in Linux? So if you have multiple of these applications running on the same box, they're able to share memory resources? So OSV is completely agnostic to that. If you're running KSM on the host and your applications are similar and it will find similar pages, it's not really an OSV question. It's a KSM question and, yes, KSM will happily merge similar pages from different VMs. Okay. And a follow-up kind of unrelated. How do you see OSV kind of fitting into like a DevOps workflow like you had on your slides? Yeah. So I think that OSV is really a great fit for that because it's so simple. It allows the developer to concentrate on the application instead of the packaging. You have just the operating system image and you can push your application on that and then push the whole thing into the cloud. It's really very simple. There are no configuration files that you need to tune on your image. I think it's a great fit for the DevOps. In fact, it was designed. It was inspired by the DevOps workflow. Questions? I'm curious. Why did you use CFS? It appears to me that all this rate code and set pools and all this stuff is also not really necessary in the cloud environment where you usually have enterprise storage backends that can do all the stuff and can do checks something in this. So why the overhead there? Yes. So good question. CFS is indeed a fairly heavyweight file system. And I think it contributes most of the code to our system. But it does have features that are good for the cloud. You don't want to spend too much time running FSCK or things like that. So a system like CFS which has self-healing properties is good for the cloud. You may not always trust the infrastructure that's provided by the cloud. They tend to be commodity machines very cheap and you might not trust it. It's not all of them run enterprise class hardware with extreme error correction and error checking. Some of them are just disk, local disk attached to the server and you just hope it's correct. So we want to minimize the chances of an error creeping in. And also we wanted something that is compatible with our license. So we didn't want to take any GPL file system. And then CFS became the best content. And there is also something to add to that because you said yourself that in the cloud, most of the time you're not getting to the actual local file system. You have some storage in the cloud. So the file system that you have underneath is not terribly important performance wise because most of the critical workloads are not going to reach the file system anyway. So there were no reasons for us to go looking for extremely high performance file system. All the manageability thing is a lot more important than that. So the fact that, as I mentioned, all the checksumming and all the easiness and so on. More questions? So I have two questions. One is, do you support ballooning? So right now we don't have a balloon driver. It should be pretty easy to add since we already support the concept of my moving memory from the application to the system. So we could also give it to the host. It should be very easy to add. I don't think that balloons are heavily used in the cloud, but it should be an easy fit. Okay. And CPUs, you can't do hot plug right now, I assume yet. But would that be something that you'd be adding? So I could add another CPU to my already running application. Yes, so you're correct that we don't support hot plug. The code is written in a way that we can support it in the future. If there is a need, if we see that clouds start to support a hot plug, right now they don't. I don't think even normal enterprise virtualization do hot plug on a common basis. But if you feel there is a need, then a patch would be welcome. It shouldn't be very hard to add. Again, the system is fairly simple. So it's not completely trivial, but it's a lot simpler than the large multi-user systems. And it's fairly easy to add such a feature. All right. I'm actually not sure how much time do you have? How much time do you have? I guess five minutes anymore. See if you have more questions anymore. One over there. What is your concept regarding supporting different hypervisors? Is there a version for this hypervisor, a version for this hypervisor, a version for this hypervisor? Because you just talked about code recognizing. Is it one thing which recognizes on which it's just running or do I compile it for that, for that, for that, for that? It's just a question of how you do it. Yeah. Good question. So we have just one image and that image knows how to talk to Zen and knows how to talk to KVM and will know how to talk to VMware in the future. We don't have, since it's just a small number of hypervisors, it's very easy to build and support for all of them. Okay. Thank you. Part of your image already? Yes. So there are just six drivers. We need the one each for network and block for each of the hypervisors we support, and that might grow both in terms of the number of hypervisors and in terms of the different types of drivers. For example, KVM has both Verdioblock and Verdioscuzzi, and we will want to support both, but still the number of drivers is on the order of a dozen, so it's very easy to support and to compile into a single image. So there's no issue moving from one hypervisor to the other. You just take the same blob and flop it down. Yeah, no issue. It's just a few megabytes of binary. It's a very thin system. All right. Any more questions? Okay. So thank you very much for attending and come here, collect your boxer shorts.