 All right. Thank you, everybody, for joining us for Big Data and Machine Learning on OpenStack backed by NovoLexD. My name is Ryan Beisner. I work for Canonical on the OpenStack engineering team. This is my colleague Andrew McLeod. Hello. Who also works for Canonical on the OpenStack engineering team. In an interesting twist, he joined us from the Canonical Big Data software engineering team. So we decided it would be an interesting combination. To take the two products and mix it with a thing called NovoLexD and see if we could achieve bare metal or near metal performance in some big data applications. So naturally, that comes with a set of challenges. Some of the challenges are our own, and some of them are similar to things that people run into, whether they're running these things on any sort of system. So one of the first challenges we come across is the performance, which is perhaps a major reason some of you here, trying to get a big data workload performing well in a cloud and trying to dispel the popular idea that you can't really run a big data workload very well in a cloud. Right. So some of the other things that we want to identify as challenges are some of the notions of cost and efficiency in operating bare metal clusters with regard to potentially unused bare metal for periods of time that could otherwise be used in an organization. And that leads us to some other things in a bit about efficiency with regard to managing these systems. We'll touch on that at the Big Data software bit. So that leads us on to data security and data locality. It's a natural challenge that you have to attempt to resolve, and then following on from that. And so Big Software is a thing. I consider Big Data, Hadoop, to be a very kind of complex workload to manage or maintain. OpenStack itself is not a simple thing. And I would consider that also as a component or as a representative piece of software that we would call Big. So Big Software is complex. Big Software is a challenge that can be expensive to maintain, to deploy, to manage, and to use over the lifecycle of a product. So there are things that can be done to help make this a little easier on organizations. And I think that modeling is a huge thing. So being able to model, and deploy, and manage these applications while encapsulating the operations in a model as well is really key. Some of the things that we do with OpenStack and the OpenStack charms have been done in parallel with the Hadoop Big Data charms, rather. And we need to be able to deploy, and model, and manage these applications on multiple substrates. Such as bare metal? Ah, yes. OpenStack with NovaLXD, OpenStack with NovaKVM, and well, any other public or private cloud, really. Such as? Such as anything, anything. All right, cool. So we want to be able to take really complex things and deploy them anywhere. This is a worthy challenge, right? This is the challenge of deploying, particularly in our case, Hadoop and Spark and all of their friends, on these various scenarios. So we do this with Juju. What is Juju? It is an application modeling language that allows us to encapsulate the operational and deployment and administrative wisdom of network engineers and system administrators representing these applications in a nice pretty package that's shrink-wrapped into a thing called a charm. These charms are, as I said, things that you can take with you into various substrates and redeploy and manage those applications on those various substrates. And one way you can do this with bare metal is to use Maz, metal as a service. So the idea behind Maz is that you can cloudify your data center and your bare metal. You can manipulate it with an API as if it was a cloud. Very useful bit there. The other bit that we want to talk about in terms of the tools that we'll use to test this scenario is something called LexD. The Nova LexD software is open source. It's part of, or it's developed in the upstream repos. And it's basically a machine hypervisor that allows us to achieve near-metal performance using Nova. We want to be able to achieve, and we can achieve with LexD, very low latency. And if you so choose, you can use it in a couple of characters. You can use this to, as I said, get near-metal performance, but also you can get much higher density than you would traditionally get with traditional VMs. So this is an example of a juju bundle, specifically a big data bundle. It's a logical view because you might see the slave service there that might represent one to five slaves, for example. This is not the exact bundle we use for the test, but it's pretty close. Right. I think the logical description here is right on because this doesn't necessarily represent the number of nodes. These are the relations of these applications that together make this big software, this complex thing. There may be 500 nodes in some of those scenarios, or there may be one. So this is the same thing, again, looking at OpenStack with the OpenStack charms that are developed upstream and are part of the upstream projects now. These OpenStack charms encapsulate the operational knowledge and wisdom and all of the various things you might want to do to manage over a long period of time, not just deploy, but to manage these things over the lifecycle, including things like upgrades, maintenance, downtime. These things are encapsulated as actions within these charms. Again, this is a logical view. In this case, you see how each of the applications relate to one another. And if you all recall back in the day, and this may still exist, there's the big OpenStack diagram that has all of the various dotted lines snaked around and things like that. So this essentially represents what that diagram also represents. So this isn't related specifically to our big data application. But speaking to the power of the toolset and being able to model the operations and model the applications, the canonical distribution of Kubernetes now has a juju bundle available where you can take this set of applications representing another piece of big software, Kubernetes, also not a simple thing to achieve as far as deploying by hand and creating these clusters. You can take this bundle, deploy it on bare metal with MAZ. You can deploy it on top of OpenStack with juju. You can deploy it on AWS and other public and private clouds. So there again, same tools, same charms, same software, deploying in multiple locations. And I think that's the real power of this toolset. So this is a summary of the toolset. This is application modeling. This is operational modeling. Indeed. So enough about the tooling. Perhaps we should get into a kind of real world use case. We wanted to talk a little bit about machine learning. Those that are in the room who work with big data probably know about some of the benchmarking tools and things that Hadoop and Spark possess. We wanted to pull in something that's not ours and it's not a canned test. And so we have this scenario from one of our colleagues in the data science industry that he graciously allowed us to reuse and rerun these Spark jobs entitled Using Spark for Anomaly Fraud Detection. So this map clearly we found it in a hypothetical Ryan's attic in a hypothetical treasure chest, even though it's digital. So on this map, you can see some dollar signs, some locations, a little clock with a tick. And this is the area that hypothetical Ryan generally buys things. His purchases are normally quite small. And the time of day is more or less exactly when he should be working. So if we zoom in to the next slide, we have the same thing. So this is hypothetical Ryan's normal kind of financial transactions on this credit card. So then hypothetical Ryan goes to a hypothetical conference in Barcelona where he attempts to buy a laptop. This is strange because it's quite expensive and it's a strange place and it's a strange time of day. So a call center rings up hypothetical Ryan from the bank and they ask him, did you try and buy this laptop? He says, yes. They say, OK, cool. They put it into the system and the unsupervised machine learning job continues with this new data. So this is a science picture. This is some science here. The blue dots are the average, the cluster in the middle is the usual purchases. This is what the system considers normal. The dots in the concentric circles are what the system considers to be outliers. So the way that this particular job works is quite a simple spark job actually. And we have the information about the job at the end. It takes the price and the distance and you feed it in a training set and then you feed it in a cross-validation set and you run that job. So that's essentially what we've done for the test. Awesome. So this is an interesting kind of thing with anomaly detection specifically, I think. And so we haven't conducted what I'm about to say, but we've got some ideas and this is kind of things that we want to share with other folks who are interested in this kind of thing. One particular use case for OpenStack operators could be things like log analysis and failure prediction, also for capacity planning. If you have some machine learning, unsupervised learning tuned to your logs, you can imagine some pretty strong possibilities of doing some pretty creative things with your cloud. I think there was a demo that you did at Strata too. So another example of this was we had some fans and we were measuring the RPM on the fans and that's fairly normal for these new fans. And then we put a little piece of Blu-Tac on the fan blades. And so the RPMs changed and the system detected that there was an anomaly and it made a pretty graph and sent some emails. Right, so that metric could have been anything like disk contention, IO network contention, things like that could be indicative if it's a pattern and that could be trained and such a thing. So who here has actually deployed or managed Hadoop Stack? And OK. And so what's the fastest time someone's done that in? A week. A week? How about a day? And maybe that's the first time you did it. But what about the second time? That was the second time. That was good. That's fair enough. So that's fair enough, exactly. So with Juju and the application modeling, we're able to deploy the same stack onto whichever substrate cloud we choose. Well, it depends on your bandwidth, but say 15, 30 minutes to download the Hadoop packages. So who knows what a pterosaur is? OK. So the pterosaur is basically the industry's standard way of testing the performance of MapReduce on some hardware. So basically it generates, you get your system to generate a terabyte of data. You map it, you reduce it, you see how long it takes. Now, we decided not to run pterosaur for our tests because of the hardware limitations that we had and the time we had to run the tests. So we ran gigasorts. So it's exactly the same job. It's just a gigabyte instead of a terabyte. We made sure to run at least 10 on each substrate, to be fair and get a nice average. And here is a picture of some of these jobs. Right. So let's talk a little bit about the systems, the hardware and the software that we used to conduct these bits of research. First of all, I want to point out that we used the same stack of hardware for three different scenarios. And those scenarios are deploying our Hadoop Spark bundle on top of bare metal directly with Juju. The second scenario is doing the same on top of OpenStack with Nova Lex-D as the hypervisor on top of the same hardware using Maz. So we've got a little level of inception there. The tooling is doing a couple of layers. The third scenario is exactly the same as what I just described, the Hadoop bundle on OpenStack with Nova KVM as the hypervisor on the same set of hardware using Maz below that. In all three scenarios, we have five Hadoop slaves that are all guaranteed to be isolated on one compute node or one piece of metal. So this is highly comparable now once we've dissected it in that way. These are commodity machines. This is 12 Dell R610s. They've got 48 gigs of RAM. They're nothing special. We wanted to conduct this really not as a comparison as to who's doing the fastest Terrasort or any of that. We wanna see this workload, whatever it be, performed in the three different scenarios. So that's what we've done here. We ensure that these Nova instances land on dedicated compute nodes when in the OpenStack scenarios by creative use of flavors and machine scheduling and host aggregates. So these are all Nova scheduler features that you can leverage to do this with Nova KVM or Nova LXD. So that's how we achieved that. Talking about the software that we used here. Again, same software for all three scenarios. Bantu server, 1604, the LTS. Dujou and Maz, of course, the OpenStack Charms. And there again, these OpenStack Charms are upstream and available through the GetGarret common workflow. You'll see, if you monitor the Garrett stream, you'll see that those are there. So this project has PTLs and governance in the full nine yards. It's an official OpenStack project. We also have the Big Data Charms, which are hosted and part of the Apache Big Top repo. So these are also in an upstream development centric, developer centric repo. OpenStack Mataka is what we used here. We started this research when Newton was about, I don't know, it was like July, August. We wanted to run some longer-term things and so we chose to use Mataka. I think that you could use all of the latest releases and probably achieve the same. And there again, if it's important, the folks who are big data centric, Spark came from Apache Big Top, 151, and Hadoop271 from the same. So what does this look like when we pull the results together? Purple. It looks purple, okay. So this first graph is the results from running the Spark anomaly detection job that we explained before with the pretty map. This is not a raw Spark timer. This is the job time from submitting the job on the command line to the completion of the job. That was run a minimum of 10 times on each substrate. That, the axis there is the test duration, as you can see, and we found that Nova KVM was one point almost twice as slow as bare metal compared to one point one times over electricity. And that was the Spark machine learning job, right? So this is kind of a mix of disk IO and CPU and RAM. It does a lot of different things. It's not particularly heavy in any one of the areas. So this next graph is one of the graphs from the Terra sort. By the way, we ran these things on public and private clouds. They're a bit irrelevant here. We wanted to show them just because we gathered the data and that really illustrates the model driven application bit where we did deploy the exact same topology with the same applications across basically anywhere we wanted to with the juju tool set. Right. The other reason was that I was able to test my jobs on those public and private clouds before the metal was available. Yeah, because I was hogging it. So these, the orange are the map jobs and this is megabyte seconds. So it's the amount of memory that was allocated for all of the map jobs times the amount of time it ran. It's called seconds, it's actually milliseconds. There's a patch in Hadoop to resolve that. We can see that LXD in this case was almost identical to bare metal. There's a slight increase on the reduce phase which we believe we can modify slightly. And again, you can see a significant difference for Novik Avion there. And this slide is about, again, we ran the same test on the public clouds. It's not a competition at all. The machines we use on the public and private clouds were much lower specced than our hardware. So we're not trying to paint a picture there. The purple here is the CPU time spent on successful tasks, more or less the same. The orange is the V core seconds for the map jobs, which is basically CPU seconds, again, it's milliseconds, what they call the seconds, times the number of cores. And the gray is the same thing for the reduce phase of the jobs. And you can see the same result there. Awesome. So if we look at these again, I wanna just look at these again. The Novilexd bit, we really didn't know what we were going to see. We assumed that things would look good. We didn't know how good. Some of the research that's been done in the past was more in terms of density with regard to Novilexd. Because if you think about this on any given compute node, with Novilexd, you have how many kernels running? Exactly one, if that compute node is hosting Novilexd instances. If you have one KVM instance, you now have two kernels running and then three kernels running. And you can see where that could balloon. So there are some things that you would just assume to be true, right? What we wanted to do here is exactly the opposite of the density story in terms of let's not look at how many of these we can pack into one compute node relevant to KVM. Let's look at what it looks like to do just the opposite and give one Nova instance an entire machine in a similar way that you would want to in a data center. But then there again, this thing is exposed to the API and exposed to the tenants and the organization through all the same mechanisms that any other Nova instance would be. So just buzzing back through these again. There were some anomalies or not anomalies, that's in the machine learning bit. There were the 11% difference when we look at what we think that is. If we think about how the bare metal job was run, you've got applications laid down directly on the iron versus in any open stack that we've deployed. In fact, both of these in our test scenarios run overlay networking with OVS. So there's overhead there in terms of the network side of things. We think that if we eliminate that with the use of VLANs that we can probably shave off a few percentage points as well in the difference between Lexity and bare metal. So I think that makes the case very strong for being able to basically forklift applications from traditional virtualization, VMs, KVM into these machine containers posted by Lexity and Nova Lexity. It's important to note that these type of containers are full machine, full system containers. You get SysLog, you have SSH, you have all of the things that you would have in a metal deployment or in a virtualized KVM type of machine to contrast that with other container technologies which may provide application level virtualization or containment rather. So this means that you can take many more legacy type applications and do essentially zero work to them and take them out of a traditional VM and put them into a Nova Lexity VM. Whereas taking some legacy applications and putting them into application containers could be quite complex. And so that is the bit about Nova Lexity and the data that we saw. What does this look like in terms of operating the cloud? It looks pretty much the same. You log in to your horizon. You have hypervisors. It's important to note that you can use this in a mix. So you can have Nova Lexity, Nova KVM, Nova other things all in one cloud and you can control these with creative use of things like flavors, host aggregates and achieve this exclusive machine scheduling if that's what you're aiming for. So here's just an example of a couple of flavors we created to achieve this. What's interesting about this is we actually gave our flavors ever so slightly less than the full amount of RAM or the full amount of disk because I didn't want to overrun whatever services like OVS and Nova Cloud Controller and all those things that may have been running rather than Nova Compute Services. So host aggregates there again is one of the levers that you pull to do this kind of thing within Nova Scheduler. At the end of the day, your instances which really represent, if you think about what we've done here, this represents a chunk of bare metal that you can treat as any Nova instance and it looks like any Nova instance. You attach networking to it like any Nova instance. We've got some feature work underway to enhance attached storage and things like that. I think I SCSI volumes are a thing now. We can do that. There are some things I think Seth backed storages on the way, perhaps in the next cycle. But today, you can take these legacy applications, take them out of a traditional virtualization type of approach into this and really free up a lot of RAM, CPU. Your disk IO has an immediate benefit, not running through the traditional QMU KVM stack. So that brings us back to the challenges we pointed out at the beginning and how we think we addressed them. So I mean, well, you saw the graphs. So we think we have addressed the concerns that you can't put big data and have it perform well in a cloud. So I think that, well, that's clear. The results that we got from running these jobs are available in our repository. And I think we've addressed cost and efficiency in the way that a lot of organizations, universities, enterprises, they have their HPC cluster, their Hadoop cluster, whatever, separate from their cloud. With this, we can bring the hardware into that cloud, meaning it's more flexible, it's one cost center, it's easier to administrate, it's all sorts of things. We looked, I mean, if you are then going to run your big data jobs in your cloud, you can put it where you want. So that's the data security aspect. Sure. As far as big software, I think that it, you know, I hope that we're all kind of all on agreement that things like Hadoop and OpenStack and Kubernetes are big software. These are challenges that most organizations probably don't have and probably shouldn't expect to find folks that are subject matter experts in that full stack, whether it's vertical or horizontal, things if you can imagine someone knowing everything about OpenStack. Okay, we all know some about OpenStack, we all know a lot about parts of OpenStack, finding folks who can sit down and deploy a full stack with all the core products without an extreme amount of either time or tooling is a rare thing. And so modeling this, modeling this knowledge, modeling the configuration of it, the deployment of it, but more importantly, I think the management and the life cycle management of that application, whether it's Keystone or Spark or any other number of things, HDFS yarn, right? Each of those have particular domains of knowledge, right? That are in some cases rare. So I think that the modeling of this from the application perspective and from the operations perspective is quite compelling. And so these tools are all open source. Juju, Maz, a bunch of server, the OpenStack Charms, the Big Data Charms. LexD itself is upstream. Nova LexD is, again, developed in the OpenStack GetGarret flow as well as the OpenStack Charms. So if there are particular questions down the line after today, OpenStack Charm developers, we usually hang out in the OpenStack Charms free node channel. Welcome you to join us there and have conversation. If you try Juju or have questions on Juju that are general or if they're about the Big Data Charms, anything that's a general Juju type of question, we're all also in the Juju channel. And then there again, things like the Ubuntu Cloud Archive which I'm sure everybody either uses and if you don't know that you use it, you probably do, those kind of conversations we hold under the Ubuntu Dash server channel. So that's what we have to talk about in the Big Data and Machine Learning on OpenStack back by the awesome Nova LexD. If there are any questions, we'll be happy to start chatting about that. And please do, if anyone has questions, step up to the microphone for the recording. Thank you. Hello. Do you have any investigation on a reasonable configuration of the Big Data, a resource configuration for the Big Data server stack? So do we have any documentation of that or examples? Or just are you asking if we've changed those settings and tried to run different jobs? For example, how many RxD VM or KBM VMs running on a single host? Right, exactly one. And what storage do you use? So these are... Local storage or... Yeah, it is, it's local storage. These are very entry-level servers. They became disk-contented for sure. It's got near-line storage, sat of 500 gig drives. So basically taking CI test environment and turning it into a rig that will run this. These are six years old. These machines are not... They weren't purchased for this task. So in terms of configuration, it's probably worth pointing out with regard to the Big Data stuff that the charms encapsulate all of the recommendations and best practices of the Apache Big Top. Right. So if you only run one VM on a host, so why do we use the... Why do we running Hadoop on the open stack? So that's an interesting question. The application here is that I want to achieve as close to bare metal performance as I can. In other words, I don't want to share the resources of some of the compute nodes. I want Hadoop Slave to own that box. I want to be able to use all the RAM, all the disk, all the CPU. I don't want to be contended with other tenants or other VMs. So that's the reason that we did this. And again, it's a bit backwards. Why would you do this in a cloud? And so I think that if you think about the university research kind of department where perhaps that metal cluster from the data science folks goes unused for 10, 20% of the year, maybe they conduct certain research, but this metal may not be actually spinning and the CPUs are not providing compute resources to the rest of the organization for 10 or 20% of the year or whatever the case may be. If we were to take what we've done here and apply that in that kind of scenario, what would happen? These machines would be better utilized the cost effectively goes down in that case. It's also easy to manage. And you can choose whether you want to have one container, one slave, one piece of metal. I mean, you can aggregate all of that into one place. So it's also probably worth pointing out that not everybody is going to want to run, even after stuff like this, they may not want to run Hadoop jobs in an open stack environment. They may be so married to this set of metal and they don't even want the 2% or the 11% difference. And in that case, there again, the exact same procedures, the exact same charms in the encapsulation of the knobs and the levers that we can configure on those charms are usable directly on the metal with MAZ and you take open stack out in that case. Does that answer your question? All of that together? Thank you. Are there any other questions? Let's keep it rolling. Great. Hi. As far as I understood, the charm also goes into the details of the Hadoop configuration and Spark configuration. So it looks like a bit replacing the original project open stack Sahara that is also in the Ubuntu packages. So what's the guideline? Like install Sahara or get rid of it on use-day charm? So I'll answer part of that because of the team I was in before we have this big data project. So it was natural that I would use what I knew already. And then Ryan can address the rest of that. Okay, sure. So in this case, our challenge was I want to be able to model my complex application and I want to do that for what I need to do today. But if I'm going to go into the work of modeling that, I don't want to be married to one particular platform. Down the line, the organization may shift directions or I may need to do some hybrid operations or I may need to do this directly on metal. And so that was ahead of all of this. That was kind of the rule number one for this was I wanted to be able to do that. Granted, through creative use of Ironic and Sahara, we could probably have achieved a lot of this. It's simply not something that we have charmed. But I don't know that you would be able to take and run on the same set of machines and achieve within that confine, even with Sahara and Ironic because you have to have a control plane, right? The OpenStack control plane consumes something there. I don't know that we would have been able to achieve the same type of measurement. If we had 30 or 50 or 500 machines, we could have carved maybe some of that out for the control plane and just forgot about it. But if we take the approach of same amount of hardware, same amount of resource, that was really what we wanted to compare. So I don't discount the Sahara's validity in a pure OpenStack environment. I think that's got use cases clearly it has users. I don't know what the reality of taking Sahara onto AWS or GCE or directly to metal is though. That's a great question. Thank you. Would just one more thing. The storage big cam do you use the comparing Nova KDM and Nova LXD? I think it was the same, what was it? Okay, so yes, these are local disks. So nothing fancy really. We deployed the under cloud bit, if you will, the OpenStack part is deployed with simply two 500 gig near lined SATA spindles. Seriously entry level stuff. And there again, the goal was not to see how fast we could do a giga-sort or a terra-sort. It was to measure the same thing three times in three different ways with the different technologies. Thank you. Yeah, thank you. Good questions. How about some more? I don't know if I got it right. So you provisioned a Nova machine, a Nova computer machine with a KVM virtual machine and then you run the containers inside the virtual machine, right? Ah, no. You provision the containers directly on the computer. That's correct. I think I understood. Yeah, good question. Yeah, so there's no nesting happening here in any sense. Thoughts, questions, thoughts, questions, thoughts, questions. Right, so there's some information. Oh, sorry. All right, good. We figured this is the, I'm sorry, go ahead. Cherry from Orange Cloud. When we did big data, we have a difficulty to deal with the terra, you know, like big data, we need to do a lot of terra, byte and process. So in the terra-sort, terra-gen, and we figure out that we need to optimize the disk attachment with the VMs and the tweaking there. And also if we want to have a, in multi-tenancy, we may have an SDN. So the optimization of the whole is a combination of the PCPU, the SDN, the disk attachments. So here it looks like you really looked at the CPU. What about the other things? And what about the terra? So the QMU KVM, first of all, good questions. The KVM scenarios that we ran, when we watched the system resources of the metal, the disk IO was completely pegged. It was contended. There were times where the disk weight was not where you would ever want to be. So that could be certainly improved by using some sort of better storage or centralizing storage in a way. And there were, Andrew, add to that. There were things we wanted to try, but due to time constraints and resource constraints, we couldn't. So we acknowledge that if you go and modify, if you go and tune everything, your results may be different. But in this repo here, we're providing so that you can take our tooling and reproduce it and test it for yourselves. But that's an example of what you do out of the box. So I don't know, does that answer your question? More or less? Very good. Thoughts. I see some thoughts out there. Are they gonna turn into questions? The images don't work that well. All right, so the images, if I may, we can actually go have a look real quick here. Oh, you know what? Sorry, there was an images slide. It's gone. And so yes, if you look at the, where you normally would get your a bunch of daily images for the server, in the same tree, you'll see the Lexity images there as root Tars. So these images are in glance. They have some image attributes in some cases, especially if you mix KVM and Lexity, you've got to kind of tag them with glance image attributes. And then there again, you basically tie that image, or rather that server or each of the NOVA hypervisor hosts, to a flavor with host aggregates. And then it just does the right thing. NOVA scheduler. There you go. I'm sorry, what's that? Those images are available. Yes, images are publicly available. So I'll repeat the questions. The question was about how we manage images clearly by my answer. Forgot to repeat your question there. Are there any other questions or things to add? I think, yes, great. Connected to what somebody had said before. You show graphs where you differentiate between the mapper phase and the reducer phase. And yes, exactly. It was interesting because, yeah, okay. Here we see that LXD and KVM, they both go higher than the bare metal, but they go higher in a different way. And this was unexpected to me or maybe it shows where the bottleneck of the two technologies is and probably it's a different bottleneck because they lose into different phases. Well, right, so when I saw the results, I thought that he'd given me broken hardware. And then I checked my results a few times, obviously, because I had the same thought. Essentially, what it looks like has happened is because of the disk IO contention, there were IO errors. So I saw IO errors once I looked at the results properly. And the IO errors have resulted in... Under KVM. Under KVM, right. Have resulted in further allocation of containers to reattempt those shuffles, sorts, shuffles, producers. Okay, so you three, sorry. It really looks from your research that you are gaining not only in CPU, but also in managing the IO from the disk. So bare metal is gaining a lot also... Sorry, I mean containers is gaining a lot also on the disk management, not only on the CPU, right? This is the outcome of the... Right, that's one conclusion that I don't disagree with that the disk IO is when you're in a scenario that's over committed or over contended or you're asking too much of the hardware that that's handled much more elegantly with Novel XD versus traditional KVM. I think that's what this shows. We did ask way too much of the metal by the way. Good questions, happy to take more as long as we don't run out of time. I know it's the end of the day. Repo. Right, so yes, that repository at the top there contains the tools we use to configure the environments to run the tests and it has the results in it. So if you want to, it would be fantastic if at least one of you could go and try the same thing because then I'm either right or not. Or crazy. Or crazy or whatever, but that would be fantastic. But if you want to have a look, that information is all in there. All right, so the bundles. Includes the bundles. Caveat, these are proof of concept bundles. Don't go like deploy these in your data center and start a new website and stuff like that. We haven't really tested those to that degree. Though these can be tailored to match and deploy in a way that's highly available and all of that good stuff that you would want in an enterprise. This stuff was clearly not highly available with 11 machines. If you're interested in Mas or juju, mas.ao is the place to go. jujucharms.com and other places. Like I said, just look for us on OpenStack Anywhere. Look for the word charm. You'll find our stuff. If there are no more questions, I want to thank you all for being here and joining us. Thank you.