 All right, so let's get going. This is, as I said, just a second ago, workloads are a matter. And we've had a lot of talk today about infrastructure, infrastructure management. I think the last session was very interesting about NFV, a very specific workload, a very specific use case. But now we're going to talk to about more general data processing, infrastructure, and application deployment frameworks. So in this session, we're going to be talking about Hadoop and Cloud Foundry, which is the leading pass for OpenStack. And as ever, it's going to be a session packed full of demos. We like to fly dangerously, live dangerously. So Samuel's going to give us some more demos. So we're going to start with Samuel, then we'll be passing over to Dustin and Compil, and then finishing on a high with Pivotal. So thanks. OK, we made that choice to talk about big data today, because while it's a pretty new technology, and it's a bit like an HSE, so everybody talks about it, and nobody actually does it. So that's what we want to fix. We want to make things that are very, very complex, very simple for people to use to consume, so that in a sense, they can still talk about it, but then they can actually do it as well. What we're going to do is try to find a news that is related to big data, and see what we can do with it, and then we can transform that news into an actual workload and deploy it in minutes with Juju. So if you remember about five days ago, IBM and Twitter announced they signed a partnership to make tweets available to companies. It's not only tweets, it's the information you can collect from tweets, and then you make that valuable data-driven decision-making. So if you look at that, and you say, OK, what should I do if I want to do it for myself? What is involved in actually taking tweets and analyzing them and turning that into data-driven decisions? First of all, we're going to look at what it represents. So in that news, and that's also what Twitter advertises on Wikipedia, they said they had half a billion tweets a day, so that's about 6,000 tweets per second. One tweet you collect from the streaming API from Twitter is about three kilobytes, so that's about 140 megabits per second of bandwidth, which turns out into 600 terabytes of raw data per year. So if you want to store that and have a replication of three, for example, it's about two petabytes per year. So that is big data, OK? You can't feed that on a single machine and you can't also process it with a single machine, so you need that big data tools to fix that. Now, what we're going to do is have a look at how, what are the technologies available to fix my problem, which is analyzing tweets, store them and provide valuable information. And when I look at that, I've got so many technologies that I haven't had it already. And that's one of the big problems of big data. It's like, there are so many solutions that they are actual problems, OK? There's no problem to fix. But there are shortcuts for this. So in our example, what we want to do is get the Twitter feed, push that into a big fat storage system, analyze it in real time, and then present it to users. We can use that set of technologies here. So you see five boxes. Kafka is a push-pull messaging system, so it moves data from Twitter to your workload. A dupe, I don't have to present it. It's kind of the canonical example of big data. Storm is a real-time processing system that was built by Twitter. Redis is an in-memory database that we can use to aggregate the views from a dupe in Storm and present it to an RGS application or any front-end. If we dig a little bit more into this, it's actually not as easy as five steps. Kafka relies on ZooKeeper to run, so it's two services. A dupe doesn't mean anything if you only have one box. So we're gonna scale that up a little bit and have four nodes, four compute nodes, and one master node. Storm also is a distributed system, so you need to scale it a little. So we're gonna also have five nodes, one of which is a master node. And then Redis, Node.js, HAProxy, we don't have to scale immediately, but those are workloads that can scale as well. Now, if you have to start from scratch with this, you've got eight technologies, at least 15 VMs to start, and more than 2,000 pages of documentation. So in your opinion, how long is that gonna take before you get a first client set up or first set of analytics on tweets? So would you see it's six months? Three, one, 15 minutes? Okay, so you do that in 15 minutes? You don't? Come on. Actually, it's 14, and that includes deployment, okay? Why is that? It's because when you design something with Juju, you can save that workload and redeploy it, so you can spend time designing things, but then you can share it, and you can deploy it on any cloud, including bare metal. So we're gonna do that, actually, together. This is jujucharm.com when you can test workloads and it doesn't actually deploy things, but you can look at what it would look like. So I'm gonna import the bundle, which is sentiment analysis. Okay, we deploy it. So you could do that on any cloud, on OpenStack, on bare metal. You can also do that if you have a powerful enough computer locally on containers. So this is what it looks like. As it takes a lot of time and I don't have that time, I did that for you on Amazon, so you can see it looks exactly the same, okay? And this is live, it's running. I can go to storm UI, can refresh that page, and I can see that it's processing things. So that says that over the last 10 minutes, it processes 200 tweets. Now this is my dashboard. I'm gonna refresh that so it's gonna get empty, okay? And after some time, it's gonna load new tweets and you can see that in that area here. Okay, it's coming in. So you can see a positive score, a negative score, and the text that was analyzed, which is the text of the tweets passed and where we removed all the words. Now eventually, you're gonna have to scale that. So it's fine to have something that works just for starters, but you want more nodes in storm, for example. So you can also hook that up to a monitoring system. So that's an example made with Xabix. You could use Nagios, you can use that to any monitoring system you have. You can deploy that monitoring system with Juju, or you can use your own existing infrastructure. Now I'm gonna go to storm. I can see that I've got many nodes here. Oh, sorry, just one more step. I've got more workers. I can see I've got four workers. If I go back to my Juju admin here, and I go to my workers, I can see I've got four working units. So from there, I'm monitoring the load on the servers, and if I find out that one of them isn't actually working properly, or if I want to scale up because I've got more and more tweets, then I can just add a unit. I made that manually, but you could use that automatically. Now going back here, you can see I've got a fifth unit starting here. So you have a scalable scale out workload you deploy on the cloud or at home, then you hook that to existing bits you have, and you can have auto-scaling automatically. So it answers an issue. You want to analyze tweets, you want to store them, you want to make from tweets valuable information, and you want that to last. That's an example of what you can do with Juju. Thank you. Any questions? Got time for a question? I think we're good. Thank you, Samuel. Yeah, I need to switch to my laptop. Fantastic. My name is Dustin Kirkland. I work at Canonical. Developer, product manager, work on a number of things. This is my colleague, Kapil. He too works on a number of things. Related to Juju, Maz, OpenStack. Here we're going to talk about two very special workloads to us that we often find ourselves running on OpenStack, and that is Hadoop and Cloud Foundry. Now for anyone who's been to an OpenStack summit before, anyone who's certainly been here the last couple of days, you've probably seen us deploy OpenStack. You've probably seen us deploy OpenStack on these orange boxes, which we're going to use in this demo, perhaps on some much bigger hardware, noisier hardware, data center caliber hardware, such as these AMDC micros, and we've done that using a tool that's very special to us. But is that all that you've seen? Have you seen Juju deploy Hadoop and scale Hadoop yet? It looks something like this, which looks remarkably like this, right? Lots of components, disparate components connected, some tightly bound, some loosely bound, some replicated, a bunch of common services. This is what a small subset of the Hadoop ecosystem looks like. You've got, of course, the Hadoop master and the slaves, but you've also got Accumulo and Yarn and Pig and Hive and MySQL and Spark and HBase. This is a really interesting collection of big data technologies, and this is what it takes to actually do some of the hard work that Samuel just showed us in his bundle, the sentiment analysis, involved multiple components of technology. But it doesn't stop there. We can use that same tool to deploy Cloud Foundry, which is our next most interesting workload that we can deploy. And Cloud Foundry, much like what you saw just now with Hadoop and much like what you just saw with OpenStack involves, what's that? 17, I think we counted, 17 separate services that all have to work together to provide a platform as a service, to provide the best platform as a service you're gonna find in open source today. And some of these components seem a little bit similar. You've got a logging mechanism, you've got EtsyD, you've got MySQL, you've got an HA proxy, you've got a Cloud Controller and so forth. All of these pieces have to be up and running and connected to other pieces. You can see that the Cloud Controller over here on the sort of your lower right connected to almost every other piece in the entire infrastructure. That's part of what Cloud Controller needs to do. It needs to have those connections associated with everything. And that is fundamentally a difficult problem to solve and something that we've done for you with Juju and with the charms that Juju deploys. That's Juju, that's the power of Juju. And we're talking about three of the world's most complicated workloads today, all deployed by a single tool. All deployed by a single tool. And what's important about that and why we're so passionate about this and why we're so interested in sharing this with all of you is that it is one piece of technology with very reusable components. You'll see MySQL is used in all three of those deployments. And MySQL is fundamental to OpenStack. It's fundamental to what this workload is doing with Hadoop and it's fundamental to what's happening with Cloud Foundry. That MySQL charm is the exact same code and logic and expertise that has been baked into that one bit of code that we call a charm. And it contains everything that's necessary for that MySQL to operate in any environment where MySQL service is required. And the same goes for the rest of those pieces, the common pieces, especially the common pieces, the MySQLs along with the rabbit and the XCD and so forth. So that's three really complex workloads that can be deployed to three very important places, public clouds, private clouds, and bare metal. Now, Gigi supports a number of public clouds, including AWS as well as Azure, Google Compute Engine, Joyant, what else am I missing here? Azure. Azure. And we're adding more. These are Canonical's public cloud partners. Any charm that can be deployed can be deployed to a public cloud. In fact, you see OpenStack being deployed to a public cloud here. Where would that make sense? Certainly in our testing and our regression testing, we can actually gate our commits on deploying an OpenStack to AWS for the sake of ensuring that those services come up and work well together. That's actually where we do some of our IPv6 testing is actually deploying OpenStack into an environment such as a public cloud. It also can be deployed to bare metal. And that's the piece that hopefully you've seen a couple of times today. But it's not just OpenStack. We can do the exact same thing with Hadoop and Cloud Foundry. We can deploy Hadoop to public clouds, private clouds, and to bare metal. We can deploy Cloud Foundry to public clouds, private clouds, and bare metal. But for the sake of today's talk, we're really interested in two specific workloads. That's Hadoop and that's Cloud Foundry and we're interested in one place. That's deploying it to OpenStack, to a private cloud. So we're gonna start off with a Cloud Foundry demo and I'm gonna turn it over to Kapil to do that for us. All right, Kapil. Thank you. So I'm going to assume that most of you know what Cloud Foundry is. And if you don't, the quick intro is that it is a enterprise pass for simplifying operations and deployment in a uniform way across an organization to allow for rapid application delivery. I've got here a sample application written in Node.js. I've pointed it, it's a simple app that just looks at a GitHub repo and will calculate the number of the most valuable contributors in a 80s as key arcade style game. And I can reload it. And it's running across on OpenStack and here, oops, here is that Cloud Foundry. And if I go to, so we've got three levels of inception here. Let me interrupt you just for one second. When Kapil says it's running on OpenStack, it's actually running on these two boxes. He's running Cloud Foundry on this box and we're going to look at Hadoop in a minute on this box. And by on this box, we're running an OpenStack on 10 nodes and then Cloud Foundry on top of those 10 nodes of OpenStack. Sorry, just wanted to link your eyes up to what's going on right here on stage. So earlier today, I was running a benchmark on this High Scores app, just doing the equivalent of a patchy benchmark on it. And I ended up running into this little problem. My Cloud was overtaxed and I needed to add some more capacity to it. So what did I do? I went over to my mass, which had my inventory of nodes and saw that I had two additional nodes that I could use. And then I went over to the juju admin, oops, not that one, that one. And I went to Nova Compute and I added more units. I happened to have these other pages cached so it takes a little bit of time to show it actually going out. So effectively what I did was I just came here, added more units, and now we've got all 10 lights on that orange box deployed running as compute nodes. And we're going through three level layers of scaling and inception here. We're doing OpenStack, adding more compute nodes. We're going to Cloud Foundry to add more DEA application worker nodes. And then we're going to the application itself to scale out the application, to tell Cloud Foundry that we want to run multiple instances of that application. And that looks a bit like this for those who haven't seen it. And now we'll pick up our little benchmark, which is just using a HTTP perf to hit it a little bit. And I've actually got a separate instance of this running just because it takes a little while to add the DEA nodes, but I'll go ahead and do that here just to kick it off. The DEA nodes are sort of the central component of Cloud Foundry that runs the application containers and scales them out. If I can make it out. Left side, right there. Thank you. So I'm going to go ahead and add three more DEA nodes and commit that change. And so now I've scaled that OpenStack. I've scaled out Cloud Foundry and I've scaled up the app underneath. Now technically, or on top, technically I would wait till the DEA nodes were added so that Cloud Foundry will automatically load balance the application instances across all the DEA workers. In a separate view, we have another charm here that does an administrative view on Cloud Foundry. This is an open source application from IBM that just gives sort of a administrative view into Cloud Foundry, allows you to see what all the application components are, how they're doing, what apps are running, and what the overall, how many organizations there are, all of the sort of internals, management details of Cloud Foundry itself. So with that note, I'm going to go ahead and kick it back to Dustin to look at Hadoop. Cool, thanks Kapil. Yeah, so let's take a look at a Hadoop demo, which I'm sorry, but is gonna look a whole lot like the Cloud Foundry demo in terms of the steps that we're going to do, but hopefully that drives home the point how reusable this is. So this is our current Hadoop. We've currently got one, two, three, four, five, six, seven instances of slaves in a Hadoop cluster. Now let's say I want some more. So the first thing I might do is go and check my capacity in my OpenStack Horizon dashboard, and I'll see that, wow, that pie chart on the left is almost full, I've got 11 of 12 of my virtual CPUs are used, I've got a bit of space left as far as memory, but my disk is actually starting to get, my disk usage is sort of creeping up, and that's one of the important aspects of a good Hadoop cluster. So then I'll say, you know what, I want to scale out my actual OpenStack cluster. So I'll next go to Maz and see if I have any available machines and in fact, if you look at node seven and node nine, those nodes are often ready and you can see two blank spots in the orange box where we've got some spare capacity and that we can actually scale this up. So what we'll do is go to our juju view of our Cloud Foundry deploy, which, I'm sorry, our OpenStack deploy, which looks like this, and I can see I currently have three units of Nova Compute deployed and I'm going to take that and I'm gonna add that, add two to that, I'm gonna add two new units to Nova Compute, I'm scaling up Nova Compute, I'm scaling up OpenStack and I click commit and confirm and very shortly you'll start seeing the lights on, somebody tell me when they start seeing lights come on on that far, far right orange box there. If we go back actually to our, our horizon, our UI, and instead of looking at the hypervisors, or sorry, looking at the hypervisors tab, we've got some lights coming on. Got some lights. Yes, this might actually work. Oh, I've got to log in again. Thank you for timing me out, horizon, I really didn't want someone logging into my laptop and messing up my demo and we're gonna do it, again, because it just wants to make really sure. Okay, so we'll see, we currently have three Nova Compute nodes, node one, node five, node four, it doesn't really even matter where these are, all these machines are identical inside of here. That part really doesn't matter, that's kind of the beauty of the way the, the way the cloud works, which is the way the orange box works. What we, what we now should see inside of our Juju GUI, I zoom in a little bit here and we really look at that Nova Compute, you'll see that we have two pending units, we have two units that are in the process of coming up. As soon as those come up, we'll then see additional hypervisors in our Nova Compute stack, at which point we can then go to another level of Juju, another environment where we've deployed Juju, and this is our Hadoop layer. Okay, so we've got to build up from the bottom, we've got MAZ at the very bottom taking care of the hardware, the physical hardware, the CPUs and the disks and the network, the physical machines. On top of that we've got OpenStack, which is providing a virtualization layer that's carving up those, the physical machines that MAZ gives OpenStack. OpenStack is carving that up into virtual compute units, right? Virtual storage, virtual networking and so forth. And then we're actually running Hadoop in virtual machines on top of OpenStack, right? So you've got that, you've got Hadoop on the top and then we've got OpenStack in the middle and then MAZ at the bottom. Juju is talking to all of those, right? Juju's talked to MAZ and deployed OpenStack, that's our big OpenStack deployment right here, where you can see all the pieces of OpenStack, right? But we've got yet another instance running inside of OpenStack. So if I click on our instances tab, this instance down here, our jump server, is actually another Juju bootstrap. That's another instance of Juju running inside of OpenStack as a virtual machine that has then provisioned all of these other instances into OpenStack. So we're running all of these, we have a count of 10 instances. So we got 10 instances currently running in our OpenStack and that's what is pegging our hypervisor right now or our resource utilization is pegged right now. So as soon as those units come online, we'll have more capacity and then we can go into our Hadoop view and change this from seven units to something more than that. And we'll come back and do that in just a second. Let's give this a minute to finish doing what it's doing. Oh, let's take another look at Mas where we had these two nodes that were not utilized. If I refresh this page, we'll see both of those machines are being deployed and do we have all lights on? Still got one light not coming on. All right, we'll look at that in a minute. In the meantime, while this is doing its thing, I had a couple of questions in the hall about what are the orange boxes. This is the most portable cloud I think you'll ever find. There are 10 physical servers inside of this. Each one corresponds to the lights on the front of the box. There are five servers sitting along this heat sink and five on this heat sink. And that's right, these are heat sinks. So the CPU and the video card are nested against this block of aluminum right here. There's no fan on any of the servers. There's one fan on the back cooling the power supply, but that's it. 10 identical servers, each of those 10 servers has a four-way I-5. So there's basically four CPUs on 10 different servers in this box. Every one of those servers also has 16 gigs of RAM, 16 gigs of DDR3 laptop-style RAM. And every one of those servers has a 120 gigabyte SSD, really fast storage, and that's part of the reason why this box is so small and light, and we can do everything with literally these fanless heat sinks. So in aggregate, those 10 servers, that's 40 cores, that's 160 gigs of RAM, and that's 1.2 terabytes of solid state storage, all inside of a box that's probably as big as the last PC that you owned underneath your desk, right? Now, that's certainly not impressive compared to a proper data center box, but the beauty of this thing is that I've carried this to four continents in the last four months, and that is remarkable that you can lug that cloud around and do what you will with it. Look at that, our Nova Compute has scaled up in the time that it takes to describe the hardware inside of an orange box. So now we come over to our hypervisors tab and we refresh this. We need to give it just a minute for those to recognize that. We've got a number of relations that are going to happen. If we come over, we look at this, we can see we're setting up, this is Juju doing this work for us. It's setting up a bunch of relationships that has to happen. If you go back and you look at that Nova Compute tab, you can see all the things that Nova Compute has to talk to. Of course has to talk to the Nova Cloud controller, but it also has to talk to Ceph and it has to talk to Glantz and Rabbit and MySQL, and then it's also got a shared NTP server. We wanna keep the time straight on all of our compute units. We've got Celiometer and these are all the different things that we would need to take care of if we manually added a bunch of Nova Compute. Well, let's see if we're done here. We refresh and there we go. This has just happened live. We've added two new compute nodes, node nine and node seven, and now our pie charts have gone from red and yellow back into the blue state. So we actually now have more physical capacity in our cloud at which point we can come to our Hadoop cluster and add a couple of units. Let's add five units. We're gonna confirm and we're going to confirm. We're gonna confirm and now we're waiting to have 12 units of Hadoop. So where is this happening? If you remember, multiple levels of inception is like that fantastic movie, part of which, the best scene of which takes place right here in Paris, actually. So what's happening, we're now at the layer, not at the physical bare metal, we're at the layer above that. We're at the open stack layer. Open stack is now creating is now creating virtual machines for us. This is a normal Nova, a provision of Nova. Give me a new virtual machine in Nova. Give me a new instance in Nova. And Juju is doing that for us right now in the background. So if we come back over and we look at our instances, we had what, 10 instances running before and we now have 15 instances, right? We scaled up our Hadoop by five, right? So we have 15 instances of that. Now that's actually gonna go a whole lot quicker than the physical provisioning of the system, although I will say being able to physically install an entire OS on a piece of hardware, install Nova and add it to every single, every single, relate all of those services together within five minutes is, I think, still pretty impressive. But what we should see here with Hadoop is this should happen even quicker because these are virtual machines that don't have to go through an installation, a whole provisioning system. And so we'll just give this just another minute. I'm told we have like one minute left. Let me take a question or two while we're waiting for this to complete. Yes, sir. Fantastic question. It's every block represents a service. There are 12 physical units hidden behind that service and in fact, there's two different views to Juju. There's the services view, but you can also see what's happening on machines. So on this machine view, we can now see all the different assigned systems, the aspects of that system. All of these are one gigahertz, two giga ram, 20 gig backing disk. You can actually then create containers. And if you've seen a theme, if you've been to any of our talks today, containers, containers, containers, you can actually put a bunch of services into containers. In fact, if we go to this view, which is a lot more interesting one, you only see one image here per service, but if I then click on the machine view, we should see something kind of interesting. We see a bunch of physical machines, but we see some co-located services. We didn't have to give MySQL an entire system and rabbit an entire system. We actually put rabbit and MySQL both on the same system, which is a lot more efficient usage and something we really had to do to optimize a large, complex service like Cloud Foundry, like Hadoop, like OpenStack to work on bare metal. Yes, sir? So we've decided in our YAML file how to co-locate those bits. So in our YAML file, we can, I can show you that later, but we've said, you know what, it makes a lot of sense to put this and this on these systems. In fact, stay for the next talk. We have an entire session on our reference architecture, on the recommended architecture, what we've learned from the field, from customers, and so forth. So Josh is coming up now. Josh is joining us from Pivotal, where he's gonna talk to us about Cloud Foundry, and I will get back to you on our scaled Hadoop here in a minute. Awesome. Thanks, Dustin. Most of you know me. Already I've been around in OpenStack for a long time. I moved over to Pivotal recently. I'm moving up the stack, and I think I've got just a couple of minutes, so I mostly wanted to focus on why you'd have Cloud Foundry at all, in the sense of, hey, we can do all these amazing things, just using something like Juju on top of OpenStack, so what's the point of having a pass? Pivotal is not the only company who's decided to focus on Cloud Foundry, where they're really the founders of it, but it does have a massive and growing ecosystem, and in fact, there's a lot of overlap between the OpenStack ecosystem and the Cloud Foundry ecosystem, in terms of probably most of the large committers to OpenStack have joined and committed to Cloud Foundry in the last year or so. The reason there's this relationship between the IaaS layer and the PAS layers, I stole this diagram, if you grab a copy of the slides, a link to the original authors is in the slides, but this pizza model of Cloud, I think, is really useful, and it helps people understand why do we keep adding new layers of as-of-service on top of other layers. It's containers, containers, containers, containers, right, except it's not. You put containers on top of operating systems, operating systems inside VMs, VMs on top of hypervisors, hypervisors on top of hosts. We're doing this for actually good reasons. You get different isolation and performance characteristics out of these different primitives, right? I generally tell our customers, don't put containers on bare metal yet, please. Without getting super technical, although you guys have been really technical today, so I was expecting to have to stay very high level, if you have access to bus mastering, you can poison the firmware. If you poison the firmware, the next time that piece of hardware is rebooted, goodbye hardware. And so far, today, there is nothing that protects you from that attack other than virtualization. Don't put containers on bare metal. Put them inside VMs. Which sort of brings us back to, you know, why PAS, what are the benefits we're getting out of orchestrating at this level, right? So just going high as gives you a better SLA, it gives you flexibility, it gives you speed, it gives you availability. VMs are a lot more portable, it's a nice generic abstraction. There are these security boundaries. Going to PAS gives you this insane time to market benefit. When you start looking at, hey, I just write my code and I can deploy it to dev, to staging, to test, to prod without ever making changes, all of the configuration is separate from the software. I have these built-in mobile and data services. The agile development in DevOps actually exists at the PAS layer. It's really hard to get all the way through DevOps just working with pure ISM. We see this all the time. We see, you know, most of what I do for Pivotal is fly around and talk to people who've already stood up OpenStack environments, some of them two or three years ago, as in Asia with customers, got a very large, fulsome deployment. They didn't deploy it with juju, so now they're not quite sure how to upgrade it. But they're saying, look, you know, IaaS got us only so far, typically folks say, hey, it gave us a 30% cost savings, which is huge, but we realized we can get another 50% cost savings out of going to PAS. And the other piece is PAS is not just for developers. The PAS model for operators is amazing because this is the day two problem, right? Day one, I deployed the app, it's awesome. Everyone can show a demo of deploying app. Day two, I need to upgrade that app without turning it off. That's something that, you know, building in a CDCI environment, you have to be able to go through day two. I don't have a clock up here, so somebody's gonna have to give me time fingers or something, thank you, two minutes. You look at, you know, okay, what am I actually doing when I'm interacting with this PAS? I'm pushing these apps, I'm scaling these apps, I'm trusting the platform to deal with HA, right? And again, Cloud Foundry has HA at four separate levels, so it deals with HA by using multiple data centers, multiple open stack clouds, or AWS clouds, or Azure clouds, or whatever. Then it deals above that, making sure that the VMs are managed, so VMs die, they get relaunched, and make sure that the containers inside those VMs are managed and the container goes away, it gets respawned or replaced somewhere else, and then inside the container, make sure the process is running. It's actually no point in having a VM stay up if the process inside is hung or dead. So keeping track of your application as being this entire stack, you know, process inside container, inside VM, inside some cloud, and being able to abstract all of those layers with the appropriate tools, just, it's so powerful. I don't wanna belabor the point on Cloud Foundry, but, you know, aggregation of logs, and aggregation of metric data, and just a standardized interface for roles and spaces is just super, super effective. So I wanted to close this with this idea of Hadoop. I helped a very large telco recently do benchmarking of Hadoop inside open stack versus Hadoop bare metal. Two months, 120 physical servers, large boxes, four racks of gear. The final penalty, just for the folks, take this home, it's 11% slower in terms of actual performance of Hadoop inside open stack compared to bare metal. If you do it right, it's 11% slower, it's about a thousand times faster to manage and operate. You know, one person manage an entire data center worth of Hadoop if you're doing it with these kinds of tools. So if you're thinking about your data as just data, and you're not thinking about data and apps occurring in the same place, you're creating a world of pain for yourself in the future, go into this, same tools and processes for both environments. That's it, I'll take questions afterwards if we have time. Thanks, thanks Josh, appreciate it. One quick refresh, because I think we may have gotten what we wanted to see out of our scale. We have now scaled up Hadoop, and we should go from having seven instances of our Hadoop cluster to 12, right here live. Thank you very much. We hope you'll stay for the next talk on our OpenStack infrastructure by MarkSquared. Oh, MarkSolo. Okay, cool, thank you very much.