 Hi, folks. Thank you. I know it's beer 30 right now for some of you, and I appreciate you being here at 5.20 right before all the parties. So I'm going to try not to keep you past the 6 p.m. cutoff so you can make it over to HP's party. So I'm Doss Kamhout. I work at Intel IT. I'm a Principal Engineer, and I'm responsible for all of our cloud efforts, basically everything from software as a service down to our infrastructure. And what I'm going to be talking about today is a lot of our work on specifically and what we're doing with OpenStack. But I'm going to paint the story of what we're doing and why. So I'm going to go over our cloud journey, our hybrid open cloud, what the goals are, and what we are with reality right now and then an overall summary. I did include here, it'll be on the slides, but we actually have a white paper we just published about three or four weeks ago that goes into a lot of the details about the architecture and the solution that we're using inside of our Intel IT open cloud. So just a little bit about Intel IT right now. We're about 6,000 employees. We cover pretty much everything inside of Intel that includes our design environment, which is about 50,000 grid servers. That was my background where I got started at Intel. It also includes our manufacturing environments and then our traditional IT. We have about 91,000 Intel employees, 67 data centers, about 75,000 servers across the globe and lots of devices, lots of handhelds, and it all comes together into what we do to support our end users. So I wanted to first talk a little bit about a cloud maturity model. So this is something that the Open Data Center Alliance, and we use inside of Intel IT to describe where we're going. I've heard a lot of talks while here at the summit about things about cloud architected apps, enterprise IT, and sense a lot of confusion onto what's actually going on and what an enterprise IT shop does. So my background's in the design grid space where we're pretty used to the concept of destroying servers. It's very similar to what you'd experience in a cloud environment today. But most enterprise IT shops are dealing with what I call legacy app. If you caught the CERN discussion, he compared to pets and cattle, pets being something you have to nurse. But fundamentally, the difference is, and cattle, you shoot. But fundamentally, the difference is that with the legacy applications, they need a resilient infrastructure. They expect that the infrastructure stays up. These are all the concepts that came back from even mainframe days where you assume things would stay up. So if you go inside of enterprise IT shop today and you look at their ERP systems, or if you look at even things like the mail systems and their collaboration systems, all these things are built with the assumption that the server is always there and they have none of these concepts that we like to talk about about cloud architected apps. So reality is these legacy apps exist and they'll probably exist for a while. So when we talk about cloud, it's, you know, and again, this conference is very much infrastructure focused. But the reason we do cloud is actually for our end users. These are people that are trying to get access to their apps and data anytime, anywhere from any device. And a lot of what's going on in the industry today is software as a service. So you're seeing, you know, we had Workday's IPO recently, very interesting company, working against former PeopleSoft guys. But if you look across where we're focusing, it's really on the end user. But how does that happen? We have app developers that need to build applications that support these end users and what they're consuming is hopefully cloud infrastructure. So when we focus at cloud, it's about how do we build a low-level infrastructure that's exposed to software through APIs? So we picked OpenStack to do that and get the software guys the ability to then expose that out to the end users. So you got to see this from a pretty full picture to understand how you're going to operate in a cloud movement area moving forward. Another thing to point out is as you move down this path, SaaS becomes more and more interesting for Enterprise Shop. Today, we run lots and lots of on-premise licensed software, but, you know, this is changing. So today, the problems that we have about how to run these legacy applications isn't going to be here in probably the next five to 10 years, or if we're moving at a very fast pace of technology, which I think a lot of us feel we are, you know, this change to legacy apps to move the SaaS route is going to move pretty, pretty quickly. So what that leads you to is how do you focus on your application space? We break it out into two things. We have the cloud aware and the legacy apps. But as I go through this discussion, I just want to be clear that there are these two distinctions. Our focus and the Open Data Center Alliance focus is how do we drive down this path through the open industry, get things normalized, and into a path where, you know, we don't talk about public or private cloud. There's no hybrid cloud. It's just this big federated interoperable open cloud. It runs across the globe. We have APIs that can talk to each other. We have federated identities. We have a market that allows us to connect things. People won't have conversations about how hard it is to get to data and pull your data back out. And so this is where we're going. And we feel, you know, there's some step functions to get there. Part of it is investing in infrastructure as a service solutions. But we're moving pretty full bore for Intel IT in regards to enabling hybrid applications. And so I'm going to be talking quite a bit about that. From an IT cloud strategic direction, we really have three main points we're looking at. One is what I talked about earlier. How do I make sure that my end users, which are all the Intel employees, can get to their apps and data anytime? And that means I got to supply infrastructure solutions so my software developers and various applications can make that happen. The second point is driving the transformation to a large-scale automated hybrid cloud infrastructure. So another thing about traditional IT or enterprise shops is most of it is run by button clickers. So we have people that utilize GUIs. They're very familiar with reading through manuals and pressing buttons, not necessarily operating at scale through automation. So we're fortunate enough at Intel IT to also have this very large-scale grid environment. So many of us grew up with the concepts of automation and working at scale. Cloud is a great thing because it brings all these concepts to bear to the wider market. So one of our focuses is driving this shift. And the third is one thing we care about too is how do we help other enterprise IT shops and the whole industry move to cloud? This is both the private and the public because if everybody bands together and has a single voice, we believe that the community can move the needle much quicker. So we're very, very big on finding other enterprise IT shops that are willing to help pave the path of where we need to go on cloud. So just a little bit of background. We've actually been running Enterprise Private Cloud. If you look at NIST definitions, it pretty much meets all of these on-demand self-service, measured services, elasticity, resource pooling. So we have a pretty much most of our environment today is virtualized for traditional IT. Just one note, our grid environment, which is about 50,000 servers, there's no virtualization whatsoever. It's purely bare metal, fairly large scale, very high utilization. 80% of all of our new services land in our internal cloud environment today. And it's not that sexy anymore, but it's about one hour to deploy infrastructure. So we went from like 90 days to down to one hour. So it was a big contrast for us. But of course, you can go to a Google Compute Engine or AWS and you can get infrastructure in minutes, right, or in some scenarios, seconds. But this was a big change for traditional IT shops is how do you enable this acceleration? But we realized, you know, this wasn't enough and a lot of our work was internal. So it was technical debt that we were building. We were taking a lot of proprietary locked-in solutions and we were taking our own code, putting it on top and making all this work. And we knew that in the long term we couldn't make this happen. So we made a decision late last year that we felt that OpenStack was ready to start investing in from our technical team's perspective. So where we're going with this is how do we land applications in minutes? So it's less about just a server. An application means many, many servers landing the application stack, getting it all configured, getting the load balancer, making it take load. So it's a much more sophisticated approach. You know, press a button or call an API and get it up and running. We also need a bursting capacity. So I'm a pretty firm believer that most enterprise IT shops never need bursting capacity. They're either going to run a private environment or they're going to be on the public cloud. Most don't have a scenario where they need extra capital environments because if you go look at the utilization in most enterprise IT shops, it's 10%. The ones that are pushing the envelope are maybe 20%. So, you know, you have enough buffer already that bursting doesn't make a whole lot of sense. And then SaaS for non-differentiated apps. So what I mean by non-differentiated, Intel is a design company. If you guys haven't heard of us, you know, we make CPUs, we design them, and we do massive manufacturing, so high volume manufacturing. So we focus, you know, really on these two core aspects. So the question I often ask our IT team is, you know, is email something we actually need to run on-premise or is it at the point right now that it works great as SaaS? You can take that even further, you know, people say ERP or, you know, the systems that run a lot of our supply chain stuff. Is this still differentiated or is this at a point or getting to a point pretty soon where this stuff will run, you know, in the public cloud as software? So, you know, that goes to the point I was making earlier about the maturity model and the move to SaaS. So that was a little bit of the background. I'm going to jump into our IT hybrid open cloud. So first a few key concepts that we took in on this project. So first was we need to abstract our users from the underlying cloud providers. And the key word there is providers. So we need to support multiple cloud providers. So one thing that I heard a lot here is a lot of focus on OpenStack. We like OpenStack a lot, but the reality is there's many, many clouds and we expect there to be. We need to be building solutions that allow us to abstract all the users across these many different environments. So Amazon is very real. Google Compute Engine is very real. I'm sure we'll see more clouds show up. If you do business in China, there's different cloud environments there. You have to be able to, if you operate a global scale, to make it easy for your users to use all these different options even though our choice internally is OpenStack. We need common identity for entitlement across all these interfaces. And we also made a choice internally that we'd go open source first. So rather than building a technical debt, we can minimize or proprietary API lock-in and we can utilize the community to help scale these solutions. One thing interesting that I found is most IT shops, we all have the almost identical problems. Nothing is new. The only difference is where you are on your maturity model and what you're doing in your environment. But I haven't seen any IT shop that actually has a large difference in what they're trying to achieve. So we're pretty happy that we're seeing something like OpenStack working to the community now. And we also have to stay pragmatic as we scale. Most enterprise IT shops have a fairly large investment in solutions that are already running, whether it's their storage environments. We chose in our first OpenStack implementation to go greenfield, but there's a high likelihood that, you know, we will need to be connecting to solutions that we already have in our environment that we already have spent capital on. So it's good to see things like Cinder thinking about how do you have multiple storage solutions on the back end. So we encourage this and like to see this approach to allow us to have many, many solutions behind the APIs. So just on an overview, too, where we're going, I talked a bit about the end-user view. We need those reusable services at every layer of the stack. So first of all, you know, we are Intel, so we use Intel underneath. But it's not all about, you know, one type of cookie-cutter server. We have multiple types of server solutions that we would use. For instance, there's Intel Atom, which, you know, he'll have plenty about ARM, but Intel Atom, we believe, would be a good solution to meet that use case. Intel Xeon, you know, everybody's pretty familiar with, you know, your traditional box. And then we've been recently exploring Xeon Fee, which is a competition to the traditional GPUs. The point is, there's a heterogeneous environment which is what's needed in order to run your infrastructure services above storage, compute, and network, all three. So OpenStack, obviously, is very much focused at this layer. But however, in order to make this actually work for your end-users, you need this entire stack. You need some application platform services. We're investing in pass. We have a pass environment running inside. We use Cloud Foundry and Iron Foundry today. And then, and then as you move up, you need application services that people can use. So you need location, how to make it easy that people can reuse these APIs across the stack. I just want to go over to our approach to release cadence. So we've also heard some other enterprise IT shops say the six-month of OpenStack releases is too much. I feel differently. I think it needs to be at that pace if we're gonna keep up the speed that's necessary. I have a slide later that talks about a lot of the gaps that we see. But in order to keep pace where things need to go, we need to be at six months and we need to move fairly aggressively. We are at a hyper-evolution of technology and it's important to move at a faster clip than you've ever moved before. And then figure out means of how you do continuous rollouts inside of your environment. So we see, you know, physical infrastructure. You see new stuff about 12 to 18 months. Obviously not refreshing anything that fast. You see OpenStack at six months. And then we have this area of manageability and the API on top that we expect to be moving at a three-month cadence. And where I'd like to be towards the end of next year is where we're just releasing code similar to what you see at eBay or Etsy where they're just rolling out code every week and the end users don't even notice. So this next slide is about the gaps. So I don't want to leave the wrong impression. We like OpenStack. We think it's a good solid first foundation. But I wanted to highlight some things in red as areas that need to be focused on. Some of these are just areas that we haven't implemented yet. Some are areas that actually need to be addressed by the community. If you go and look at some of the public cloud providers today, even just that infrastructure is a service, you'll see quite a few of these functions on the left available as APIs, right? So our focus, as I stated earlier, was everything needs to be exposed as software, as APIs. So therefore, you know, even if we have some infrastructure solution today, we need to be able to turn on an API so we can use multiple solutions behind the scenes and give our software developers the ability to have these reusable, backwards-compatible APIs. So like, for instance, you know, we use Puppet, but we don't have an API in front of it. So people have to go into a Puppet to actually build out their domain models. So we need some solutions that allow us to have APIs across the entire stack. You heard Chris Kemp talk about DNS. We need DNS solutions inside of OpenStack. There's nothing there. And you can go to AWS and see Route 53 and other solutions. So my point really here is, as you start building out entire infrastructure as a solutions service stack, you need to, we need to build out more in order to have a full, robust solution. So what do we have right now? So we, as I stated about, in December timeframe, we made a decision to do our cloud environment on OpenStack. So this is a green-filled space. We have a fairly large, existing enterprise private cloud that's mostly built by us. But this is a green-filled space that we decided, hey, we're just gonna go basically pure open source as much as possible. There's still a few things that are not open source, but we're working to make it all open. Just call out a few of the key technologies. We're using Essex now, making plans to roll out Folsom. Nagios for monitoring configuration is using Puppet. And we, of course, have hardware load balancers on the edge and handling our VIPs, connecting back into our internal environment. So this is all an externally-facing space. So we picked our most complex use case in order to prove out what OpenStack could do. And today, we are running in-production, cutting-edge web services on a predominantly open-source cloud. Just one point, too. I don't actually have this in the slides, but as I mentioned earlier, we are running a pass environment on top of here, too. So we basically put Cloud Foundry and Iron Foundry on top of OpenStack, and that runs quite well. So some of the details on our cloud approach. So it wasn't about just landing in one data center. We worked across two of our data centers and an external provider. The reason why is because we needed to run in an active, active, active mode to achieve a 4.9s availability. 4.9s availability is about 52 minutes of downtime a year. So in order to do that, you really can't have humans involved. And also, since if you're familiar with OpenStack, you know, your VMs can die. There's no concepts of live migration. So we really needed something that was cloud-architected and can work at scale across many locations. So we have all three locations working together. You can take load in any location at any time. We're using MySQL replication solutions. We're treating it all as one data center, though. So if you think about what I said earlier about how do we work across all these different environments, if I choose next year I want to add another external provider, I want to be able to turn that on fairly easily. But for my software developers, I want them to notice no difference, right? They should just be getting their compute, getting their storage, getting what they need to run their software, but not knowing necessarily that I've actually turned on yet another cloud provider. So I talked about the 4.9's availability. One of the things that we did that we designed in is the ability to do self-remediation. So we built this concept. We tried to keep it really simple. Obviously, you know, having those various solutions from the network fabric up to the application layer. But we used just simple concepts of watcher, decider, and actor. So our watcher that we picked was Nagyosh, but it's the concept of watching that's important. You know, we could replace Nagyosh with something else in the future. It fit the use case well for what we're doing. They had a combination of operating systems and a pretty rich community. So we watch all the various capabilities that are running in the environment. We can feed this into a decider that does analysis correlation. A fairly simple analysis right now, you know, should it turn something off, right? Because what we want to look for is if something's having an issue in the environment, we want to shut it down so that it doesn't cause issues for the end users. And then an actor that initiates action, so where Puppet can take action against the environment because it understands how the entire environment operates together and is able to make, take those actions based on the decision. So a simple flow, but we feel this is necessary to give us four nines and where we're going with our five nines goal the next year. So another thing that we had to focus on was a cloud-aware application. So most of our developers were familiar with traditional development practices. And so we basically said, hey, you know, this is actually quite a bit of different space. Most of these developers weren't familiar with grid computing. They thought, you know, everything should be stateful. If your server dies, you know, that's a problem in the infrastructure. So one thing we do is we go out and we try to help them understand what are the changes that you have to take in order to develop for the cloud environment. So you have to shift the stateless. And I do call out, by the way, there's a white paper up here, too. Our friends at Disney put this together. The guys run Disney.com. They're part of the Open Data Center Alliance, too. And they publish this. There's seven rules. So if your developers are not doing cloud-aware yet, I encourage you to take a look at that. It's pretty detailed and covers some of the principles we go after. But some of those, if you're not familiar with it, you know, shift the stateless. So assume failure pretty much at all layers. You should scale horizontally. Most enterprise applications today only understand scaling up. And if you scale up, you always hit a ceiling, right? So you want to scale horizontal to meet the needs of your demand signal. Eventual consistency at the data layer. So, again, a lot of enterprise apps assume that you have to have constant consistency at the data layer. But that's not necessarily true. And there's ways to get around this. You can't obviously probably do this in the banking industry today, but or if you have a situation like a hotel reservation, if you had two people come in and trying to get the exact same room, you know, your hotel could be oversold pretty quickly. But for many, many use cases, eventual consistency works great. We push heavily to shift to the DevOps model or the NoOps model. So DevOps was fairly familiar to a number of us on our team. But, you know, the focus we want is never wait on IT, right? If everything's exposed as a software service, your software developers should be able to build solutions and do this at scale without ever having to even talk to an IT person. And what we really want is the IT infrastructure team to become invisible, right? You shouldn't have to talk to them or call them. They keep the infrastructure running. They keep the API exposed. They meet the SLAs and quality of service requirements. But really, they become invisible, which is similar to, you know, the NoOps model. And implement true web services for consumption. So this means things like make sure that your web services have quality of service, they can throttle, and they themselves scale and utilize these concepts. So that was on cloud architecture, cloud architected apps. The other area that we deal with is where I talk about the legacy and traditional enterprise IT-type applications. So these are a few of the areas that we need to close for enterprise. It's been encouraging to hear at the summit a lot of progress being made in some key areas, but these are still some of the ones that we see as issues. One is in traditional IT, you got to keep that server up. And if your host dies, you know, you don't want to, if you're running 40 VMs on a host, running it pretty hot, you know, you just lost 40 VMs. And if somebody's running an email server on there, you know, that's not going to go over very well for your end users because probably that email application wasn't built for the concept of a server dying, right? So you got to solve this problem, or we have to solve this problem for traditional legacy apps because they will be here for a while. Even if they go to the SaaS model, I guarantee, you know, most IT shops will have these. So there's some, you know, three major points here. Shared block storage, so you need to have a shared block storage solution which lets you do things like live migration. So just a simple thing, like a maintenance of a host, or even if you want to upgrade your host, you know, most enterprise solutions today that do virtualization allow this, of course, because it's a use case that's pretty important to IT shops. Restart of instance when a host fails. So if your host is going to fail, I'd like to make sure those 40 VMs start right back up on another host. You know, it's not insurmountable and these are pieces that should be coming into the environment here. And then the next part is enabling a federated hybrid cloud environment. So talked a bit about how, you know, we like OpenStack, but it's not the only cloud solution we're going to use. There are public cloud solutions. It's good to see more public cloud providers exposing OpenStack APIs, but the reality is there's a pretty rich ecosystem that's not OpenStack. So how do you make sure you connect to them? We want to end users be able to have seamless access across the zones, the regions across clouds. Today, even if you have two inches of OpenStack inside of your environment, there's not necessarily something built today that allows you to interact with both of them, you know, in one call. We need identity federated across these. So if somebody has an identity, just like, you know, we as end users today we're pretty used to a single sign-on or you may even use your login with Facebook button because you're a consumer, but the same type of approach for a lot of software developers, they want an easy way to be able to authenticate and get their identity federated across multiple environments. And then orchestration. So since I'm dealing with many types of cloud environments, I need a way for the software developer to be able to consume xCloud, consume the OpenStack cloud all in a seamless fashion. We don't want them to have to learn the innards of every type of cloud environment. We want a single API facade that is exposed to them so they can deal with compute in one way, you know, they can deal with object storage in one way, they can deal with networking in one way, but really give it to the software developer so that they can function seamlessly across multiple clouds. And we also need high availability of infrastructure services. So the cloud should be built as a cloud, right? You should make sure that the components that are inside your cloud, if they're failing, you know, they should be stateless as much as possible so they can run on other systems. You should use concepts like eventual consistency. But there's core features of OpenStack that we're not using because, you know, they have single points of failure. A big one that's, I didn't hear a lot of people talk about, but in order for this to work, and a lot of us that have even just SOX compliance, you have to go and deal with the security and the auditability of the environment. So we need role-based access for people, right? Sometimes, you know, the person that runs the infrastructure should never have access to the application. So there's certain segmentation that's required as well as you need to be able to audit who's really getting at there. If you've ever worked with the regulation groups out there, you know, they're very, very strict. They even hate things like multi-tenancy. So I think it's important to understand that companies that are regulated have to go through quite a bit of rigor to make sure that their infrastructure and their applications are meeting the conditions of the regulator. So it's not a simple thing. It's a big problem to solve, and we encourage to see that as we move forward, the community helps make this happen. This isn't a common problem for startups. I thought it was interesting. In one of the keynotes, you know, 80% of startups are using public cloud today, but I guarantee most of those startups don't have the same type of regulatory requirements that a lot of the big enterprises have, nor do they have some of the issues in regards to liability a lot of the large enterprise shops have. So it's important to get some focus onto these tough challenges that we have. We have published a much larger list. You know, it didn't fit on a slide, so I just made sure it is public. These slides will be up on SlideShare later, but you can find it out on the Internet. And we want to have this in open dialogue. So we would like to get some other enterprise IT shops that are interested in making this happen. We've set out a goal we believe within nine to 12 months. Opposed grizzly, a lot of these functions that we need are actually, it looks like they're going to be there. And what this gives us is the ability to start taking traditional IT workloads onto our open cloud environment. And if anybody has solved these already, we also want to hear from you and want to talk to you and figure out how we can partner together to make this happen for the whole industry. So just on a wrap-up. So our direction, as I stated earlier, it's a federated interoperable and open cloud. What we mean by that is cloud services should work seamlessly regardless of what cloud vendor there is. If you listen to some of the big cloud players today, they're not really that interested in interoperable solutions because in reality, the market's actually young, right? So a lot of stuff is happening, but getting Google and Microsoft to sit down and talk about an API, it's not going to happen very fast. And if you try to use a standards body today, like standards bodies, but they don't move fast enough, right? Our technology is moving at such a fast pace that we need to find a way to have the community to insist that we need this open environment. And there's lots of cool things going on. We have OpenStack. We have the OpenCompute project. If you're not familiar with that, I'd take a look at it. And we have the Open Data Center Alliance. And even beyond that, we have people making brick manufacturing solutions that are open source. So point being is OpenStack is happening everywhere. And we need this in order to push the industry forward where we all need it to go. We've had some pretty strong success with our enterprise private cloud, as I showed earlier, pretty good measures on what we're doing. But it wasn't enough and increased a large amount of technical debt for us. And we felt the community was really pushing the envelope in regards to what's possible with infrastructure as a service. We are in production now. We're running production workloads. But these are for cloud-architected apps or cloud-aware applications. And our target next is for enterprise IT applications, one that expects resilience. We think there's lots of space for people to contribute. So our personal Intel IT team, we're working through our legal process right now. We intend to contribute. We have some interesting areas of code that we would like to put into the community, either directly into OpenStack or around. We'd like to see other enterprise IT shops take the same approach. We've seen some of them be very successful in getting started here and we'd like to encourage this. And as I said earlier, seeking other large-scale enterprise shops to help pave this path, there's a lot to do and there's a lot that we can make happen. And again, everybody's problem is almost identical. So it's not like everybody has to go reinvent the wheel. But I do want to point out, too, that as I talked about SaaS, I think enterprise IT is going to change massively in the next three to five years. Just like most startups are, you know, 80% on public cloud today, I think if, as those startups grow, will they really go private? You know, it's a good question. We've seen companies like Xinga that have, you know, gone public and then pulled all their infrastructure in with their Z-Cloud. But in many situations, what we do in enterprise IT shops, a lot of the mail systems, the collaboration, a lot of this should be running as software as a service at scale. So large change happening next two to five years. A couple resources for you. I think I talked about Open Data Center line quite a bit. It is a group of about 300 global IT teams that are working together to set the path of cloud for enterprise IT shops. Obviously cloud exists, but enterprise IT has certain requirements that we need for the environment to happen. And Intel IT, we're pretty happy to share what we're doing. We like to have two-way dialogue to figure out, you know, how do we move the industry forward? So we have lots of information about, you know, pretty much everything we do. We're pretty open about sharing that. So I know there's an HP party that starts in 10 minutes and you guys have been great. And you probably want some beer and some drinks. But let's see if there's any questions from the room in the end of the day. And if you can come up to the mic where we have some video, if you can use the mic. Can you talk a little bit about how you have implemented multi-tenancy within your private cloud? Yeah, so basically each tenant... Yeah, how have you isolated the tenants together? I've used VLANs, Firewalling. So we're basically using security groups. So each one gets their own VLAN environment and then we carve it up with IP tables to do network segmentation across them. We're using Nova Networks today and our intent is to switch to Quantum with Grizzly. Okay. So following the previous question, once you're associated, you know, floating IP to a different content, they are interconnected. Is there any concern for that? That's one question. Secondly, you talked about your migrating Greenfield application to this cloud infrastructure. And can you elaborate some examples? Yeah, so for the first question, I'll let you talk to the networking guy right over there. He'll connect with you. But for the second one on the Greenfield apps, these are basically Internet web apps. They're exposing APIs. So fundamentally, they're going to have a presentation layer, an app logic layer, some caching in between and a database system running in in triplicate, active, active, active. So the database system, eventual consistency, but basically a workload can hit any of these environments spread out between a global load balancer, take load and respond back to an end user. So these are the first types of apps that we're putting there which are very standard web 2.0 type applications. And our focus next is how to take more traditional Enterprise IT workloads and have them run there. Any additional questions? Yeah, you mentioned there were some components of OpenStack that you weren't using because of single points of failure. Could you elaborate on which ones they are? A Nova volume. That's really, I think, the only one. In your slide about the watcher and the actor, are you automating any of that or is that manual process today? So we have automation working in dev space, but right now that just basically goes to human. So the self-remediation framework is set up to allow it to be automated, but we're making sure there's false positives and false negatives don't get triggered and shut down a data center. Is that a home-grown solution that you open source? The deciders, but all the concept is pretty straightforward and we're using mostly off-the-shelf open-source stuff. So it's just the concept of knowing what to watch, knowing if something goes down, what to do about it. So it's all things that we do as humans naturally. So our decision was how to just automate that. So if you tie it all together, it's automation solutions that exist today and we're free to share all the information of how we do that, too. Any additional questions? Okay, thanks, folks, and take care.