 I'd like to welcome you all to our open-stack architecture for the Enterprise. We've got quite a lot to get through today, so we're going to go through it as quickly as we can. So, I'm Keith Ovan. I'm part of Dell Cloud Services. I'm based in Ireland in the Dublin Centre of Excellence, and this is Greg Jacobs. Also in the Dell Centre of Excellence located in the United States. Okay, so... Pardon me? Do you want to go under? So, part of where we're coming from, I suppose, in this talk is to look at open-stack in the context of Enterprise, as opposed to the context of provider scale. So, we had a number of design goals when we said about doing this, and part of our design goals was to really focus on the requirements that enterprises have, as opposed to the requirements that a provider has. And in some parts they're the same, and in some parts they're different. So, we said about having a number of design goals in this architecture. And the design goals, like we looked at, was zero downtime across the control layer. So, the most important part for us, or one of the important parts, was to have that zero downtime. No data loss. So, again, the idea being that in this model that if there's a failure, that we actually have no data loss within the control layer. Durability. The model that's fairly well accepted within the provider model is to provide for forward-leaning applications. But in enterprises, not all applications are going to be forward-leaning. We're going to have some applications that are backward-leaning, some legacy applications that are basically enterprise-grade applications. And these applications have their resilience expected within the actual stack itself, and not within the software. And we need to be able to provide for this in the enterprise context. So, the other part as well is around minimizing the management tasks that's based around running an enterprise open stack. So, we started to look at this, with these design goals in mind, we started to look at this as sort of, you know, what we were trying to achieve. And what we've done was we said about designing an architecture around this. So, this architecture we deployed today for our customers in Dell. And over the next couple of minutes, we'll take a look at some of the elements that go into actually realizing this architecture. So, when we started to look at this, we suddenly began to realize that the architecture for enterprise is more than just the element of open stack. So, we have storage. So, we looked at the stack inside Dell and what we had to offer. And we started to look at, from the top down, we started to look at, you know, multi-cloud manager, the ability to sort of manage a cloud in a way that allowed you to just have more than just open stack in a private setting, but also you might want to have, say, some of your cloud resources in a public setting. We started to look at open stack itself, which we'll take you through in a minute. And we started to look at storage and what we had to offer on storage. And we looked at the sort of two areas of what Dell had to offer. Part of it being equal logic, which is, you know, your traditional storage. And we have a set of drivers for Cinder that supports this. We started to look at, you know, some customers like, you know, want to use open source software. So, we've chosen to go down the route of SEP. We started to look at deployment. And one of the parts for us was that deployment is something that should be easy. I mean, it shouldn't take a long time to spin up a cloud. It shouldn't be complicated. Crowbar. Crowbar is a project that we looked at. And we said, you know, inside Dell, it was open sourced. And we said, this is a very good platform on which we base our development. We'll take you through all this in more detail as we go through. We looked at the networking. We said, you know, best in breed networking, force 10. And we looked at server range. We chose to go down with R620, R720, and C6100. I mean, these are all cloud, you know, grade servers with features specifically for cloud. Security. Security is one of those things that was absolutely a must for an enterprise. So we started, we talked, you know, we worked with our security team to build, and at a very early stage, the architecture that shows that it was secure. And then wrapping all of this up is the ability to have this managed if you wanted it managed. So Dell Cloud, that's what Dell Cloud Services does. You know, if you don't want to manage it, they will manage it on-premise or off-premise. You get this one. So Crowbar. Yes. So deployment. In our space, we have the Crowbar tool, which is a great tool to deploy the... Crowbar. Okay. And we looked at it from the deployment model. Crowbar offers an excellent deployment model. It's one of the few products that actually manages the complete stack from the actual hardware provisioning up to the software layer. Okay. So one of the things we have in Crowbar is sort of unique is the ability to provision hardware. Okay. This provisioning of hardware is something that starts with the idea that, you know, when you kick off a run of Crowbar, what you actually get is it will go out, it will seek all the nodes that are available. And once it finds those nodes, it basically starts to interrogate those nodes and starts to figure out if the BIOS is current. And if the BIOS isn't current, what it will do is it will start to upgrade the BIOS. It will start to look at the individual firmware on the individual cards on the server. And it will start to upgrade the firmware on those cards. It will start to do the configuration of the BIOS. And it starts to do the configuration of the network cards. And all of the PCI cards within the actual server itself. So it actually brings your hardware infrastructure up to a stage where it is at a known stage. Okay. And your current. It then starts to lay down using what they call a set of bar clamps which are set to modules within OpenStack. They basically define how the actual deployment is going to look from an OpenStack point of view. This is how which nodes are going to be control nodes and which nodes are going to be compute nodes. And it starts to lay down the software based on the bar clamps within those nodes itself. You know, it's within a very short space of time, you know, typically we're looking at a rack of servers and you can provision up a rack of servers in a very, very short space of time. It's got a web UI. So this is web UI driven. So it allows you to become very familiar very quickly with the tasks that it can carry out. The bar clamps themselves are something that you can create yourself. They are basically internally their chef recipes with a wrapper of a bar clamp around it which basically augment the chef recipe. So if you've got proprietary requirements you can actually create bar clamps for to add this functionality inside your infrastructure. So if we look at sort of just a small, on the left-hand side, you know, you've typically got your bar clamps which would be sort of your database bar clamp, your identity bar clamp, your image bar clamp. Your block storage, networking, Nova Horizon. These are the elements that go into OpenStack. OpenStack basically does a pixie boot on all your servers. It lays down what they call a sledge hammer image which does all the interrogation and all the actual firmware upgrade and the configuration of your firmware. It's quite a considerable amount of time goes into it. When you think about it, a considerable amount of time goes into the provisioning of the hardware itself and doing all of that upgrade of the firmware. And one of the nice parts about it is once your infrastructure is in place, the ability to upgrade afterwards and come along and lay down new firmware on the system is pretty easy. So one of the elements that's built into Crowbar is that it uses Chef. Chef is familiar to many people, so it's pretty easy to create those bar clamps and extend it. And on the right-hand side, you basically get an idea that it creates your cloud for you. So we had Crowbar 1.x. And as the project has matured over the past 18 months, we've seen a huge amount of change within the industry. We started to refactor our thinking around Crowbar and we started to move towards Crowbar 2. Crowbar 2 is an open-source project, so it's currently under development. And we're very interested in people that want to get involved in the project. We're sort of moving towards the timeframe for Crowbar 2 as we move towards the middle of next year. So it's going to bring a lot of new functionality. So within the infrastructure, when we looked at this, one of the areas that we saw as a sort of complication was around load balancing. Many of the models that exist today sort of work on the basis that you'll have an active passive arrangement. And there's nothing wrong with this. I mean, this works very well and it's tried and proven. You've got something like Keep Alive D and you've got your load balancer and you've got your VIP. The incoming requests basically come in through the VIP and hit the load balancer and are spread out to Keystone or any of the API endpoints. And this model works pretty good and in the failover situation, what happens is the VIP through Keep Alive D it will actually move over to the other node in a failure situation and the other node basically picks up where the original node controller 0 picks up its work. I mean, does anyone see anything wrong with this model? Okay, here's one of the areas that I see and we saw as being a difficulty. It doesn't scale. You have one server that you spend a substantial amount of money keeping there running and it's not really doing a whole lot. That's one area. The other area is bottlenecking. I mean, it doesn't scale. What you're doing is you're pulling all of the traffic. I mean, you're talking about traffic around Keystone. You're talking about all the traffic around Neutron. You're talking about the API traffic around Neutron. You're not the individual compute node traffic. You're talking about the Cinder API traffic, the Keystone API traffic, the Nova API traffic. You're talking about the scheduler because the scheduler goes through RabbitMQ and RabbitMQ is low balance behind it. So you've been formed a picture. You've got Horizon and we've got Glance. All of these services basically are bottlenecked through that one point. Your network cards, or cards if you've got them bonded, are going to saturate. You're getting enough network traffic through there. That's what you're going to get. So we started to look at this and we said, we need to load balance the load balancer. Sounds a bit crazy. But we do. And that's what we decided to do. So within networking, we've come up with the concept. We've seen others do it. It's not unique. We've taken our network switches, our force 10s, and basically the routing within the actual network switches, we're using OSPF and ECMP. And we're basically spreading the traffic amongst the load balancers. Basically what's going on is inside the networking switch itself, you've got a router. And inside each of the individual controller nodes, in this case we have three controller nodes, we have got Quagga, which is a router. The end points are all serviced internally within the node. It means that you can't really get the end points directly. You have to come through Quagga. So what's going to happen is when you make an API request, it's going to come into one of the switches, where it's going to find its way, the packet is going to be routed based on the criteria we lay down within the router itself. And it's going to land on one of the Quaggas to be routed onwards into the API. Because the API is held internally. And once they hit the API, they're hitting the HA proxy, and the HA proxy just does its normal job of distributing the load amongst the Keystone. This arrangement works very well because it's load balancing the load balancer. And if you get a situation where you lose one of the controller nodes, the OSPF is going to recognize that that route isn't there anymore. So the packets are not going to be sent down to try to send those packets down to that particular controller node. So it's a very resilient model. It's a very resilient model. So one of the outside of the load balancer, we started to look at Neutron. And with Grizzly, we found that the advantage that had been brought in Grizzly was the ability to distribute the L2 and L3 and DCHP agents amongst nodes. This is the model that a lot of people will see. You're running your networking through one point. This is the bottlenecking problem coming back again. But this is based around the tenant because all the tenant networks will be hosted inside the actual network node. That network node could be on a controller node. So you still end up with the same problem. No matter where you host the actual Neutron API and the Neutron services, you're still going to end up with bottlenecking all of that traffic into one area. We took advantage of what Grizzly had to offer and we'd gone with a different model. And we chose to distribute our load amongst all of the computers. So what we do is we install the L3, the DCHP and the OSP-FE agents onto the compute nodes. And we place, on the controller nodes, we place Neutron, the Neutron API, which has got the scheduler built into it. And what happens is that when you get a provisioning request for a new network, the scheduler is going to basically look to the compute nodes for which one is available to take that tenant network. By spreading out the tenant networks amongst all of the compute nodes, we're spreading the load amongst a wider, we're spreading the tenant load amongst a wider server base. This gives us a big advantage in the case of failure, where we've got to spread over a wider area. So we're less likely to take down as many tenant networks. If we do take down a tenant network, we can just reprovision. So in this situation, we can reprovision the tenant network very, very quickly and get the tenant network back up and going again. So RabbitMQ, the model that we see used in a singular sort of server situation, just wasn't going to work. So we took advantage of what was there, and we're using a cluster model with Mericuse. This works very well for us. We've placed it behind the load balancer, effectively, again, across our controller nodes. And in a controller node situation, I show three controller nodes there, but this can be up to end times the control load. It scales pretty linearly. So we end up with a situation, basically, being a clustered RabbitMQ, and it services all the requests coming into the load balancer. Because it's the dynamic load balancer, again, we've got that added advantage of it being load balancing the load balancer. So the heart of most open stacks is going to be the database. The database isn't stable. We're into problems. So this is a little bit boring, but I want to take you through this for a reason. This sort of just shows a single model where you've got a single database, and we all know that that's going to fail. If it fails, that's it, you're dusted. It's your open stack is down. Single point of failure, that's the biggest disadvantage. We started to look at the active passive model that would traditionally be used. It's a good model. The data is asynchronously replicated, and in the event of a failure, the VIP moves over like we saw earlier on, and the load is taken up by the other server. I mean, this works. But it's got asynchronous replication. The question is, does anyone see any problems with asynchronous replication? I'll leave that for a minute, and I'll come back to it. You've got that. You've also got a scale limit. This model only scales so far. You've also got the fact that you've got one server there sitting, most of the time doing very little, only waiting for the other one to fail. It's a waste of resources. So we looked at DRDB. This model works pretty good. The data is replicated. Again, you have this problem in that the server is basically one server sitting there. I won't say doing very little, but it's accepting the data replication from the other server. But again, it's doing very little. It's a waste of resources. And in the event of failure, it fails over pretty much the same as the active pass model. And again, it limits its scale. And we looked at multi-master replication. So this would be your typical MySQL approach with nothing unique here. In this model, we have two applications and a load balancer, two streams of queries coming into the server cluster and being serviced and the replication going on between the servers. And this is your typical scenario. In the event of a failure, the traffic is routed to one of the other servers. The server that died is okay and everything just walks away as normal. But there's a problem. There's no guarantee that at any point the data is consistent across all nodes. Because of asynchronous replication, the best you can hope for is that the data got to the other node before it failed. If it didn't, you're into a scenario where that server is now has different data than the rest of the cluster. That means when you go to bring it back online, you have to get manually involved in it. You have to somehow try to figure out what is in this table versus the current tables. And you're into problems. One of the other things that we looked at was single-tread replication. So the typical out-of-the-box MySQL uses a single thread for replication. This increases your probability of actually having inconsistency within your cluster in a failure situation. Because you've only got one thread. And in this slide, you can see, I've marked that it's using one processor core to do the replication that is equal to one thread. And this means that you've all the server available to you. You're only replicating using one thread. So we spoke about the disadvantages. They should be starting to become obvious. So we looked at what could we do. We decided to use the Precona model. Precona has got synchronous replication. That means within the cluster, at any point in time, the data is always consistent. Let's speak a little bit about that in a minute. Let's go back to one slide. So it's also a multi-master model, much like the multi-master model we spoke about earlier. It's got automatic node provisioning, which I'll speak about in a minute. And it does a parallel apply on all nodes. So synchronous replication. This means that when a query arrives on a controller node, and there can be multiple control nodes, I've just simplified it here, it is wrote to all other nodes within the cluster, and all other nodes accept, and only when it's accepted is it deemed a valid query. It's wrote to all nodes. Like I said, all the data is consistent across all nodes at all times, meaning that you can get a failure at any point in time, and you can be guaranteed that all nodes are consistent. It's very, very easy for bringing the failed node back online, because you know the data is consistent. You've got multi-treaded replication, reducing the problem that you saw earlier on. You've got multiple cores on your server. The replication is taking advantage of all the cores. So replication doesn't become a bottleneck. Automatic provisioning. You want to add a new node into the cluster? You don't have to worry about getting the data from one node to the other, or if any of those complications or problems, it joins automatically. There's no difficulties involved. So you can scale this out pretty easily enough. And as we spoke about, it's a multi-master model. So it gives us a stable platform on which to build our OpenStack infrastructure on. We looked at block storage, and one of the things we started to sort of look at was enterprises, they've got both that forward leaning, and maybe as we move towards the future, that forward leaning application, those enterprise applications, and they've also got legacy applications. So it's not a very clean world. It's a lot cleaner world in the world of public cloud where people are tendency to move towards the more forward leaning application, where the resilience is built into the software. Enterprises don't have that luxury. They have applications, they've been sitting there for 10 years and they're not going to be upgraded. But they want to move them to a private cloud to take the advantages of it. One of the ways we have to deal with that is by providing storage, that's reliable. We use Equalogic. Equalogic has a driver for OpenStack. It's a two-head node configuration. It means we've got a certain amount of resilience. For people that really want to go down the OpenStack, the open source route, we have SEP, and we put that on our R720s, which is a very good platform for a performance point of view where we use SSDs. So I suppose to wrap up and take a very just quick look at what we do for the rest of the services. The rest of the services are all based around sitting those APIs behind the dynamic load balancer. It's about sitting the glance behind the dynamic load balancer. It's sitting the Novi API around behind the load balancer and Keystone and all of the other services. And again, we sit the MySQL behind the load balancer as well. So this gives you the model and it's a fairly resilient model. Within the lab environment, particularly where we do a lot of the fall testing, it's proven to be an extremely solid base for an enterprise-grade cloud. And I suppose that's the part that we're trying to differentiate is an enterprise-grade cloud. So hopefully we got to that pretty quickly. If anyone wants to dig deeper into the elements that we spoke about today, you can join us at the Dell Boot. We'll be there for at least 40 minutes. You're all quite welcome. Thank you for joining us.