 Hello. First of all, I want to thank everyone for actually showing up at 5.30 on a very long day. This is actually my third session today. And I have three more this week, so if I get a little punchy at the end of the week, you'll know why. So welcome to this session on bridging the gap, open stack and VMware administrators. We're going to try to do things a little differently. We'll talk about that. We're taking a different approach than we've done in some of our previous talks, including the one we did last year in Hong Kong. So a couple of quick introductions. So my name is Kenneth Hoy. I'm a technology evangelist over at Rackspace, focused specifically on open stack, but also on hybrid cloud, including VMware. I'm also a V expert for the first time this year. Thank you, Ken. My name is Scott Lowe. I work for VMware. I'm an engineering architect in the networking and security business unit. As you can tell by the shirt, that means I work on NSX and Twitter ID there in case any of you guys follow me on Twitter. Long time in the VMware space and now spending a lot of time focusing on open stack and other open source initiatives as my primary focus within VMware. I'm going to go over the agenda and kind of kick things off, and then Ken and I will be trading back and forth as we go through the deck. If you do have questions, you're more than welcome to ask those questions anywhere along the way. We do have a Q&A section at the end, so if you want to wait until the end, that's fine. If you want to ask questions in the middle, that's fine too. The only request that we have is that for the purposes of the recording and for those who are going to watch afterwards, let's try and use the mic in the middle. So if you have a question, just jump up, run over to the mic. You're not going to offend me or Ken. We're pretty easy going, so there you go. All right, our agenda. First, we want to take a few minutes. We're not going to take a great deal of time. We're going to talk the phrase here, a tale of two workloads. Talk about why or how and why you might be using VMware with OpenStack and some of the advantages and reasons and business justifications for that. From there, we'll move into a fictitious use case that we have created that we hope you'll find both amusing and also useful. We'll follow that up by looking at a proposed solution of an architecture that actually leverages OpenStack plus a number of VMware products together to build a solution to match the fictitious use case. We'd like to use the use case as the framework around which we build our discussion on helping people who are not familiar with the OpenStack environment and terminology and operational model but who are familiar with VMware's traditional virtualization model help bring them into that environment. We're going to use the use case as our framework around which to structure our discussion. As I mentioned earlier, we'll have a Q&A section at the end, but you are welcome to ask questions anywhere along the way. I'll start by talking a little bit about what I'd like to call a tale of two workloads. One of the things in any kind of systems engineering, truisms is that I did a workload dictates architecture, not the other way around. If you have the wrong architecture, your workload will suffer as a result. I want to talk a little bit about what those two workloads are. One is traditional design workloads that you're used to, particularly if you're a VMware administrator or architect. These types of workloads tend to be the state-based kind of monolithic architectures. I think maybe an Oracle server that sits on one server and one server only. If that server runs out of resources, you either need to add more memory, add more disk drives, more resources to that one server, or you have to spin up a second server but do a completely separate instance of the Oracle database. Your scalability for that kind of infrastructure, it's basically scaling up, not scaling out. Again, if you're a VMware administrator, you're very used to this idea that the tools you have are very operator-focused. For example, how many of you in a VMware administrator would give your users, your developers, access to the FeeSphere web client to set up your own VMs? Probably no one that I've ever heard of. So it's not designed for end-users, it's designed for an operator and administrator to use. And because of the nature of that type of application, that it's stateful, that it's monolithic, it requires a very resilient infrastructure. Again, think about an Oracle or an exchange server. One of the reasons you want to put it on a VMware environment is because you want to leverage something like VCA or HA so that when the underlying physical server fails, the application can depend on the VM restarting somewhere else. And as a result, you'll get, sorry, you also use a lot of traditionally very specialized type of hardware, like the stuff you get from EMC or VCE or NetApp, like a VBlock or FlexBot, because you need that rock-solid infrastructure in order to maintain that type of application reliability. And then that second type of workload, the one, so that traditional workload now is probably 90% of the enterprise, but there is that growing 10% that you're starting to see pop up that tend to be types like mobile apps, web applications, kind of stuff that is, used to be the provenance of companies like Twitter and Facebook, but now going into the enterprise. So, example, I talked to a number of banks that are talking about creating applications on the web to do all their banking for their customers and to sell them new products. So those are a different type of application, and the design principle for them are very different. So instead of a one-server type of database, for example, a lot of times you'll have a single database that's spread out across many, many machines. And in that environment, everything's distributed across so that when there's a failure somewhere else, the assumption is that application will continue running, even though nothing's being automatically restarted somewhere else. And also, these types of applications tend to be very developer-focused. So you'll hear a lot in the cloud world about this idea of APIs. So what's the big deal about APIs? The idea behind APIs is everything that's in the infrastructure, every service that's available, it's exposed to the APIs to be activated by a user. So a developer doesn't need to go through an admin anymore, right? Again, think about a public cloud environment where a user doesn't fill out a spreadsheet or an email asking for 10 servers, right? They basically go and spin up 100 servers by themselves, never needing to talk to a network admin or a storage admin. So those are kind of the next-generation applications that even enterprises are starting to develop today. And the design assumption behind those interesting left in that cloud-native environment is that the infrastructure is not at all resilient. The assumption is, in fact, that the infrastructure is going to fail at some point, and that failure is okay. And so what we want to do, and the reason that is the case is because when we think about cloud, we're really architecting for rapid scale, right? The ability, again, for example, in an environment where you may start with 10 servers, and perhaps in a matter of minutes, you need to spin up to 200 servers with applications running them. In that scenario, what you don't want is an infrastructure that holds you back from doing that. You want to be able to move as quickly as possible. And Jonathan Breitz talked about that at the keynote this morning, right? This idea that OpenStacks enables you to move fast, fast, and faster. But one of the things that's also true about rapid scale, right, very large environments growing very quickly is that failure is inevitable. Again, if you are a VMware administrator, think about having to host thousands and thousands of servers that spin up and down very quickly. At some point, something's not going to work. The software's going to fail. Servers will fail. The storage that you're relying on to help provide H8 will fail. So at that point, if you tie yourself to a traditional, resilient infrastructure, there is a possibility that you will hit scalability limits much quickly than you would otherwise. And so the key to designing for a cloud environment is you should assume failure. You assume things will fail everywhere at any time, and you design around it. So your app, instead of relying on infrastructure to be 100% bulletproof, you design what I would actually call availability in depth. In other words, you want resiliency, yes, at the infrastructure layer at some point, but you also, mostly, you want resiliency at the application layer so that if anything fails, you can still, you still can rely on the application handling and restarting. So a couple of principles around that, guidelines then is, again, application handle zone resiliency. The key to having a rapid scale type of cloud is that you need everything to be loosely coupled. So an example is you don't want an environment where most or all your servers are connected to the same shared storage. Because that shared storage actually becomes a bottleneck for growth. You also don't want an architecture where there's a management server that all of you request has to go through. Because, again, that management server or a set of management servers becomes your bottleneck. What you want is a loosely coupled message base where when you generate requests of resources, those resources get spun up in parallel. So in that way, you can spin up thousands and thousands of instances and volumes and know that you won't hit a bottleneck in terms of scaling up quickly. The other thing you want to do is to do scale up in that environment. Because, again, it's much easier. You can scale up faster by spinning up 100 servers and running the instances application across them than trying to upgrade one server at a time. So a lot of small servers is better than one large server. And that kind of gives birth to an analogy that some of you have heard of, right? Probably the idea of cats versus paddles. Cattles versus pets. Pets are basically your traditional virtualization environment, right? Where you know those servers really well. You give them your name when there's a problem. You spend hours fixing those servers. In a cloud environment, you assume these servers are completely disposable. So when you have a VL in a cloud instance that's having problems, you don't worry about fixing it. You just shoot it and rebuild and create another instance. Because you can, right? Very quickly. A good example is Netflix. So when Netflix does upgrades, they don't upgrade their 20,000 VMs at the same time. What they do is they create another 20,000 server instances, run the new code on it, try it out for a little while, and then they kill off the old 20,000 and run everything on the new 20,000. They do that iteratively again and again. And the other thing they do is they actually, in their VM, for instance, the average life of a VM is 36 hours. And the idea behind that is if you run a server for a very long time, you start getting issues like memory leaks and other kinds of problems. So if you start over, right? You basically start with clean VMs, clean machines on a regular basis. Again, that's allowed you to handle failure more easily and more quickly. By the way, that cattle versus pets analogy doesn't just apply to VMs and environments as well, it also applies to your host systems, your host hypervisors, where typically you'll have a lot of them and you will spin them up very, very quickly, add them to your cloud computing infrastructure and or remove them, typically all in a very automated fashion. Scott mentioned that the way we want to kind of present this vSphere of OpenStack architecture is through a sample use case. So Scott, you want to talk about that a little bit? Yeah. So I'll first spend just a couple of minutes talking about the fictitious use case that we have come up with and then we'll move from there very quickly into the proposed architecture that we built and we'll use that to structure the rest of our discussion. All right, so our customer here is Acme Corporation and they are in the middle of a business boom because Mr. Wilde Coyote is a huge customer. So business is booming, no pun intended for those of you that recognize the cartoon character. In this particular case, we have a customer who has a vSphere environment. They're running a lot of their applications on vSphere as many customers are and some of those applications would include the kind of traditional stateful applications that are more scale up than scale out. So things like Oracle databases, enterprise applications from Microsoft, like messaging, those kinds of things. Their IT department at Acme has recently tasked with building out an environment for new mobile applications. They have an initiative to build a lot of mobile applications because let's face it, Mr. Coyote is often out in the middle of the desert and doesn't have time to go back to his office and use his regular applications so he needs to be on the go. So they used to do that with AWS. They were prototyping applications out there but they want to bring that in-house now. They feel like there's some value in having that prototyping environment in-house for them and so they want to bring it in-house but they want to maintain the AWS-like experience which is probably something all of you have been asked at some point. And again, as they mentioned, their plan, since most of their applications or a significant number of their applications are going to move into these mobile-type architectures is to leverage some cloud-native application architecture, some of the things that Ken mentioned in terms of relying, instead of relying more on scale up but instead going to loosely couple systems that are built around a scale-out model. So with that use case in mind, let's go ahead and take a look at the proposed solution that Ken and I came up with for this use case. Okay. So quickly, how many of you, before you came into this room, saw this picture, knew that you can run VMware technologies underneath of OpenStack. Excellent. That's good news. That's actually good news. So obviously there was a perception for a long time that OpenStack and VMware were at odds with each other and could not work together. Not only can they work together, but actually VMware has contributed enough code into the OpenStack trunk that you can actually run Sphere as a hypervisor along with other hypervisors like KVM and Zen underneath of OpenStack. So in the proposed architecture that we would want to give to ACMI, basically we want to have a single management plane, which would be the OpenStack layer, managing two different work, two different art infrastructures, one for their traditional applications and one for their next generation web applications. So you look here, there is on your left, right, there is the KVM layer where we're going to run your next gen web applications and those are actually separated from a set of compute nodes that are going to control the Sphere layer. And Scott's going to get into a little more detail about how that works on the VSphere side. The key thing here to note is that the way OpenStack works is you actually need to have different compute nodes running for different hypervisors. So let's say Hyper-V was also a hypervisor that you wanted to run in the OpenStack. You would need a set of dedicated compute nodes for that. In this case, we have a set of dedicated compute nodes that are running KVM as the hypervisor. And then there is a compute node that is talking to a vCenter EXSI cluster in order to... And Scott will wonder why that's important, why it is that VMware has set it up so that you would want to talk to vCenter as opposed to having the compute nodes talk directly to the EXSI host. But one thing we'll talk about later is how do you make sure that you place the right workloads in the right environment? As you note here that we're running the next-gen type apps on KVM, we're going to run the traditional Oracle Exchange stuff on the vSphere clusters that are being managed by this OpenStack environment. And notice the key... One of the key facts here is that we're placing workloads where the underlying infrastructure is best suited to manage that workload, right? So applications that require resilient infrastructure that typical KVM architectures aren't capable of supporting today. In most cases, that's where we apply applications that can handle resiliency in the application itself, applications that assume resilient infrastructure than would run on a vSphere hypervisor, and the hypervisor can take care of that stuff. All right, so let's delve into a little more detail on exactly how this stuff works. So most of you are aware, you know, OpenStack has its compute scheduler that is responsible for looking at requests coming in to schedule workloads and then determining where those workloads should run. And it has a database of all of the hypervisors that are available to it and how much resources each hypervisor has available to it. And so what we've done in this particular case, what VMware did as of the Havana release continuing forward into Icehouse is we have a vCenter driver. It's in the trunk, so when you download it and you get it, there's nothing extra you have to do. You configure it in NovaConf. And what we do is we present an entire vSphere cluster as a compute node, right? So we're spoofing a vSphere cluster as a compute node. Now the benefit of this is a couple. One, it allows Nova to take advantage of vSphere without actually having to require any changes upstream in Nova, so very consistent sort of appearance to the higher level Nova scheduler code. It just sees a compute node and it goes on. We don't have to make a bunch of radical changes to that architecture. Second, it allows us to bring features into OpenStack like vSphere HA, like vSphere DRS for dynamic workload balancing, those kinds of things without requiring that OpenStack again be changed or modified in order to support those. So we kind of hide some of the complexity behind this. Now, one of the very interesting things of how this works is that we have this compute node which is typically a VM, right, running the Nova compute services and then configured as such to point back to a vCenter server. One of the popular misconceptions is that the actual number of resources behind this sort of spoofed compute node might be inaccurate because, you know, out here in the KVM world, you've got, you know, compute node and hypervisor and compute node and hypervisor, compute node and hypervisor so I can see that. The vCenter driver actually knows how many resources are available and all the hypervisors underneath it. So we've got this vCenter server. If I've got 16 hosts underneath it, the information it reports up to Nova actually is the aggregate of those 16 hosts, right? So even though it sees one compute node, it sees that compute node is having the resources that are the aggregate of those 16 hypervisors, right? Question here? If you don't mind. Yeah, run over there real quick. Yeah, if you can, try to remember to use the mic so we get that for the recording. Go ahead. So you mentioned that you have to have a compute node for cluster. Is that a compute node per cluster or compute node for stop? There's a couple different ways we can handle that. We can actually do cluster per compute node or we can have a single compute node which will run in multiple clusters. So in the cloud house release, I think in the Juno release, they will support resource groups, multiple resource groups per compute node. Resource pools. Getting more and more granular. Yeah, being able to take a cluster and divide it into a resource pool and represent that as a compute node. So we can do it either way. We can have clusters represented as a compute node. We can have multiple clusters represented as a single compute node. Either way, the vSphere driver will actually aggregate all of those resources and present that up. So when you go into the administration section inside OpenStack, it'll see 192 gigs of RAM, 144 cores or whatever the number is, right? So that it knows what is actually available even though it's one compute node and we have 16 hypervisors or 32 hypervisors or whatever is behind it. And so what you see here is, you know, we've got cluster one and cluster two represented as a single Nova compute node. So this is an example of having one Nova compute node front-ending two clusters and then we have another cluster cluster three which is represented as another Nova compute node. All of those guys talk to a single vCenter instance or could be configured to talk to multiple vCenter instances so we can aggregate multiple vCenter instances up into an OpenStack compute infrastructure. Then though that vCenter instance or vCenter instances would then manage one or more clusters of ESXi for the actual hypervisor support which then would leverage a shared data store. That shared data store could be a traditional sort of ice-guzzy sand, vibrational sand, something like that. It could be NFS data store. It could also be vSan which is VMware's scale-out storage technology that was introduced in 5.5 so that these hypervisors could actually be taking their local storage and aggregating them to create a shared data store that then they would all leverage from there. And then looking at how images work, you know, you're going to still use glance to pull images out. Those images will be VMDKs, right? And what will happen when you first launch an instance is it will pull the image out of the glance image store. It will download it into the shared data store. From there, it will actually use linked clones which is a VMware technology that allows us to clone. Think of it as something like a QCAL snapshot. So it will use a linked clone for every subsequent instance that you spin up on that cluster, on that data store, from that image, right? So the first one might take a couple of minutes to download the image from the glance image service and then from there it will just do linked clones which will be very, very fast and also very space conservative. So basically what we're proposing, we believe the best way to deploy this to infrastructure time environment is this idea of workload zones. So you want the workload that needs a resilient infrastructure but maybe it doesn't need rapid scale to sit on the vSphere side and you want the workload that it needs that can tolerate a fairly fragile quote-unquote fragile infrastructure but needs to scale very rapidly. You put that on the other zone. But one of the things that Scott mentioned is this idea that in the vSphere side the compute node sees the entire cluster as a single resource, right? Which kind of brings up an interesting consideration which is that there is a potential, right? Because the way Nova schedule works, Nova schedule will low balance depending on which compute node at any one time, at the time that you want to launch a resource has the most amount of resources. So you could have a scenario where a KVM compute node might have eight gigs of memory and then you have an EXXI cluster with four servers. Each one only have four gigs of memory but because Nova schedule sees that as 16 gigs aggregate it's going to keep placing load onto that cluster. And also because, again, we're saying there's two different types of workloads. You want to make sure that you don't put your Oracle workload on the KVM side of the house by mistake or that your MongoDB stuff on the vSphere side, you don't need to. So the way to do that is there are a couple of ways. One is to create this workload zone in an open stacks corner availability zone. So you could create two availability zones, one for vSphere, one for KVM and then when you launch your instances you basically have your developers choose the zone that matches the hypervisors that they want to place on. The other way to do it is use something called host aggregates where you can basically specify that certain nodes have certain capabilities. So one may have an HA capability while the other doesn't and then you can use the APIs when you spin up the instance to use the correct host aggregates. So there's different advantages and different advantages using aggregates versus azs which we don't have time to go through but the key there is there is some configuration that has to be done ahead of time to make sure that your application is sitting in the right zone or the right type of compute node environment. So for example in this here you see there's an Oracle host aggregate as well as a KVM host aggregate. Again to specify where you want to put the particular workload. So Scott's going to talk a little bit. So obviously it's great to have all these compute instances but at some point you need networking to make it all work together. So Scott's going to talk about that and how that works in this multi-hypervisor environment. So what we needed here was something to do cross hypervisor networking. We needed something that would also would connect into VMware vSphere and provide a full set of networking services and network support there as well as something that would reach into the KVM zone. We wanted to leverage OpenSec Neutron and so VMware NSX provides the cross hypervisor support that we needed supporting a virtual switch on vSphere that allows us to be programmed by the central controllers as well as supporting Open vSwitch on KVM which also be programmed by the controllers according to instructions received from OpenStack. And so in this particular case what we have and I gave you I just picked out an address here. Anybody know where this address range comes from by the way? This is an address range that Class B address range that Microsoft owns. It used to be used in the old Microsoft training books. So if you go back that far enough, right? I used to be a Microsoft certified trainer so that number is like ingrained in my head. But so let's just assume that this is the address space you're using and it's in your environment that ACME is using in this case, right? We have a couple different options for how we handle this, right? So traditional workloads probably are not going to be really happy about having some sort of NAT device sitting in the middle between them and all of their clients. So one of the things that we can do in this particular design is we can actually spin up logical networks and those logical networks can be automated via heat templates. They can be automated via API calls. They can be automated through the UI although manually doing it through the UI isn't really automation, but that's a different story. And we can actually assign them blocks of address space out of the so-called public space that belongs to ACME Corporation, right? Now that public space could have just as well been in RFC 1918 10-Net or 172-Net or whatever. But we consider it to be publicly routable for that organization. And so then those routes can then be propagated into the rest of the network and they appear as just another subnet sitting off the network and then you place all of your hosts. And I used a single logical network here with one subnet, but it just as well could have been a logical network with three different subnets and logical routers in between them and all this kind of jazz. I mean, the logical topologies that you build can be as complex or as simple as you need them to be. Either way, NSX coupled with OpenStack Neutron will handle all of that for you. In addition to that, these are kind of your production workloads, right? So Oracle Exchange, other types of stateful traditional services. In addition to that, one of the key goals for ACME was bringing in this development environment, right? For their developers to do prototyping. And so what we can also do is we can spin up logical networks and ultimately there are limits, but basically you could have a very large number of per developer, developer specific logical networks. These logical networks could be exactly identical to each other if they wanted them to be. The example I gave here is that these developer logical networks are exactly the same. They're all using 192.168.1 address space, but because they are all isolated and fully separated from one another, that's no big deal, right? And so then the developers can use these environments. They can spin them up very, very quickly. They can spin up instances. They can do the prototyping for the applications that they need. So on so forth. And then they can tear them down when they're done. And all of this, again, can be done programmatically through APIs, through heat, templates, or just going through the UI for OpenStack Horizon. Either way, same system, tying into that, all being driven by OpenStack. So it's all integrated into the rest of it and all coordinated. So as instances are spun up, networks are created and instances are attached. Rules of security profiles, if you have them and want to use them, are automatically applied so on so forth. So one of these I'll call out to is if you are familiar in the OpenStack, in the KVM of OpenStack well, using the OpenV switch or using the vSphere switch in the vSphere world, because NSX is acquired, you're actually using it not using either of those switches anymore, correct? Right. So there's a virtual switch in vSphere that is compatible with the NSX controllers. It's programmed by the NSX controllers. It's called the NSX virtual switch or the NSX vSwitch. So that's what's used on a vSphere host and an OpenV switch on the KVM hosts, right? And these networks are logical networks. They're overlay networks. We could have just as well used bridge networks which would put the traffic back out onto essentially provider networks that are managed externally to OpenStack, but in this particular case because ACME wanted more control over the networking and have that control kind of programmatically and have to go back out to the networking equipment to do that. That's why we use logical overlay network, too. So questions before we... Yeah. Questions? No. Okay. We're quick about storage. So obviously as a vSphere customer, one of the things that ACME wanted to be able to do, right, is to leverage new technologies that come up with VMware for the traditional apps because they have value in them. So one of the things that VMware has done that's a little different than some of the other hypervisors is, so how many of you are familiar with Cinder? Block storage, right? Great. Okay, so I don't have to explain it, but essentially virtual volumes that you can attach to an instance. So typically that is a volume that's exported out from a servant acting as a storage node or from some storage subsystem. And it plugs in almost like a USB drive to a cloud instance. That's what I'm going to compute now. What VMware has done, however, that's a little different is they basically enable a VMDK, a VMFS to become a Cinder volume. So instead of just exporting directly from a storage array, you actually have worked through another layer, the VMDK, VMFS layer. And the main reason for doing that is, as you may know, VMware is getting into software-defined storage. So they're doing things around VSAN, around VVOL, which is coming out hopefully this year. By leveraging a VMFS-based Cinder volume, when you attach that to a vSphere host that's managed by OpenStack, you actually get all of the capabilities of vSAN or VVOL as well. So that's something that's very unique, and it's only available currently today to a vSphere host, a EXX host that's attached to OpenStack. There was a KVM host with that capability. They would use the old traditional way of doing Cinder block storage. So we also want to take some time to talk about some of the... So we talked a lot about the architecture and the technology to kind of enable this pro-solution. But we also want to talk about some of the operational challenges of trying to run these two types of infrastructure under OpenStack. Yeah, and just as a side note, some of these operational challenges are things that we at VMware have addressed, have run into directly. We maintain a fairly large OpenStack cloud that has both KVM and vSphere internally that we use for dog-fooding our own product, as well as for developers, as well as for doing POCs, all that kind of jazz, all kinds of stuff runs on this. We just run vSphere 5.5 with vSAN and KVM with Gluster. And so some of these challenges that we talked about here are challenges that we have had to address as well. One of the first of those is the fact that because you have two underlying hypervisors, and these two underlying hypervisors have basically different sort of disk formats and different expectations for VMs, you need to maintain two different sets of glance images. And so when you go to build a glance image, let's say, oh, okay, Ubuntu 14.04 just came out, and I want to be able to make that available as an image to users of my multi-hypervisor open-stack cloud. Then I need to go create a glance image that can be consumed by KVM, typically a QCAL2 disk image that you would create and optimize and then upload into Glance. And then I also need to create an Ubuntu 14.04 image, typically a VMDK, that I would then upload into Glance to be consumed by vSphere. And the way that this works is that the image, and if we were to add another hypervisor, like if we were to add Hyper-V, we would need yet another image that would be compatible with Hyper-V. And you heard Ken talk earlier about the idea of host aggregates. As you create a host aggregate, there's a certain set of metadata that is associated with that host aggregate that metadata is matched up to a metadata that's associated with the image that helps the system to know when I pick an image, which hypervisor it needs to be scheduled on to run. So we would basically associate the images with the host aggregates, and therefore, when you select an image, it will assume the correct host aggregate and then be scheduled there according to the NOVA scheduler. You want to take the next one? Yeah, so there's obviously some challenges with development testing on KVM. Because you know to make this work, right, in the real world, is you want to have your testing environment match your production environment. And so one of the things I often hear customers talk about is, hey, how about if I use KVM for my test dev, and now use VMware for my production. And the issue with that is, again, some of the stuff of community compatibility, but also the way the technologies are used, it doesn't always work. So you're sort of, again, having to maintain two separate infrastructures for both your tests, your QA, and your production environment. And that can add some operational overhead. Absolutely. So networking is another challenging area. It provides that cross-hypervisor support. There may be other products out there that plug into Neutron. I know there's a number of STN-related vendors in the marketplace, so you can check and see. And we know that NSX does that. It was designed specifically to do that. If you didn't have a solution like this that provides that necessary network virtualization, you would have to draw back something like NOVA Network. And that could be challenging. At that point, you know, I don't want to say you're going to fall back, but because of the way NOVA Network it will basically assume the presence of VLANs. And as soon as you assume the presence of VLANs to start doing any sort of tenant segmentation, then we start having to bring physical network equipment into the picture, which means that I want to go spin up something new that I may have to go and configure physical network. Which then, in turn, just by the nature of what's involved in doing that kind of slows me down and kind of introduces some challenges or some potential concerns when I start looking at a very complex networking environment or a networking environment that maybe is supported by multiple vendors, et cetera, et cetera. So using a network virtualization solution like NSX allows us to abstract that away. The control, essentially the control layer then moves up to neutron, right? And being integrated with the rest of OpenStack allows us to direct the API calls there and then it handles it from there to abstract away differences on a per-hypervisor basis or anything of that nature. Now, the last point, operational and staff readiness to support multiple hypervisors. Who's familiar with layer eight reference? A few of you, right? You have the seven layers of the OSI model, which is networking, and then layer eight, you have people, right? And so this is more of a people problem than a technology problem. So are your people ready to be able to support that? If you have a vSphere team, have they been trained in supporting Linux-based hypervisors like KVM? If you are primarily a Linux-based org, have they been trained to support vSphere? Yeah, and up to be fairly honest, right? 90, I would say 100% of the time when someone asks me about, hey, can I do one vSphere of OpenStack? It's because they have vSphere's job. They aren't that many KVM-only shops that go, hey, I want to run VMware tomorrow. So the challenge then is the VMware administrator is the one who often takes responsibility for OpenStack and it is a challenge. Right now it is a challenge not to know Linux and not to know some Python and be able to effectively operationalize OpenStack. So that means traditional VMware admins who never had to delve that world, you have to learn that world. Both Ken and I came from being VMware guys, right? So we've kind of had to go through the pain of that transition and know that if you aren't familiar with Linux, it's going to be a challenge. If you don't know certain things here and there, it will definitely be a challenge. We have a question here. Talking about operational challenges, how about if a customer has already written the products, like whether it's portal, catalog, automation on top of VMware, how would you decommission all of that and bring under OpenStack management? Sure, sir. So naturally, if you're going to introduce another management layer in, if you already have tools that have been written to talk directly to the vSphere APIs, then they won't be able to see this KVM infrastructure that you're adding. So you will have to look at retooling to look at the kind of the new architecture that you're building, which has OpenStack spanning both KVM and vSphere. I know that a number of vendors, VMware included, are taking management tooling that was typically directed just toward vSphere and now adding OpenStack awareness. So earlier today, I did a demo down in the booth area about showing how tools like VMware Loginsight and VMware vCenter Operations Manager are becoming more OpenStack aware. And I know that other vendors are doing the same thing kind of retooling their management and operational tools to be able to not address just vSphere, which is kind of the lion's share of what's out there today, but also OpenStack and other solutions as well. Does that answer your question? Yeah, I mean, I agree. I think the kind of larger picture beyond just that portal thing is you are, if you take this on, this dual infrastructure OpenStack, you're effectively now doubling your management points. Not just for management, not just for provisioning, but for management and for monitoring and collecting logs. I wish I had a better solution. I know that vendors like VMware are building our tools that hopefully will be able to aggregate all of that. But right now, you do have to separate monitoring, for example, environment for VMware and a separate monitoring environment for KVM for Zen. We want to end with a summary and then if people, I know we're running into time. So after the summary question, I think Scott and I can be around for a few more minutes and you can be happy to answer any questions. So you want to talk about some of the key takeaways? Sure. So I guess one of the key takeaways is that OpenStack and VMware are complementary in a number of ways, right? VSphere as a hypervisor plugs in very nicely under OpenStack Nova. Additional tools plugging in to help complement the ecosystem around which is necessary for an enterprise to put applications in. You know, tools like monitoring and logging and auditing and those kinds of things, right? And so there's a lot of ways which there you can do there. The transition for a VMware administrator really isn't as great as you might think. If you are a VMware administrator looking to make that transition over to OpenStack and you don't have learn Linux at the top of your to-do list, I would say put it there because you really need to have that sort of background. But because they are very complementary, there are a number of ways that you can help make that transition. I'll talk about a couple of those as we move on. Why don't you go ahead and finish this. Yeah, so again, back to workload dictates architecture, right? So each of these workloads we're trying to tell you have different requirements and you need to pick the right tool, the right infrastructure for the right environment. One thing I get a lot is people coming to me, customers, potential customers saying can I use OpenStack as a free version of VCloud? And the answer is no. If you try, if you want OpenStack with KVM or Zen and you try to run Oracle on it, your users will not be happy with the results. And vice versa, there are certain workloads that if you want to run it on V-Sphere, on V-Cloud that they may not be happy with the results versus what they could do with OpenStack. So again, it's really important to, you can run both infrastructure together, but you need to run the right workload and the right infrastructure to tie them together with a common management point. Right. Which again, I'm just going to use the right tool for the right job. Now, just a few things that aren't listed here and then we're going to go into Q&A and because this is the last session you guys are welcome to stick around and we'll talk and they're not going to kick us out or anything. But one key thing to take away, if you are looking at as a VMR administrator and you are looking at making that shift or have been asked to make that shift by your organization or you're just interested and you want to join more, a couple different ways to do that. One of the easiest ways is to download something called the VMware OpenStack Virtual Appliance or Vova. If you just do a Google for Vova, it's a virtual appliance which you can download, plug in some information, deploy it onto an existing vSphere infrastructure and then you've got an up and running all in one node of OpenStack that allows you to basically experiment with it. Right. It's not a production grade tool. Right. So, you know, I'm not telling you guys go out and deploy your production cloud using Vova. It's a single virtual appliance. Right. But it's great for getting your feet wet, getting used to it, testing the APIs, testing command line clients, etc., etc., etc., seeing how it works, getting custom to it. So that's a great tool to use. Absolutely free. You can download it, deploy it on your existing vSphere infrastructure and go from there. Ken, you know, we should have put a list of URLs up. We didn't include that. But Ken has a great series of articles on his site, cloudarchitectmusings.com. Right. Okay. Yep. And so he goes into what is it, five or six different series of articles on, you know, design principles and considerations around deploying vSphere with OpenStack. Lots of great information there, so that would be another place to check. All right. So that means that we're officially out of time and we'll open it up to questions. I think we've got time to hang out here. So feel free to come up or hit the mic, whichever works best for you guys. Yeah. Mic would be preferable if they're, I don't know if they stopped the recording or not. So, you talked about the two different images data or image metadata come back together again? So the image metadata is actually associated with the image itself. Right. Yeah. So there's like if you were to do a glance image list you would see the glance image listing and then there would be these additional metadata fields that would help tie it to the host aggregate. Okay. So does that get then I'll have to have two different paths. If I want to get all, keep all my, you know, information about my images in one spot, I'll have to come to it in two different ways. So glance is the one place you will do that and it maintains that metadata and the image stores. But I thought you said I'd have a glance for VMware. You have to have two images, two images, one instance of glance, two images stored in. All right. Thank you. Okay. No problem. So, my question was around going back to where if you have open stack top layer, right, and then you've got V sphere sitting underneath it, you could continue to use technologies like SRM to do your DR, whatever you want to do. Is there anything, well I'm assuming that's the case from my understanding, is there anything that you can't do? So if we bring open stack in on top because it's going through V center, I'm assuming all the traditional VMware admins could continue doing what they do and not even be oblivious that we're using open stack on top and not have any problems. Right. So you could but you generally won't want people so you can use the V sphere tools as a, as an operator tool like to you know, to troubleshoot, make sure things are working that kind of stuff but you wouldn't necessarily want to have them like renaming VMs or migrating them to another cluster or something of that nature because you could be interfering with how open stack perceives that to be. Okay. So you're generally going to want to do the majority of your tools that can be done in open stack to be done in open stack and only those tools that would absolutely be necessary to be done back in on the back end. I think the biggest issue there is like typically when you do a management orchestration layer is you assume that management layer is a single source of truth. That's right. And by kind of going outside of that you've now that single source of truth is no longer a single source of truth. Now there are some things that you can do. For example, I can do a V motion of a VM. Yeah. Or I can put a host in maintenance mode that kind of thing and the V sphere driver will know about that and it will be able to communicate that information up to open stack and that's fine. There are certain things you can't do, right. For example, you can't migrate a VM from one cluster to another cluster when those clusters are separate compute resources, we talked about how a cluster is represented by a Nova compute instance. Yeah, yeah. If I have Nova compute instance one representing cluster one and Nova compute instance two representing cluster two and I do a V motion over here, then that will break open stack because open stack doesn't know that. It doesn't know how to handle that. Are those lists of possible gotchas documented anywhere? They may be. If they are, I don't know precisely where we do have an open stack community on the VMR communities page and it might be. I've documented some of them in my blog post. Oh yeah, I was going to say that. I know some of them, yep. Just what you said about moving stuff. So in SRM for example, if SRM kicked in and started doing a DR recovery, it would be screwed basically, right? It would work, but open stack wouldn't necessarily know about it. Let's talk offline after. There is another option if you don't want to do it. If you're concerned about that, is actually to not put VMVC underneath of open stack, keep them separate, but then have an orchestration layer above open stack and the vCenter to manage that. So one of the things I'm hoping to give a talk on at the end world is using VCAC to manage vSphere open stack public and private or from one interface. So that would be one way to act. You could put another orchestration layer on top of that to help if you wanted to do that. There's a couple different ways to slice the beast. Thank you. I always enjoyed listening to you guys. My question was actually an extension to what he was saying. So most of the customers that you go into with VMware already in place, they have thousands of VMs already running and hundreds of applications already out there. What you're addressing here is a green field workload. What about the brown field? Are we doing anything in that area at all? So there's a couple of different things you can do there and Ken I want you to weigh in on this as well. Some customers have chosen to basically slice off a piece of the infrastructure that would be managed by open stack. Maybe they'll take a cluster aside and they will give that to open stack and so you have open stack consuming one cluster or one portion of a cluster for workloads that will be turned up from the very beginning in open stack. That might initiate sort of one of these shuffle game migrations where you put up the new workload and then you put everything over there and then you decommission the old that kind of thing until everything's put in there. They suck all these instances out of eSphere and represent them in open stack. We've looked at that. What about in place management of that? At least doing some life cycle operations through that. Can you elaborate on that? Say for example I have a thousand VMs that are already out there and spread across a couple of different clusters. If they are made visible through the open stack interface I could do something with it. At least I'll be able to do some life cycle stuff. Give me your contact information. I'll pass that to developers. We're also looking for customer use cases that make sense where that would be useful. Catch me afterwards. On the Nova side, actually not that difficult to do is actually the networking and only because you're now incorporating NSX. So now it's a migration from vSphere distributed switch maybe to an NXX environment and obviously there's some work involved in that. The Nova piece is actually you're basically just taking a computer and a server and manage it. Thank you. No problem. Come on up, that's fine. This could be a really basic question but typically it is recommended that the way it works is if you have multiple hypervisors you have different controller nodes and then you have open stack on open stack and manage this the whole different hypervisors. So typically in the production environments what complexities does it add when it comes to when you have multiple controller nodes you have to manage. You don't need to use you can use the same controller nodes. It's the compute nodes that are different. So you have to have a compute node that manages vCenter sort of proxies and then you have compute nodes KVM. But the controller nodes are the same for both environments. So yeah, you would have like in the diagram we showed we had a redundant pair of cloud controllers but those cloud controllers will control all the hypervisors in the environment. So if we had added Hyper-V to that environment we would still use the same redundant pair of cloud controllers to manage that third hypervisor in addition to KVM and vSphere. It's how those hypervisors represent themselves to Nova where we introduce some other differences like KVM has a Nova compute node that's going right on it. vCenter is sort of proxied as Ken mentioned by a Nova compute node which may represent one cluster or multiple clusters. Does that help at all? Thanks. Come on up guys. This is maybe more for comment than a question to say but I get the feeling that there's a lot of people who seem to think well I could use vSphere or I could use OpenStack and they kind of do the same things. Is there any clear criteria for applications that you want to run and how you decide which is more suitable and why you would choose one over the other because there seems to be a lot of to me just well we're going to try out OpenStack we're going to see what it does but it's like no I've got specific applications I need to run. How do I know if it's suitable or not and do I just try it out? So we talked about the big game to talk this idea is that a loosely coupled distributed application versus this monolithic kind of stateful application so basically the application and how it handles failure dictates which infrastructure you need the application can handle its own failure properly then maybe can sit on a cloud if it relies on infrastructure you use then it probably needs to be suited for a vSphere type environment the real question or the real thing to remember though is that vSphere is a hypervisor with a management solution wrapped around it OpenStack by default does not include a hypervisor we have to add KVM to that so what most people I think say when they say I'm going to go try OpenStack is I'm going to go try Linux and KVM being orchestrated by OpenStack that part is both that's one of the reasons why I can't do these talks is they'll be able to understand that that hypervisor is a perfectly viable solution now in this particular use case that we selected disclosure I worked for VMware we could have run the entire infrastructure on vSphere we didn't need to bring KVM in now there's a very specific use case that I want you to share in just a moment where the way vCenter serializes tasks could present a problem for a certain use case but from a hypervisor perspective you could run it all there but there might be other considerations maybe it's a budget consideration where as a commercial product you have to pay for licensing that's how we survive if that is a consideration a design constraint for you then yeah maybe KVM is necessary but you need to go through the design process to determine what are the capabilities that KVM provides as a hypervisor that are exposed up through OpenStack as the orchestration layer what are the capabilities that vSphere provides as a hypervisor that are exposed up through the orchestration stack in OpenStack to be consumed and then map your application to get that the broad breakdown that Ken and I have provided is that if you need an application where the underlying infrastructure can handle failures for you then you probably need vSphere because none of the other hypervisors on the market typically do that yeah so I just wish they would talk about OpenStack and separately talk about and we absolutely agree that it's KVM and that this is a whole package and 80% of my discussions with customers is all about that what is actually suitable for KVM under OpenStack versus vSphere you guys have talked quite a bit about integration between OpenStack and vSphere level do you have any recommendations for companies that are using vCloud Director and have a lot put into a catalog and the vApp concept run in vCNS so the key word the operative word here is co-oppetition so to be frank vCloud Director competes with OpenStack vCloud Director is an orchestration tool although it does a little more sort of abstraction things than I think OpenStack does which is good or bad depending on how you look at it there is no sort of hard and fast guidelines I guess you would need to evaluate those two again the same sort of thing evaluate the functionality they provide against the business requirements determine which one is the right one make sure that you're clear on VMware's roadmap with regard to vCloud Director and where they're going with it which is another story that I'd prefer not to get into and then make your decision from there but they do serve the same function in many cases so if you were looking at something where you're maintaining a vCloud Director type environment my advice would be then to look at something like Service Mesh or vCloud Automation Center as the layer sitting above both OpenStack and vCloud Director and using that as your management and orchestration tool player and to migrate say from one to the other I know NSX is a platform that can be used across all of that is it possible to use that as a visualization? Potentially that would be something we'd have to explore in more detail thank you for your questions so the glance images you mentioned that the images have metadata etc etc right so let's say for example we work today in RMS we have two images Windows one and the next one and we put everything on top of that so we would have to create four images two for Windows and two for Linux but how does that add to the metadata was it an extension so if you want the metadata to be represented there are some cases I'll give you an example from our internal cloud one of the things we want to be able to do is run nested hypervisors you can't run nested hypervisors on KVM it just doesn't do it so anytime somebody says in the image that I want to run a KVM host that image metadata because the only hypervisor in the physical sense that can do that is vSphere so some of it is when somebody says they want to run ESXi we know that it has to be scheduled on vSphere because that's the only place you can run it or somebody says I want to run an Ubuntu KVM host which is one of the images that I know I have to schedule it over here otherwise you would have to represent in the name of the image for them to distinguish if you want them to distinguish which one is which you might represent that through SLAs like Ubuntu H.A. something of that nature to distinguish let them distinguish how those are going to be distinguished in terms of which one they're going to select does that make sense at all? Yeah, the question was around you mentioned that there was a use case where you Yes, we didn't go to that You want to cover that one? I have a customer who has built so he used to think I'm six weeks to give their developers a full stack infrastructure to Tesla so they went to VMware and they went to vSphere with BMC and they were able to get it to Dan where they could spin up 20 VMs full stack application install in 15 minutes pretty good, right? Six weeks, 15 minutes then they went to the lines of business and presented that and they said sorry, not good enough we need you to spin up a couple of hundred VMs with application install we've done it under five minutes because that's what we're doing in AWS and you want us to come back so they actually did some study and they realized they couldn't do that in vSphere and the main reason is because vCenter can only handle eight simultaneous requests so basically when you're trying to spin up all these resources you start bottlenecking plus the way the hierarchical I'm getting a little deep here the hierarchical layer the way vSphere is architected you sort of have to build out these pieces some of it in serial so again, you're bottlenecking there as well so now they're looking at using OpenStack of KVM or Zen because we don't have vCenter actually essentially working as a bottleneck and plus the way OpenStack works you're able to build out all these resources in parallel and then use an API and pull them together at the end we've actually got that exact use case the key limitation there is that because we're driving everything through the vCenter server vCenter's serialization of tasks ends up being a bottleneck again, vCenter gives you great functionality that some of your applications require so again, it gets back to the right tool for the right place now you could work around some of that multiple vCenters and that kind of thing but that's just an architectural decision so yeah, there you go does that answer your question? Yes? Hello, first I want to say thank you for this talk it was very helpful, especially we've been thinking about doing something like this so with what you presented today I'm curious is do you have anything online that talks about this architecture so what I'm thinking of is being able to present this to different people in management so they can understand what this is what the proposed thing is today part B of that question is what's for the future it sounds like this is a good first stab at it but it seems like there's still some warts that need to be kinked out but I am appreciative of what you guys have done so far so what's the future and then is there anything online for what's today? Let's get to our future I can't comment on futures necessarily for say product futures we are committed to the code we are the number four contributor to in terms of the amount of code that contributed so we are obviously committed to OpenStack as a company we are going to continue to develop products so vSphere enhancements will continue to come we talked about resource pool support so being able to subdivide a cluster and have that represented as a compute node this gives you a lot of flexibility in terms of how you do things how you divide up resources and multi-tenancy and that kind of jazz so we are constantly exploring more things if you were in this room prior to Ken and I getting on one of the guys up here he was talking about scheduling options so obviously we are deeply involved in all this kind of stuff so more will come and we will continue enhancing we will continue developing so just stay tuned for that in terms of online stuff there is kens series of your number of articles and then these sessions are always recorded we did one in Hong Kong which is recorded and available online so a number of other various presentations on it to be frank there isn't a lot out there right now obviously there is OpenStack docs that will help some so you have to pull together a lot of things so my suggestion is you email Scott and say Scott can you write a book and they put the logo up before you can email but yes, you are more than welcome to email my address is eslo.vmr.com so feel free to drop me an email first initial last name at vmr.com so talk with you in more detail I don't know about writing a book talk with you in more detail if you have anything like that last question I promise I know these guys are eager to get out of here they have been great control the compute node for the cluster the way you have shown it is one node so I have a question I keep thinking of it as a physical node obviously that would be a waste of resources if you make it a virtual node where does it sit and if it fails what do you do do you lose your resources or can you just put up another one and continue from where you left off so there are a couple of different ways to do that one very common design principle where vSphere environments have a management cluster where resources like vCenter and that kind of thing sit so in a large scale environment like this we would say represent a management cluster and in that management cluster we would host on a vSphere HA enabled cluster the compute nodes that would point to vCenter instances for your other clusters yeah kind of vInception alright thanks everyone feel free to come up and talk informally thanks to the AV guys, appreciate it