 Hey, so my name is Scott Carlson, and I'm a senior engineer at PayPal. And what I'm going to talk to you guys about today is how we are combining our OpenStack cloud to have both KVM and ESX vSphere 5 hypervisors within it to enable multi-vendor agility within PayPal. First, let's talk a little bit about what PayPal is. I think a lot of people know who PayPal is and maybe have used PayPal, but we have 137 million users across 26 currencies on 193 markets. We have devices like PayPal here, some chip and pin technologies, and is currently the world's most widely used digital wallet. So for PayPal, it's very important that we have a solution that scales globally and is able to handle all these users and currencies. So first let's talk about why we went to ESX, because I think that's an important part of the PayPal story. Starting in 2009, 2010, prior to that, PayPal was 100% physical. So all of PayPal.com and all back-end and front-end systems were all just one-you pizza boxes scaled throughout the data centers across the world. In 2011, we decided that it would make sense to start to virtualize PayPal so that we could get into more automation, more cloud things, et cetera. So we selected ESX on VCE Vblocks as our first place to implement. The reason we did that is because PayPal had no virtualization, we also had no virtualization skill in-house. And taking a look at the industry, the best way for us to get into the virtualization game was to go with the technology that everybody knew and that we could buy and initially implement right off the shelf, which is what led us to VCE. We brought in a number of Vblocks across all of our data centers and standardized on that infrastructure for our web tier and mid tier. So over the next 18 months, I was part of the team that virtualized 90% of the PayPal front-end on ESX, which amounted to roughly 10 or 12,000 VMs across three data centers. One of the things that we looked into when we went with VCE and the Vblocks is we needed a solution that we could measure and monitor via the way that we scaled our workload. When we take a look at how PayPal scales, we have a very defined C++ compiled workload that scales very horizontally as our traffic volume goes up. It's very nice that it works almost exactly out to spec int. So if you take a look at a processor and it's spec int, divide by our cool little formula that our capacity people have, we can buy hardware based on the number of spec int it gives. You can translate that pretty quickly to a V mark under VMware. And so we can scale four core VMs until we run out of infrastructure and scale paypal.com on VMware. So it was a very easy understood test harness under there. So in 2012, the executive management of PayPal declared that we were going all in OpenStack. So here comes the cloud, right? The team that I'm on was said, let's go do OpenStack. Everything has to be on OpenStack going forward. So what does that mean? Standard, elastic, scalability, our internal cloud model. Everything needs to be in the cloud. Everything needs to scale, and it has to be easy. One thing that also we want to think about on this, because we sort of paid to get in the virtualization room, right? VMware and V blocks are not the cheapest thing you can buy. So they wanted to start to reduce the cost of every payment that we took by bringing in OpenStack and bringing in more scalable infrastructure. And also enable the developer and enable the business. Consider this the standard DevOps model, right? Today at PayPal, pre-cloud, we'll call it, there are some stories floating around that have been talked about at other conferences that it took a developer who wrote a piece of code, 168 work tickets to get that code from compile into production, which was firewall rules, VMs, multiple code pushes, QA to prod, et cetera, 168 tickets. And we are trying to simplify that down, ultimately, to where the developer can go in and click, and the PAS layer will do it. But initially, it's about at least give them a VM that works in minutes, at least give them the application that they want. In our case, it's the Java, Jboss sort of deal, or C++, to let them go quicker, right? We're trying to go initially down to seven days, and then one day, and then minutes. Let's talk a little bit about our cloud, because this sort of plays into where we went from ESX. Standard sort of best practices, if you look at other people who have talked about this sort of thing. We wanted open source, right? We wanted to be able to manipulate the code. The thing about PayPal and most big, large, Fortune 500s, especially the ones that I've been at, nothing that you buy ever works out of the box. So you need to customize it for your business practice. Maybe you shouldn't, but you need to be able to, I think, especially for some of the PayPal practices. We have PCI requirements because we deal with a lot of money. So we need to be able to PCI certify, we need to go to other vendors. We need to run all that through. We also didn't want to do vendor lock-in, right? We're big enough to want to have the ability to spray across industry globally and not do that. Also, PayPal is a subsidiary of eBay. I don't know if everybody knows that. eBay is a large commerce marketplace, I think, known worldwide. PayPal's the payment subsidiary of eBay. And eBay had already made considerable investments, both in VMware, OpenStack, Nasera, other cloud technologies. So we wanted to combine our heads with them and see where we could go forward. So that brings in OpenStack. If anybody just went to Jonathan Pickard's main session, it's about agility, right? OpenStack is about agility for PayPal. Let's get from this, six months to deploy six servers for this application to six minutes. And that's about agility for PayPal. So now let's get to the technical details of how we're doing this and why. This is our technology stack at PayPal here. The blue boxes are the kind of stuff you would expect at this conference. I did this exact same talk at VMworld. So I had to explain what all the boxes were. At the OpenStack conference, I won't explain what the OpenStack words are. But effectively, the blue is the stuff that sort of comes. The green is the stuff that PayPal has manipulated a lot or a little. And the red is the thing I'm going to talk about right now. So we use OpenStack right from trunk. We're fulsome and grizzly in multiple cells right now. Cell is our word, not the grizzly word. And we have a custom load balancers of service. DNS is a service. And firewall is a service. The firewall as a service is not quite all the way there yet. But the load balancers and DNS are used real time in production today and have been for six months. And then our deployment portal, it's a cool little front end that the dev team mocked up for our developers to build stuff in QA. I think everybody, if you haven't, please go check it out, we took Asgard from Netflix. We ported it to OpenStack and we contributed it back to the community a few months ago. And so we have an Asgard-based front end that we're enabling for multiple clouds. And that's our deployment portal for most people. So we're building this OpenStack cloud. We had our first set of hypervisors in production in December of 2012. And since then, we've deployed a lot more. So if you take a look at about June of 2013, you end up with this picture. FC is fault zone. And when you take a look at PayPal, of course the standard high availability rules of a big company apply. We need four of everything. We need three data centers, et cetera. So you end up needing 12 of everything. And in our ESX world, because we're based on Vblock and because of the capabilities of ESX, we consider that appropriate for stateful workloads. Something that needs to retain data, it needs V motion, it needs the SAN, it needs all those cool capabilities of VMware. But we sort of decided that maybe that's not appropriate for our web tier. If you take a look at how PayPal scales, and I think anybody from the US knows this, or can imagine this, both PayPal and eBay are three biggest days of the year, Black Friday, Cyber Monday, and then the last shipping day before Christmas. Those are our three biggest days of the year. If you take a look at the traffic volume on those days compared to January, for instance, it's hundreds or thousands of a percent difference for the different business lines. So that ramp up and scale up needs stateless things. So when we take a look at our cloud, the big orange box up there, our goal when we started to bring it to OpenStack was how can we make it cheap? How can we make it massively scale? And how can we make it all stateless? So when we picked KVM, one of my jobs is I'm the hardware guy on the team. So we picked a platform from HP, or HP based at PayPal, on the hyperscale SL230 product line. So we have 96 servers in Iraq, all local disks, Sandy Bridge series, let's see, I think I have more details over here. But effectively, so we have a big stateless OpenStack cloud and a big stateful VMware cloud right now in this case. And roughly it's probably 20,000 cores, 20,000 physical cores on each. So we have a lot of horsepower in each of our data centers to run the various things. So let's take a look at this as it applies to some of the people who are trying to use it on my sysadmin team. When we brought in VMware, we had all these grand visions of taking the vSphere and vCenter APIs. We were going to write some cool tools in front of it. We were going to put it in a little portal, and we were going to let people use that. What we ended up using is a PowerShell script using the Perl module that one of my sysadmins wrote on the weekend. And that has been our deployment tool for the last 18 months. So if you want to deploy something in VMware, you have to log in, start a PowerCLI, and run the build script. Well sysadmins don't like that. And then you go to the cloud, and you get the cool Asgard interface. You click 8, you click what zone you want, and you click submit, and it's done. So we have two different sets of environments. Each vBlock is managed separately, because it has a zone vCenter. So we end up with 20 unique environments to manage. And that's why we want to put this all together under OpenStack. Originally, we started to have the debate, and it's still ongoing. Why don't we just take KVM and get rid of VMware? Why don't we do that? It would be easy. And being one of the guys who doesn't believe that we should do that, the reason why we don't do that is because we have things like a vBlock. VMware has certain capabilities that I need for the workloads that need it. Scott's opinion now, things like KVM are just not there at the quality of vMotion and DRS and auto placement. All those kind of things just aren't quite there yet. And so when you deal with things like that and the stability of the vCenter hypervisor, you need that VMware in your environment, especially in big corporate environments, where you have those commercial workloads that fit that. So it makes sense to have both. Real quick, back to our compute racks. Because we're stateless here, we have, at this point, introduced no shared storage into our cloud. We're just starting to get into some shared flash cinder-based things. But in general, we just have 96 servers in a rack, and we scale them horizontally. And our front end has 400 servers in a couple of racks. If 10 of them break, nobody cares. We follow that whole rule of 10%. I think Netflix has talked a lot about it. You don't care until 10% of your environment breaks. Then you care, and you go fix it. So after integration, it'll be all pretty. The idea is that we'll have a standard cloud management zone, we'll call it the management network. It'll have our cloud in it with the Asgard front end. All the hypervisors will continue to live in their current form, but managed by OpenStack. So when PayPal talks about managing ESX and KVM, or vSphere and KVM, with OpenStack, this is what we mean. We are not taking and mushing the things together. We're using what vSphere is good at in the places that it's good at. We're using KVM in the places that it's good at. And we have put OpenStack on the top of it to manage it all. Because those people want a single interface to log into to manage everything. And then they can just pick the fault zone or availability zone or host aggregate that they want to deploy to. I like this slide because it's sort of tech speak. And this was a lot more important at the other places where I've given this talk. But a lot of people have asked me, don't you replace ESX with OpenStack? No, you don't. You don't replace ESX with OpenStack. First of all, ESX and vSphere are not the same as OpenStack. Nova is not the same as vSphere, vCenter, or ESX. Nova is kind of more like vCloud Director or vCloud Automation. And KVM is like ESX or vSphere. I use them interchangeably, sorry. So when you start talking about replacing one for the other, and you talk about hypervisors, you talk about KVM versus vSphere. And then you have to talk about this stuff on top of it. You don't just say, I'm going to replace VMware with OpenStack. I'm going to replace vSphere with OpenStack. So how are we going to do this? So as of today, we have ESX hooked to our OpenStack cloud in our QA space. We missed our end of year moratorium to get it in production, but we'll be doing that in Q1. As you can imagine, we stopped working on the site many months before Holiday to make sure it's rock solid. So what did we think about when we actually tried to bring ESX into our cloud? These things right here are sort of important criteria, things we needed to solve. First, we started looking at the functionality. Does KVM and ESX do the same thing? Yes, we're pretty good. They can all host virtual machines. When I bring ESX under OpenStack, can I do the things my sysadmins want? Can I build machines? Can I turn them on? Can they turn them off? Can I VNC to them or do the virtual console? And can I rebuild them like they were, keeping the same IP and the same hostname? And can I delete them and make them go away? Yes, all those things are there. We're good there. Auto configuration, that's more interesting. We have sort of the standard T-shirt methodology for VMs, extra small, small, medium, large, bare metal sized. Can it apply those configuration settings on build time? We're good there. IP address management is more interesting. I'll get to that later. Can I build from snapshot and template? That's interesting in this case, because especially when you start talking about multi-hypervisors, you can't use the same snapshot on different hypervisors. So you have to maintain glance images for a hypervisor and potentially every image size and potentially every pattern of application on image size per hypervisor. And so depending on how you build, you can end up with a whole bunch of images in your glance repository. Let's see, can we deploy in the fault zone? Yep, that's just a Python tweak. And can we put it in one interface? Yes, because we're considering all these a fault zone, host aggregate. I guess we're using the term host aggregate now. So what version of vSphere do I need? Did a lot of testing in our lab. I originally started on vSphere 50U1. And I found that I could make it work, but I had to start backporting a whole bunch of things. If you take the vSphere 50 vCenter appliance and you tried to build it, there's unfortunately a couple of bugs in VMware's implementation that they were missing the XML API libraries. So you had to copy in a whole bunch of stuff and make it work, et cetera, et cetera. So it was just a big problem. So we stuck with ES6-5.1. Since I did this slide, I didn't upgrade it to 5.5. So 5.5's out now, it works better than 5.1. So 5.5 is where I would go if you're gonna do this if you can. If you're going brand new, do 5.5. But if you have 5.0, I recommend 5.1 vCenter. I think 5.1 will manage a 5.0 vSphere just fine, but you at least need the APIs in the 5.1 vCenter to do everything we're doing here. So the hypervisors is interesting. Because PayPal has PCI, and because we have no proof that anybody has ever certified a KVM hypervisor for PCI with multiple zones on the same hypervisor, right? A PCI and a non-PCI workload on the same hypervisor. We made the decision that a single hypervisor could either be PCI or not PCI. In the VM world, you can use vShield and you can share. But in the KVM world, we have no proof that that's actually been certified, so we go separate with that. And so we started to have our VMware workloads follow that same pattern just for ease of use, right? So if we build a hypervisor, it's either a PCI hypervisor or it's a non-compliance, non-PCI hypervisor. And the open stack management just follows our standard network. We share with the vColonel network sometimes and we have it separate from the vColonel network other times. When you take a look at the storage requirements of the data store support for the ESX driver, the vCenter driver for Nova, it has, you need a data store cluster in VMware. I should have probably asked if people speak VMware in here, is there a lot? Okay, enough. So you can sort of cheat and you can create a data store cluster with a single disk and you just call it a data store cluster. So you can get around this requirement if you don't have a shared storage. But you need shared storage for this to work, the data store cluster anyway. You need DRS enabled with auto placement. So if you have shared storage, it works with multiple clusters. But if you have a single disk that you're cheating on, of course, it just auto places on that one data store. And you have to create this in advance or at least we did when we implemented it. I believe this is all checked into a Vannano where the Cinder support is there. I have not tested that, but I think the Cinder is there. And so I think people know this OpenStack grizzly picture from Ken Peppel. How do we actually plug this in to OpenStack? Now that we have, you know, I have like 12 vCenters flying around the network and I have all these KVM environments, how do I plug this in? It's all about Nova, right? So this is what makes this easy. How do you put ESX in my cloud and how do I make it easy? And you do it like this. There's a couple of different ways. People who know KVM, it's just a libvert thing and I spelled libvert wrong and I forgot to fix it. But with VMware and vCenter, there's two ways to do it. We'll start on the left. You can either point at vCenter. So if you already manage yourself with vCenter, use the vCdraver and you modify the NovaConf to point at your vCenter. You give it the IP address and all that kind of stuff and it manages through vCenter. But if you have standalone hypervisors, you can just use the ESX driver and point it directly at that ESX. And so these features for Nova are already included out of the box. Unless you want to download some of the fixes that VMware has contributed back, you can basically just use Grizzly or Havana trunk to do what you need to do for VMware. A quick sample of my lab config of Nova here. Pretty simple to get this to work. Once you, so this is implemented against a out of the box vCenter 5.1 appliance with all the defaults built in. So I put in the host API in my lab, root VMware is the standard appliance password. I had built a cluster called open stack test in my vCenter. It had two hosts with a net app behind it, I think, and this is the wisdom location for where the APIs are for the vCenter appliance. And this is all you need in order to build VMs from the Nova side. When you take a look at the cluster name, I'm not sure I talk about it in here. But originally out of the gate in first Grizzly, you could only put a single cluster here. And so what that meant is if you had something like RV block or you had multiple, multiple clusters inside your VMware farm, you needed a Nova compute node for every single cluster you wanted to manage. And so if you have lots of clusters, you end up with a whole bunch of Nova compute nodes. But they added a feature so that you can just comment delimit this now. And so this single Nova compute node can manage any amount of clusters on this vCenter server. But you still need a Nova compute node per vCenter. One to one mapping. So what about glance? Glance needs a raw VMDK format disk. Sort of the standard way you would build disks under OpenStack. The one that we built here is our standard Red Hat 20 gig VM. Has to be saved in VMDK format and imported that way. They may have fixed this by now, but originally it had to be thick provision. There was no thin provisioning allowed. It ballooned it when you deployed it. And in the VMDK world, it had to be all merged back into one VMDK. You couldn't deploy split VMDKs where you had one as one disk and one as the other disk or multiple disks. You had to merge it all back into one VMDK. In our front end environment, we had two VMDKs, so we had to combine those back together and figure out a way to deploy like that. And I talked about this before, but in your multi-hypervisor cloud, you have to maintain an image per hypervisor inside your glance and then deploy them as appropriate depending on which hypervisor you're deploying to. When we go about deploying the OS, this is kind of an interesting debate that we started internally. How do we take an ESX template and turn it into what we want? Either we can have a huge VMDK that we copy all over the place that's fully deployed or we can deploy a small pixie booted one effectively and use our cobbler kickstart infrastructure and let it take 10 or 15 minutes more. So we started out this way. We built a real tiny root disk, a couple of gigs, and we just threw it at our kickstart environment and we set it up so that it just kickstarted everything on that network and it built a small VM. It was real basic, it was just the only OS, but it ended up at the point where it was usable at that point. At the image deploy side, if you take a look at what KVM has today, it has the config drive sort of features where it can mount, it can make all the OS changes and then boot it back with all the characteristics that. At time of our testing, config drive wouldn't do that with a VMware VMDK image, so you had to use the VMware Tools APIs to attach to that disk and make the changes pre-boot. Alternatively, if you go the kickstart route, you can use the S99 route or you can use kickstart or cobbler criteria to make sure your machine ends up. Where it goes. So we're in a CIRA shop and we're quantum neutron based. In order to integrate this in our environment, which was already in CIRA, requires 3.2. VMware vSphere 5.1 cannot talk directly to allocate the port. NSX is integrated in 5.5 now with the kernel, so that all works. But in 5.1, you have to use the NSX-V app integration bridge. So effectively what you do on each hypervisor is you create a helper VM that's run in the V app and a separate transport zone in the CIRA. And then every VM that is built on this hypervisor gets put in that transport zone and attached to the VDS. And when it does that then this VM, even though it lives next to a standard VMware one, will talk to your quantum neutron network. This is not real elegant, but it works good. 5.5 is native in the kernel. Like I said, so again, if you're starting from scratch, then you should probably start with 5.5. I did not do any testing on 5.0 with that to see if it worked just 5.1. So what do we have left to do? Like I said, we're moving this into production and our goal for this year was to have just basic management build new capabilities. Early next year we'll introduce that build new capability. What we're not gonna do, at least right now is we're not gonna suck in and rebuild every VM we've ever built under VMware because that's just too much work. So we're gonna be doing new deployment testing. But what do we need to do? We need to do at scale testing. So we're pretty big. We need to build thousands at a time. We haven't done those thousands at a time. People who've used vCenter knows that you can do eight things at once usually. And if you're building a hundred VMs and you only do eight at once, then it could kind of take a while to do things. So we need to figure out how to scale those tens at a time, 200s or 1000s at a time across multiple vCenters. We're working with VMware to figure out how to do load testing on the drivers for those things. Almost everything that we had a problem with is fixed in Havana now and contributed back. We're grisly in the environment that we're running this in. So we have had to back port some of the Nova fixes from Havana back in. It wasn't too many, but enough to just fix a couple of the bugs like the multiple cluster thing. Multi-datastore enumeration is an interesting issue and I actually have to talk to them to see if they fixed it. But effectively, when you deploy a VM, it always picks the first datastore. So if you have 50 datastores, like the way our environment got set up is we had 52 terabyte datastores because that was most efficient at the time. It would be much better to have 100 terabyte datastore, honestly. But because we have 52 terabyte datastores and I can't delete them all, I have a problem right now where it always tries to use that very first datastore and then I have to sort of move things around, which is not real good. And then we're also working with VCE. I don't know if there's any VCE people in-house, but VCE is actually certifying a Vblock as an open stack Vblock. I think they announced it a couple of months ago. And so we will be able to call them as our integrated provider for help with open stack on Vblock. They've got this thing that they announced, I guess it's called intelligent operations. And basically, open stack, we'll talk to that for monitoring because it's sort of a converged watcher of that infrastructure. I know a lot of the other converged vendors, Hitachi, the NetApp hook in with VMware, a couple of the other ones, UCS, all have this little monitoring thing on top and those integrations up into the open stack monitoring is coming as well. So let's see, I think that's it. Reading materials that I used here, it's interesting it'll be in the deck if anybody wants it. Canon, Canon, Canon, both have contributed a lot in this space and VMware has one of the leading contributors now to make sure that VMware is a leading supported hypervisor. Everybody saw the canonical announcement that they are sort of the preferred open stack framework now to manage VMware. And please come use PayPal and if I have some time, I'm happy to answer any questions that anybody has. Yes. They reside in a Nova compute VM. It's exactly right. So we built a very small, a two core VM running KVM that we turned into a Nova compute node and then we made this Nova config on that compute node because I only have one of those. I want that to be controlled by vCenter with DRS and vMotion. So I'll let that compute node manage that cluster from within the cluster. Right, so FT, I would solve the problem with FT with VMware FT fault tolerance. That would be why I'm not concerned. If I didn't have that, yes, I would have the one-to-one mapping concern. But remember the only thing you lose is manageability. So if that VM dies, you can just reboot it, have the auto-reboot feature set in VMware and then that VM will come back in a couple of minutes unless it's totally broke. But you don't really lose anything in the meantime. The vCenter. So if we take a look at that compute model that we deploy where we have 96 compute nodes in one rack, if you can imagine having 96 standalone ESX-5-1s which need 96 Nova compute nodes to manage the 96 hypervisors and if you have five racks of that pretty soon you have 450 more Nova compute nodes. Just to keep track of those, it gets out of control too quick. Correct, we have some of the ESX drivers in the lab. We are not gonna do it in production. We're always gonna use vCenter. I can't answer that question if it's gonna remain or not. I know VMware is working on the vCenter driver. So it's there, right? Who Citrix wrote it originally, I guess. So it probably will live for people who need it. Okay, is Nasir required is the question. It is in our case because we're quantum based. If you're Nova flat DHCP, then you can allocate, yeah, if you're quantum, yes. Quantum, neutron, yes. If you're not, then no. At least in my experience. Do the application team decides if they're KVM or ESX? They don't actually. We separate it by environment effectively right now. Nobody really gets a choice to be stateful anymore, honestly. Unless you're really important. Everybody is required to be stateless. So we tell them to be stateless. So which department decides? So the team that I'm on is, what do we call it? Platform engineering and operations, infrastructure. And so my role is compute owner and I'm kind of the hypervisor owner with four other people in one of my other teammates in the room here. And I guess it's our responsibility to come up with the data to make the decisions on which is appropriate. At the beginning though, when we declared let's go open stack, we also declared that we would use whatever the common industry things were at that point, which were KVM, ESX fulsome. And so all net new stuff stateless was just gonna be that because it was free, right? That tagline that I talked about. And so there really wasn't a debate right there for that. But now when we talk about mission critical workloads, the compute team, along with the infrastructure architecture team has debates over whether you need three nines, four nines, five nines. So we kind of do that model. Does the hard, if the hardware gives us three nines, does your application have to be five nines? Or for instance, do you have to have the five nines infrastructure for your application? And that kind of decides it at this point. The quality of service. We're red hat shop today, yeah. Did we look at OVF as an exchange format? I'm gonna probably say no because our very early tries, I guess we didn't build new OVFs. We tried to convert existing VMDKs to consumable OVFs and that failed miserably with the crazy way we built those VMDKs. So we kind of gave up and for sake of speed went pure VMDK. So when PayPal made the open stack decision at the CTO level at PayPal, we decided that it was very important for PayPal and eBay as an entity to be an engineering leader in the open stack community. We wanted to have our name out in the industry. We wanted to be open stack and we wanted to show that we could take this tool manipulated to a huge Fortune 500 company. No, we did not say we're gonna get rid of VMware. We said we're gonna take this product and make it work and be agile with it. Like I said before, there's a lot of really important uses for VMware inside of PayPal and eBay. Core infrastructure, everybody knows there's a lot of things that come with VMware OVFs and that's the only place you can deploy it. Some people are getting around to KVM now, but I can tell you that that was not one of the criteria out of the box. Let's get rid of VMware. We brought in something that was open, that we could be leaders in, that we could go fast with. Anybody else? Please find me anytime if you have any other questions. Otherwise, thank you very much. I appreciate the time.