 My name is Chiradeep. I work for Citrix and the Cloud Platforms group and I'm here to tell you about Zen-based clouds and how do you get multi-tenancy in Zen-based clouds. I hope most of you have heard of CloudStack or you want to try it out CloudStack and so this is more of a user-level talk rather than a developer-level talk. This is an idea. The idea is to get you acquainted with the different ways we can run multi-tenant loads in on Zen-based clouds, which is the most popular hypervisor for public clouds and I'm going to talk about a little bit about CloudStack just as a refresher and then I'll touch on what are the challenges of multi-tenant IAS and how the hypervisor does and does not solve the issue. Then I'll talk on how we virtualize the network and align with that piece of technology, something called SDN or software-to-py networking. And then I'll talk about a quite a different piece of technology called L3 isolation, which is also used to isolate tenants in the network. Finally, I'll talk about CloudStack's network model and if I have a chance, which I probably won't, I'll talk about CloudStack's native SDN approach. So Apache CloudStack was a product from StartupCloud.com and I was there right from the beginning and before we got acquired by Citrix, we open sourced Cloud.com software as GPL in May 2010 and after that, we got acquired by Citrix in July 2011 and in April 2012, the software was donated to the Apache Software Foundation to incubate as Apache CloudStack. And so the the brand and everything else was donated to Apache Foundation and after incubating for less than a year, we graduated as a top-level project, just like Hadoop or Tomcat or any of those other projects as Apache CloudStack. And it's a fairly mature product. We have had customers running clouds on it since 2009. So it's a good four years old. And it's got just got a lot of deployment, especially after it was open sourced as Apache CloudStack, but including commercial customers of Citrix. So when we start building CloudStack, we were looking at Amazon as the template. So if you look at how Amazon builds this cloud, it takes some commodity servers, acts on networking and some commodity storage. They use the open source Zen Hypervisor. Then they have a proprietary orchestration piece. It's not OpenStack. It's not CloudStack. It's their own orchestration software. They have an API on top of that software, some of which you may be familiar with, the AWS EC2 and S3 APIs. And then when you want to sign up with the credit card, etc., and the billing and everything else, they have their e-commerce platform, which of course they know the nuts and bolts of, which they've been doing for more than a decade. And actually when Cloud.com saw this, we also actually started with the open source as an hypervisor, but not being a hypervisor company, we switched to using commercial Zen server. So how can you build a Zen-based cloud? You can take the same networking servers and storage. It doesn't have to be commodity. It can be world-class Cisco or NetApp or whatever you want. You use the Zen server or XEP as the hypervisor platform. You use CloudStack orchestration software and you can either use the CloudStack API or the AWS API in order to run this cloud. Instead of having an e-commerce platform, you can buy an optional portal from Citrix, which can help you vend or sell the service if you're a public cloud, for example. Right? So that's the high-level overview of Apache CloudStack. So, and then, so that's the software stack. Well, what do you do with the hardware? So you start off with a bunch of hypervisors. For the purpose of this discussion, it's going to be Zen server. You put them in a network and you attach some shared storage to it, and then you can start some VMs on it. But you're probably not going to be limited to a few hypervisors. So you start adding more and you connect those hypervisors to the to the public internet so that the VMs have access to the internet, and then you start adding more and more subnets of hypervisors, kind of, you know, cookie-cutter style, and then you tie it all together with the CloudStack management server cluster, which will give you an admin API and an end-user API, right? And then the CloudStack orchestration software uses MySQL and then it can communicate with the hypervisors to orchestrate the hypervisors. You can communicate with the storage and the network to orchestrate all three parts of the net of the of your data center. And finally, you get a secondary storage, which is meant to be large in scale and not necessarily perform it, but it's meant to store your more of a permanent object of a permanent nature like images and snapshots and ISOs. So let's go through what happens when you start a VM through CloudStack in your data center. So we have some images in the secondary storage. So when the API request comes into CloudStack to start a VM, we pick a free hypervisor and then we tell the we orchestrate the movement of that image from secondary storage to the appropriate shared primary storage and then we tell the hypervisor to start the VM based on that image. And then you can do all the stuff like you can take a snapshot of that image and then start more VMs and then finally you want to be able to talk between those two VMs. And so we will orchestrate the network as well so that those two VMs have visibility into each other. So diving a little bit more deeper into what multi-tenancy from a network perspective means. Let's say you have these two racks of servers, Rack 1 and Rack 2 and you have some Zen hypervisors there and tenant Joe comes in and starts a few VMs and and because depending on the occupancy of these hypervisors, you could start them anywhere in the on these hypervisors, it may not be in the same barcash domain. And tenant David comes along and starts a few more VMs and they could be on the same hypervisor that Joe is on and Joe starts a few more VMs and then Chris comes along and starts his own VMs as well. And now the challenge is of course that now you need to isolate at network level between these tenants. And not only that, now these VMs require services at the edge of the internet and so they want services like load balancing, they want firewalls, maybe VPNs and so at the same time these firewalls and VPNs have to be isolated from each other so that tenant A's traffic doesn't go into tenant B's traffic. And diving even deeper, Joe might actually want to run a regular multi-tenant sorry, a multi-tier application. So he's got a web tier and then which is isolated from the app tier and then which is isolated from the DB tier. And now you want to write rules saying that web tier can access the app tier but not the database tier and then the app tier can access the database tier but not the web tier. And not just that, he wants to be able to load balance the web tier, maybe connect back through MPLS back into his premises. Or maybe use an IPsec or SSL VPN back to his his data center. And if you look at the sum total of all the services that need to be orchestrated, it's not just the load balancer and the VPN but you need to do DHCP, DNS, some routing, some ACLs, do a little bit of NAT, power port forwarding and firewalling. So this is what Apache Cloud Stack will take care of for you. So multiple tenants, Joe's and David's and Chris's could come in and stand up these multi-tier web applications on shared infrastructure and then operate their production or development workloads. So what are our options here? So the first thing you could do is each network or tier could be isolated at L2, which is at the Ethernet level. So then each network would be a separate subnet. But it's also desirable, for example, that each tenant gets their own space and those spaces, under space and those other spaces can overlap. Right? So if Joe has 1011 slash 24, then David might want exactly the same thing and we should be able to support that. And in this scenario, it is a zero, for example, that all the web VMs are able to see each other and at an L2 adjacency level. And depending on the application needs, they may want to do multi-cast, broadcast and for example, that they want to do an ARP resolution or a DNS-SD resolution. They should be able to do that. Then there's another isolation method called L3 isolation and what you do at L3 isolation is that you even though there are multiple tenants and the same hypervisor, you write rules on the hypervisor on DOM zero so that those VMs are not able to see each other except when explicitly allowed by a rule programmed through the CloudStack API. In this case, there's no L2 adjacency, no multi-cast, no broadcast and but it is a very scalable isolation method and I'll explain. It's a little confusing, but I'll explain that in a later slide. Last but not least, CloudStack will also support PVLAN, which is like a poor man's VLAN or to scale beyond a 4,000 VLANs. So what happens here is that multiple tenants are placed on the same VLAN, but then you write rules into the open V switch in DOM zero so that they're only able to talk upstream via the router. They're not able to communicate with each other even though they're adjacent on the L2 basis. Again, we don't allow a multi-cast or broadcast and this tends to have very limited use cases because once you cross the router there's no isolation. So when we talk about L2 isolation options in CloudStack, the first thing that jumps to mind is network virtualization and this is the illusion that you're giving to your tenants that they're running on their own exclusive network even though they're on the shared network. So the first thing you do is you the old standby of the VLAN has been there for decades and you can use either open V switch or the bridge on Zen server but then you're limited by the 12-bit identifier for the VLAN so it gives you 4k VLANs and not just that, most switches won't even support 4k VLANs, they'll probably top out at 200 or 400. And then the other downside of VLANs is that they need to be, all the VLANs need to be trunked down to all the high providers because when CloudStack starts its starts a VM in a particular VLAN, it can place that VM on any hypervisor, any available hypervisor and so unless the switches have already been set up to bring that VLAN down to that hypervisor, it's not going to work. So it kind of goes against sensible networking design where you don't really have to should be trunking all VLANs to all the ports. Nevertheless, you know VLANs are there, they're popular and they're understandable, network engineers understand them and so that tends to be the most popular currently way of providing active isolation in Apache CloudStack deployments. The last option is the the emerging option which is overlays and some of them sometimes conflicted with software defined networking. So what you do is that you run these tunnels between the hypervisors, either they're VX-Lanolays or STT or GRE and then when the Ethernet frame comes out of the of the VM, you pop it up with a GRE header and then send it through the GRE tunnel. And then you can use for example the key which is a 32-bit identifier in the GRE header as the discriminator between networks of between tenants and what this does require a orchestrator to manage those overlays so that when you start up a VM the orchestrator needs to ensure that the overlay of the tunnel has been established before that VM has started. So when you virtualize the network either using VLANs or overlays, this is what it looks like. So let's say you have one tenant here, he's got four VMs and then either through a physical device or a virtual appliance, he's getting some network services like in our DHCP far-walling and NAT. And then he's got you know a few public addresses assigned to him or her. And now he decides that he wants you know load balancing and VPN. So maybe we spin up more virtual appliances or bring in more physical devices into the picture. And you can see the addressing scheme is 10-1-1 slash 24, but now tenant B comes in, he wants a the same address space but a different set of services. So and he gets a different set of public IPs. And so this is what network virtualization looks like when it's orchestrated by Cloud Stack. And so when Cloud Stack implements this it has a view of the physical infrastructure and then by orchestrating the V-switches and the other infrastructure in the Cloud, it provides this virtualization on top of the physical infrastructure. But to give you an example of what happens when when Cloud Stack orchestrates VLANs, here at the bottom you have all the VLANs which are needed trunked into the hypervisor and the hypervisor barns these two nicks. And then Cloud Stack knows for example that tenant A and has VLAN 10 and tenant B has VLAN 20 and tenant C has VLAN 30 and then it programs the V-switches on the bridge so that these VMs are attached to the appropriate ports on the V-switch around the bridges. If you wanted to do an overlay, then what Cloud Stack would do is let's say you have this fairly large data center infrastructure separated by several subnets. Let's say this user one has created four VMs on a network. Cloud Stack would program the open V-switch on each of these hypervisors to create GRE tunnels between these hypervisors so that traffic for these between these VMs are carried on this tunnel. And the discriminator for the tunnel is the GRE key which as I said before is the 32-bit key in the GRE here. And now if user two comes along, then we set up a different tunnel potentially or if he's using the same tunnel he uses a different GRE key. And so that's actually how Cloud Stack will do its native GRE SDN controller but Cloud Stack has an extensible network model and is extensible in several ways but networking is one of its strong spots. And since Cloud Stack went into the Apache Foundation these are all the integrations we've had coming into Cloud Stack. NICERA was the first and followed by Midokura, Nuage, Big Switch, Stratosphere up out in Japan and we just got notified that Juniper wants to donate an integration with Contrail and soon we'll start on an open daylight integration as well. So all these technologies, SDN technologies support network virtualization in some form or the other and then they will integrate with Apache Cloud Stack as well. So coming back to the second mode where you don't virtualize the network but you're trying to use the existing address, the physical address space that you already have. In this example, tenant one has come in with two VMs and they've landed up in the same rack and so they have adjacent IP addresses or addresses in the same subnet. But as these starts new VMs, tenant two could come in into the same network and tenant one as it starts the new VM goes into a separate subnet altogether. So you can see that these tenants do not have or adjacent or L2 adjacent IP addresses and so what happens is that Cloud Stack needs to make sure that for instance that even though tenant two and tenant one are L2 adjacent, they can't really see each other. So for this purpose, Cloud Stack programs the IP tables in DOM zero to prevent this from happening and by default all the traffic is dropped unless the tenant writes an API rule saying that traffic is allowed between for example, tenant one VM one and tenant one VM three. And as these VMs get created and destroyed, Cloud Stack is busy updating the rules on these IP tables. And so to do this, you need the packages, the IP tables across the bridge because this does not work with OBS. The main reason being OBS does not provide a stateful firewall that is required to provide such a service. It also requires an IP set package because as you start writing lots of these rules, IP table will start bogging down and so the IP set forms a is a very nice optimization package for IP tables. It does not work with OBS but then and the advantage over VLAN is that it's in production that has been shown to scale to tens of thousands of VMs and tenants. So Cloud Stack as a tour I'm giving you, you can see that Cloud Stack needs to deal with a bunch of different services for different tenants all the way from L2 all the way through L7 and then operators of these clouds they want to use their own favorite way of isolation or providing these services whether the virtual appliances, hardware devices, SDN controllers, what have you. And some of them don't want isolation, some of them want VLAN isolation, some of them want overlays, some of them want L3 isolation. So it's Cloud Stack's job to abstract this complexity away from the end user because end user all he wants to do is create a network. And so what we do is we expose a service catalog which the cloud operator designs through the admin API. So for example let's say designs a cloud a gold service offering which says this you get whenever you create a network with this offering you get a load bouncer, you get a firewall and they all just use a virtual appliances. But if you want to upgrade then you can get the same things and a VPN using hardware appliances, right? And so here's an example so the customer's running a Dev environment, he's got a virtual appliance providing a low performance experience but he's got a bunch of useful services nonetheless in his network. And when he runs into production he wants to use hardware devices like the SRX firewall of the NetSql load bouncer and then Cloud Stack will let him transparently upgrade to that configuration without changing his VMs. And that was it. There's more information please come to the Cloud Stack Wiki, Cloud Stack Hizalist and contribute. Thanks. Any questions? There's some work going on I know in dense server at the moment looking at doing security group stuff with OVS. Can you comment on that and how that fits into the segmentation of the network? Certainly, yeah. So as it turns out security groups, it turns out to be useful even when you have virtualization, network virtualization. It's not just L3 isolation because it pushes the hardware or the firewall down to where it belongs next to the VM. And so that'll be a very useful addition to OVS. The way for example KVM does it today is they let you do IP tables and OVS at the same time. So you do OVS to do the switching and then at the port you could write IP table rules. So it gives you the best of both worlds. Maybe not performance but at least it gives you some level of integration. I believe that's done by gluing a bridge and an OVS together and traffic passing of the both. Yeah, some would call it a Franken switch but... Yeah, hopefully the thing with the send server is going to be to try and put the right rules in to achieve the segregation of the MAC addresses. I'm sure we do that level. So it can just be done in one pass very quickly. I think what we're trying to achieve is... Okay, that'll be excellent. Yeah. Okay, we got time for one last question. Anyone have a question? Let's give Chira a deep hand. Thank you.