 good morning. So I'd like to give a talk to you today to describe the kind of work we're doing on a hybrid cloud framework at PES University. So before I start let me thank EMC which has partially funded some of this work. So this is you know the reason why you might have hybrid clouds I'm sure this is very familiar to most of you and that is you know you can use that for cloud bursting for migrating workload from one cloud to another if that's necessary and of course this supports scalability and high availability. So this is the high-level architecture we're looking at. We actually the work we've done which I'm going to talk about today is federating OpenStack with Amazon. So you've also done work to federate OpenStack and OpenStack but today I'll talk about what we've done to federate with Amazon. So this is the way we have implemented it. We've leveraged the cells architecture and in the cells architecture each there are a number of cells. I think OpenStack doesn't define what a cell is it's a collection of hardware it could be you know a data center a cell could be a rack you can set up the cells whichever way you want and they can also be hierarchical. So at the top level there's a top cell which has the cell scheduler and that decides in which cell the request will actually be executed and then below that there is there are child cells. So in a normal OpenStack cloud what happens is let's suppose you have an OpenStack cloud with four data centers and you create each of the data centers you make each one of them a cell. So then you'll have a top cell and four child cells underneath. So what we have done we have created a new kind of cell the extended architecture to have a new kind of cell called a pseudo cell and this maps so any foreign cloud like Amazon and so on gets mapped into a pseudo cell and since the child the cell architecture is general and it's hierarchical within Amazon also you can have a hierarchical mapping if you wish and I think the advantages of this kind of an architecture will become clearer when I continue. So this is you know the way that this has been deployed in the bottom left hand corner you see three cells the top cells in the private cloud cell which represents the OpenStack and then there's a pseudo cell which represents Amazon so the bottom three cells make up the hybrid cloud and then this is connected to a VPN node across the internet to Amazon which has a VPN on the other side. So again the implementation details so our we are looking at OpenStack at the primary cloud Amazon at the foreign cloud and leveraging the cells architecture and the way we've implemented the pseudo cell in a high in a kind of high higher level so the pseudo cell is a data structure which contains statistics like you know the utilization of the cell and so on so forth and it also drives which can perform operations on on the machines in the pseudo cell and there's also communication between the pseudo cell and the top level cell in that the pseudo cell periodically updates the statistics the top level cell. So what we have done is we've rewritten these pseudo cell drivers so that for example the pseudo cell driver that gather statistics about Amazon and it gather statistics and it also can spawn VMs on Amazon and things like that. So one of the questions is how do we handle security right now? So currently we are not this is not integrated with the federated security work that is going on. We are using the EC2 APIs for authentication and so we store the EC2 credentials and we're using that in our driver but in future we definitely do plan to integrate with the federated security APIs. So this talks about you know the flow of the work so you have the pseudo child cell at the bottom and at the top cell you have the interactions as I already mentioned the pseudo child cell updates the database and then makes cause the interface to change statistics and so on so forth. So this represents the way a typical workflow occurs. So let me mention so for example suppose there's a request to create a VM. So what happens that the request to create a VM it arrives at the top cell and the top cell looks among all of its child cells to decide it looks at statistics about the child cell and the capability of the child cell. For example if you want a GPU or something it looks at all those capabilities and then it selects a target cell to create the VM in. So in our case what would happen is that the top cell would look at the cells underneath it which is the open stack cloud that's those are regular cells and the pseudo child cell which is Amazon and then it would make a decision about which one to which cell to create the VM in and if it decides to create the VM in the regular child cell which represents the local open stack then the flow is as it normally is it goes to child cell and then the child cell is like the physical server and then it will call the Nova compute agent and so on so forth and create the VMs. On the other hand if it selects a pseudo child cell which is represents Amazon then it will be the request will be sent to the pseudo child cell which will then use the Amazon drivers to create a VM in Amazon and once it creates the VM in Amazon part of the pseudo child cell API is to return the end points the parent cell so you can talk to the VM and so on so forth. So I think this is a diagram which shows how simple the flow is and I think it also illustrates many of the advantages for approach. For example if you take a look at it we didn't have to invent a scheduler which will decide whether to create a VM on Amazon or a VM in open stack. We used already because we are mapping Amazon into a child cell over here we are already got a scheduler the cell scheduler which does the job. So another question is that suppose in a later date you might want to make a policy-based decision about which is based on statistics like for example how many VMs you have in Amazon etc etc and if we had taken a different approach then you might have to invent new infrastructure to map all of this but what happens is as I said in one of the earlier slides there is already an interface between the child cell and the parent cell whereby the child cell periodically keeps on updating its statistics to the parent cell and so we simply just follow that protocol and so basically the advantage one of the major advantages of our architecture is that we are leveraging the code that already exists in open stack. We are not having to invent a whole new bunch of code for scheduling between open stack and you know Amazon or for you know periodically updating statistics for example if you want to monitor information about the child cell monitor information about the another example I can give is as you know NOVA keeps tracking this database of every VM that's created. Now because this Amazon is mapped into a child cell a pseudo child cell the whenever the child cell creates a VM it automatically passes the information about the VM to NOVA so NOVA automatically updates its own database with the new VM so again we didn't have to write new code for that and the advantage is I think the major advantage of our approach is that it leverages the code that's already there in open stack to simply extend the federation concept. Yeah so the proposed architecture is simple and the good thing is if you look at many of the clouds which already exist over there they all are all hierarchical open stack has felt the need for this now but there's no cloud out there which treats the entire cloud as you know just a gigantic collection of machines. In practice there's some organization into data centers, racks, hierarchical organization and we are simply leveraging that so you could extend this to other things as like eucalyptus for example. So future work we want to take this forward as a contribution to open stack we need to integrate federated security I think those are the major major piece of work that we're taking forward. So now I'll play a small video which is a demo of how this actually works in practice. Hello so we'll be showing you a demo of the hybrid cloud using open stack and amazon. Here we see an open stack cloud dashboard and open stack cloud running on one of our servers and this is the console for amazon EC2 which shows you the resources of all from our account being used. So we have set up the hybrid cloud such that as the load on the local cloud keeps increasing the instances are going to be routed onto amazon and a tenant on open stack can use instances even though they're not on amazon as their own instances giving a perfect hybrid cloud situation. So to show that launch the demo we launch instance and give the name is instance one and give the image source as seros and you launch this instance. This instance will take some time to build and schedule and load onto the system it's not point here as yet because the load on the local cloud has still not increased that threshold. Now we launch a second instance by the name of instance two and we again give a image source as seros and launch the instance. This instance will then take some time and it will get launched onto the local hypervisor of the local open stack cloud. So here we have successfully spawned our second instance and we once again check amazon and it's still not spawned on amazon saying that our threshold given on the local cloud is still not reached. Now we launch a third instance launch the instance and we wait for it to get launched or it might be spawned on amazon. So a third instance has been launched on on the local cloud and amazon still continues to have zero instances. We launch a fourth instance and wait for the fourth instance to be launched. So if you notice this one is being spawned on amazon but the instance goes through all the state changes like spawning and so on that is normal for open stack. The fourth instance has been launched and we check amazon and here it shows us that there is one running instance hence from the instance four from local cloud since the threshold has crossed beyond our memory our message has crossed beyond our threshold given the instance was dynamically launched on on amazon and here we see that this instance on the instance which says running has been launched here and we have we have created a virtual private cloud on amazon and created a subnet called hybrid cloud and given the range of ip's as the same as our local cloud that is in the 10.00 range. So hence the ip's also are same as the ip of the local instance four here that is 10.005 has been brought about into amazon and the amazon private ip is 10.005. We have routed the virtual private clouds onto the vpn and hence there is network connectivity between a local cloud running on a server here and the remote amazon cloud. So this any tenant using these instances can use instance four as for the tenant he can use instance four as it's been running on open stack even though it has been spawned on amazon. So that's the demo of us thanks very much. So that's the team of students have worked on it they're undergraduate students. I appreciate if you give them a hand. Thank you. So if there are any questions have you done anything to integrate blockstarch cinder with ebs? So the ebs is accessible only from within ec2. Cinder would be our next target right now what we have done is we have integrated the VMs in and so the equivalent cinder should be possible since it's ip storage. Thank you. Can you tell us more about the network infrastructure did you assume that that was already there is it's that spun up on the fly what are you assuming there? Yeah so for the network infrastructure right now it's very simple and that is again work that needs to be done so right now we assume that there's a subnet which connects the VMs a flat subnet network which connects that and we build that through the VPN on the fly so it's not as configurable as it might be needed. A follow up to that what was the latency that you saw from the VM that was on AWS and the VM that is on the private cloud of OpenStack? Yeah so we haven't measured that actually but I would assume it's fairly high so definitely we'll measure that and try and in the in our next iteration we'll publish that information. Hi very nice I assume there's a mapping between the local flavors and the Amazon instance types? Yes so that's again a good question thanks for asking that and that again is more of the work that we have to do so right now what happens when you ask for a particular flavor of OpenStack then we map it in the code to the closest flavor of Amazon and so it's kind of hard coded in there but obviously there should be some sort of a module which will allow you to configure it because for example right now we take the smallest instance type you can find in Amazon which still satisfies everything that you have but you may be willing to want a smaller instance on Amazon because maybe the larger instance costs more so different users want to trade off differently so that's part of the code which needs to be taken out and then made a separate module and made configurable. Great and as a follow-up to that I assume you have to do the same with images but that's not just a mapping do you have to pre-stage the image the same image in EC2 that you have local and then also map it how does that work? Yes so we've done some work on automatically converting the images but I think in a practical situation since it takes a long time to transmit the images from one cloud to another I think in practice you'll have to have an image on Amazon which is ready I think that's the state of the art now. Okay great thanks thanks. Hi there at the OpenStack consumer level are you exposing the OpenStack APIs or are you assuming an EC2 compatibility API? Sorry at which which level? So this is all done through Horizon and it uses all the OpenStack REST APIs so for example if you wanted to write a script you would just write a script using the same nova APIs that you're familiar with. All right thanks. You spoke about contributions back to the community so which part do you need to contribute back I mean the pseudo-cell part or? Yes the pseudo-cell part yes I think that will be a contribution to nova. Okay this is a nice work I was just curious if you had any issues with you know address management you kind of have a range of addresses in in the VPC that you created and then you've got OpenStack's own IPAM so did you run into any issues there or were you able to get around that easily? I'm not completely familiar with the networking part of it the details so what I believe is we have a network range in EC2 and the IP address range inside and we are somehow managing that but if you please send me an email I will get find out and get back to you on that. Hey what's happening now Trevor Powell from RMS? I had a question with the feedback loop that you have or may not have I'm just curious to know if the instance something happens to the instance on the Amazon side or another cloud how does it reflect back in horizon or in vice versa? Yeah so that's a good question so right now we don't have an alerting mechanism but what happens is that periodically the cells have to report statistics back to the top level cell and then they would know that you know that something has happened to the VM at that point and so but I think Amazon does give an alert if something happens to VM so we have to relay the alert back I'm not sure if you've done that yet I've had to check that. Hi in the context of Federation how are you managing a VPN channel between your source cloud and your target cloud? Managing in what sense? I mean how do you set up a direct link between your OpenStack cloud and your Amazon instance? Amazon entire deployment that you create? Yeah on Amazon there is a VPN infrastructure and we have set up a VPN server inside our OpenStack cloud and those two work together to set up the VPN. Okay but do you have the ability to provision a VPN through your console? No so as I said right now the networking part is to some extent hard coded and so we need to make that more flexible that is future work. Okay thanks. In terms of being deterministic with the VMs that you spin up you show utilizing private cloud initially and then essentially bursting out to a public cloud. Yes. I'd like to hear your thoughts on how you reclaim resources so if I need to spin down VMs and I actually like shut down someone's in my private cloud thinking about the mechanisms to go back out and reclaim them from the public cloud back into the private cloud for money saving. Yes so that's a very interesting question and that relates to the mechanism in OpenStack for doing that. So right now this is done through heat and so it should be it should work seamlessly in the sense that if you if heat spins up and spins down VMs then that will work with the sales scheduler and then some VMs will get spun down. So if you want to specifically spin down the Amazon VMs preferentially then we have to modify the policy. So right now the policy may say spin up the VMs and then it'll just spin up the VMs because as soon as it's an OpenStack cloud and then it says spin down the VM again it assumes it's an OpenStack cloud. So the policy will have to be now made sales specific saying that when you spin up the VMs spin it up beyond a certain level to go to Amazon and then it's spin it down preferentially terminate the Amazon VMs first. So policy has to be modified to make it sales specific. I have two questions. One is how how much how how much have you scaled this and also would this work in a public to public like an OpenStack public cloud like HP or Rackspace to Amazon? As I've answered the second question first it should work on a public to public because we've done it on this on an OpenStack to OpenStack cloud and assuming that Rackspace and HP are both close enough to OpenStack it should work. And sorry what are the first questions if you could sorry oh scale scale. Yes we have done this on our own local cluster of about two or three machines and with Amazon. We obviously need to scale it more and see how it works and that's where you know if anybody's interested in trying out the code we'll put it on Stackforge and people can if they have larger clouds they want to experiment and give us feedback let me welcome. Okay thanks very much.