 Okay, well, let's get started. Thanks for joining us. We may have some more people come in. It's a bit of a walk So thanks for joining us this afternoon. I'm Toby Owen, and this is Tim Bell We'll be presenting to you today And the talk originally was hybrid open stack clouds and and we thought we should probably rename it After the fact towards hybrid open stack clouds because this is really the beginning of of our project together And I'll we'll get into this in a minute, but start by Introducing the folks involved in this this particular project. I work for Rackspace and product strategy And I'm based in London. This is Tim Bell works at CERN based in Geneva And the third gentleman there is Merrick Dennis, who's our distinguished fellow developer working at CERN on Cloud Federation Is Merrick here? There he is so And the obligatory legal slide So let me walk through the agenda real quick and then I'm going to turn it over to to Tim to introduce CERN and their use case But so we'll start with introducing what CERN is as an organization What they're doing to evolve their IT infrastructure from a grid computing model into a cloud computing model based on open stack We'll talk a little bit about open labs Which is the project that we're engaged in together what that is and what we're doing We'll talk through kind of the use case for federating federating open stack clouds Get into some details of what we've done so far in the project and then sort of a glance ahead at what we'll be doing over the next year Tim this one yeah great Just out of interest how many of you know what goes on at CERN Okay, that's good. So I'll go through these parts relatively quickly just filling in some background for the guys who don't know the details so We're the guys who are basically running the largest piece of scientific equipment in the world It's a 27 kilometer tunnel and an accelerator Buried a hundred meters underground near the Alps in Geneva Our basic goal is to understand how the universe works and what's it made of and we do this by sending around beams of particles And colliding them together The beams themselves have the energy of a high-speed train So if you imagine doing this about 11,000 times a second Cliding them together. That's a lot of train wrecks that we have to analyze The analysis takes place using cathedral size cameras these things Hundreds of megapixel cameras They record the collisions and then based off that we understand what has been the fundamental particles that are involved For those of you that have followed physics news recently based on the research work We did professor Higgs and Engelbert collected the Nobel Prize a few weeks ago And this is the work that helped them do that So to focus on some of the aspects that we come to around the cloud work I need to explain a little bit about how we actually work on the data The detectors themselves record about one petabyte a second So this is serious amounts of data that need to be filtered down At the first level we do that with a set of hardware and then after that the data itself is sent to large farms of servers These are generally around one thousand to one thousand five hundred standard Intel based CPUs and Their job is to look at the collisions themselves and try and work out which ones are interesting and which ones are not At the moment we're in a situation where we're upgrading the the accelerator We ran for the first four years from 2009 to 2013 We now have two years during which we warm up the accelerator to room temperature from minus 271 degrees Kelvin and Now moving on to a situation where in 18 months in 2015. We will then be turning it back on again So what do we get at the end of that? We have a large data problem around 110 petabytes at the moment We're adding about 35 petabytes a year probably going up to about 50 petabytes a year when we turn the accelerator back on again The physicists want us to keep this data for about 20 years minimum How do we do it large piles of tapes currently about 46,000 160 tape drives, so it's a pretty intensive data management problem However associated with that is then the problem. What do you do with all of this data? And this is where when we came to start to analyze the data and work out the Architecture under which we could be analyzing this volume of data this complexity of data At the time the technologies that were available were grid technologies based on the Globus toolkits These allow us to construct a hierarchical system of data centers So at the center, there's the tier zero center at CERN This is where we receive the data from the accelerator and we make sure there's a copy that's stored and kept We then send that data around to 11 other centers So a minimum the data is one copy at CERN one copy the other 11 centers and Those 11 centers then communicate out to a hundred other Organizations universities research labs around the world So as far as a grid structure goes it was very hierarchical Clear roles the tier zero large quantities of storage the tier ones some local tape storage and the tier twos No local storage at all equally the workloads were distributed So that the tier twos were used mainly for simulation So that's trying to work out given theory What should collisions look like and end user analysis where you take a small amount of data and work out? Does it match the theory? So as an environment this has provided us very very useful computing capacity for the past four years that we've been running the LHC However, it became increasingly clear that we needed a more flexible model as we go forward The structure of software around this hierarchical distribution of data involves a lot of code and much of that is unique to the research and the high-end physics environments This is not something the longer term is a sustainable model So we need to be sure we're in a mode where we are following the standards the industry is defining Because in the end that's the sustainable direction to be working in So then we have the challenge and this is modeled on the the head of the worldwide computing grader Ian bird So grids in themselves. It's a distributed system with a single form of credentials We use the x509 certificate standard to validate a user and with that then the users able to access the resources across the whole grid They have associated with themselves a concept of an organization which defines which resources they should be allocated to and equally the priority of the work they're doing Users find x509 certificates extremely hard work to manage they find the using of the grid interfaces also to be somewhat confusing And at the same time the users are starting to look towards cloud computing which they're encountering their everyday lives as Being an alternative and simpler model under which we could be doing this work However Clouds themselves provide you a data center concept. So it provides you with an endpoint There is no structure to them to allow those date those clouds to be connected and structured together So in moving from a grid to a cloud you lose that structure and that ability to share The two technologies are actually very complimentary at the moment at CERN We're rolling out the open stack cloud for the use in the tier zero And in this case we're actually deploying a fair amount of the resources to run grid computing on top of the cloud So we take the legacy grid software we instantiate virtual machines, and we run those virtual machines on the cloud At such time as users want cloud resources Then we can reduce the amount of grid resources and allocate them more cloud resources So using a cloud as the underlying technology gives us a lot of flexibility At the same time when we look at the computing models, we're using a model here from cloud scaling Randy bias is always a great guy to go to when you need a picture and a clear explanation So what we have is a situation where at the top level we have what we will call the classical Parallel programings MPI low latency connections CERN doesn't actually have that much a technology in this area We use largely standard Ethernet connection between our servers We have a few programs such as the ones in engineering that need to analyze the detector Geometries and the large-scale engineering stresses and strains airflow But the majority of our computing models is actually down in the high throughput high scalability computing If you brown mind we've got 11,000 collisions a second It's very easy to take the first collision and give it to one virtual machine take the second collision Give it to another virtual machine. We are embarrassingly parallel in terms of our model So that allows us to be considering different approaches to how we can do computing So where we come across situations like this situations where we can see technology changing We have a framework under which we can collaborate with industry and investigate these areas So the CERN open lab is an ability for various organizations commercial organizations to work with CERN and use the extreme computing challenges of the Large Hadron Collider To then work on what should be the future computing technologies to address the CERN use cases So example companies are Oracle Large-scale databases Intel process technology HP on the networking Siemens on control and We have a new member recently Rackspace to work with us on this area that I'm describing here So the aim behind this is getting to a virtuous cycle. So basically we take an extreme computing challenge We then apply that and do research into what are the computing technologies that we need in order to solve that problem With that we then take solutions in this area And we throw them at the other end of this high-speed train of one petabyte a second of data and see how they cope And eventually that comes around into the standard product streams that we can then take advantage of in large-scale production So as part of this work early on in this year, we started to use the Rackspace public cloud Here we were taking simulation workflowed This is trying to simulate the universe rather than analyzing the results of the experiment And we ran that through in the Rackspace public cloud the benefit of this work is its high CPU Low-disk IO and in particular very low network IO So this is exactly the kind of workload where when the CERN computer center starts to get full We can be looking to take this workload and move it out to a public cloud And then allocate the CERN computing resources where we need the high IO throughput to the data These tests were very successful. We ran CERN standard workloads in a very short time getting them up and running with good reliability So it showed that we can use clouds outside of the CERN main one Then as we started to reflect on where this could go we then said well now if we take this as a further extension We've got the CERN private cloud currently about 22,000 cores going up to about 300,000 cores in the next 18 months We have the Rackspace public clouds So those we could potentially be taking advantage of if we can find an easy way for the visitors to move their workload between the two sites At the same time you might remember those large computing farms that are sitting right next door to the accelerator Since we're doing the upgrade at the moment. Those farms are not being used In fact one of the problems for one of them is that they have a condensation problem So they need to be leaving the machines on in order to avoid an excessive level of humidity in the servers So why not use them for something productive So there are two large trigger farms in the Atlas experiment and the CMS experiment Each one heading towards 10,000 20,000 cores So what the guys have done is they've spun open stacked clouds on those now the CERN IT department the central IT department doesn't manage it It's managed by the experiments So we've got independent clouds sharing the same technologies Meanwhile other sites High-end physics computing sites are also starting to deploy open stack. So I am 2p3 in Leon the Brookhaven National Laboratory outside New York the Nectar Center in Australia and Many others for example, I've just heard that the IHEP Center in Beijing's deploying the open stack cloud too Now what this means is we've got a large pool of resources there that's currently a set of islands and That's when we asked ourselves a question of is there a way under which we can get an easier Sario for the physicists to be using these resources Rather than the current structure of multiple separate islands That's point of passives Toby so this is very recently actually we officially started our our open lab project with CERN Kicked off the 1st of October so just a month ago And we do have a full-time developer who I introduced at the beginning Merrick So he he's working full-time on the project that I'm about to describe Entirely in the open-set community to help with the problem that that Tim just described. How do we take all of these? similar tech Technically-based clouds, right, but they're operating independently and start to make them a Viable replacement for the grid technology that they're using today So we defined a success criteria for this which is can we demonstrate? Federated identity and then additionally some aggregated services across these clouds and we're going to start as a target a Private cloud that we've installed on on some of the equipment at the CERN data center Connecting to the rack space public cloud both are open-stack based Or potentially a different cloud to look at it graphically These are the potential use cases, right? We've got a public cloud running open-stack we've got the the test open-stack private cloud that we've built for this project and we've got Actually several open-stack clouds that CERN has built and managed and really any of these permutations Could be a valid use case So some of the goals we have For this this year of research one is to develop a reference architecture for how we can federate open-stack clouds Along the way and actually some of this work is already started. I'll get to that in a minute Hope to contribute lots of blueprints and code as we progress through this To the community as well as outputting Talks like this. This is sort of a very early status update So we don't have a lot of progress to report But we hope over time that there'll be more significant contributions And then more sort of technical white paper so others can build on the work that we're doing here together So how are we doing this? Well, I mentioned we've we've deployed a Small private cloud. It's just a test bed for us to try things out try new code out without Getting too much into the the production environment that CERN is still actively running In in their data centers in Geneva We're going to investigate cloud Federation and Federation I Guess for the purpose of this talk. We're going to talk about areas such as authentication Image management networking and metering are just some of them right that Federation is one of those words sort of like cloud you can you can put your own definition on it So we took a broad look at what are the the use cases? What are the benefits of having these clouds work together and then what services need to to interoperate to meet that? And then our aim actually By this time next year, which we just learned is going to be in Paris So not too far from from CERN's home base in Geneva would be to demonstrate Some sort of bursting workload in a federated fashion where open stack is managing the Federation not Sort of homegrown application based code to be able to burst a workload from a private base cloud to a public cloud So why are we doing this now? right So a little bit about my background at rack space is that I helped evolve a hybrid cloud that that we developed which is really a combination of Dedicated or bare metal servers connecting to public cloud And in that we've had a lot of I've had a lot of conversations with customers About hybrid and and what makes sense what we found and what I found is that there are a lot of There's a lot of interest when you talk about sort of multi-cloud Federation But a lot of those use cases are a little bit down the road everyone gets excited when you hear the concept of this But if you ask the question, okay, what are you going to run on this? The question is often not really answered very well today Well with CERN we found a use case that clearly they have the data right they've got the computing needs Essentially Tim told me that any amount of resource we gave them they could run a hundred percent full out For as long as they had it and so here's a great user That can consume this in a pretty significant scale And so this is really a an opportunity to sort of be at that cutting edge of okay. Here's a real-life use case. It's Something that we can we can use to sort of push this forward in the community so we met about a month ago and decided on some priorities, so if we're going to solve Federation and have open stack as a Collection of projects Managed Federation What order do we need to tackle these in and the obvious first choice was was identity, right? If you can't log in with the same credentials to multiple clouds, you can't really do much a Good way to think about this is sort of like using your Facebook account to log into multiple web services, right? Well, it's not that hard if you have two or three accounts across two or three service providers Once you get into dozens it gets pretty hard to manage not to mention that you have multiple services within each of one of those clouds right and so I did identity is really the The starting point and this is sort of the the story that we've written to define this as a user I want to take my single set of credentials and be able to access services across multiple clouds We'll get into progress in just a minute, but I want to walk through kind of the top couple of priorities here first The next set is really aggregated services So once we have the ability to take that single set of credentials and access multiple services or multiple clouds with that single login One of the things I need to know about what services are available across all of these clouds I want To authenticate using that single set of credentials and I'd like to get a back and answer of the full set of services available to me To me based on that token based on that identity that I have And and sort of a fast follow here is I'd like to be able to take And build an image and be able to deploy in one place at one time and be able to deploy that across multiple clouds So obviously the identity work that we're starting is being done within the Keystone project service catalog Likewise will be done within within Keystone and image management is going to be done sort of within the Glantz project And that's sort of where we think we'll we hope to get within 12 months Because this is a research oriented project. It's always a good idea to sort of define more More work than you might hope to get to And so future areas that that we took a look at were okay once we have those things solved There's a whole list of things within the compute service from a Federation standpoint that could really stand to benefit And add value there's a lot of needs that come up around usage, right? Usage is complex enough within a single cloud particularly within a service provider context where you have multiple users But now extend that into to multiple multiple clouds or multiple service providers There's a lot more data that needs to be aggregated sifted through things like that and then this concept of Taking now that I've got sort of my workload management and identity problems solved now How can I add some business logic? or policy-driven Workload management into a system right so this is the idea of a rules engine That can talk to this Federation service potentially a Federation service layer But the idea of workload management that's that's a part of this this Federation So let's let's jump into sort of how far we've gotten like I mentioned we've been doing this for about a month and and four days Since our developers started so we have gotten the the the private cloud Test environment stood up in terms of identity We've already had some really good early discussions With some folks listed here from IBM University of Kent and and Red Hat In sort of collaborating on the right ways to do this We've got alignment around what some of the key requirements are we've been through five or six Markdowns on those blueprints In terms of identity We've outlined some initial development work to be done And and we're right at the point of starting to do some of that that code development Around these two initial use cases, right? So the first one is after I authenticate against a local CERN keystone service. I receive a token I can then use it On the racks based private cloud And Then the second one would be Despite having an account at CERN. I want to explicitly authenticate against the racks based private cloud keystone claiming that it's trusted It is a trusted CERN identity provider who can authenticate me, right? So these are we're taking very small steps Towards a fully Interoperating Federation or identity service layer So next steps obviously continue what we're doing with identity In terms of service catalog and images We're starting discussions now about Sort of gathering input and what some ideas are To really figure out what the next steps are and then some some very early thoughts in this It's really been I think a positive thing after four weeks We've made some meaningful progress in terms of community discussion leading up to this conference There's a couple of developer sessions tomorrow, which I've got the details on at the end of the the presentation Where we're going to be discussing Federation within Keystone And I think that shows that there's really kind of the right timing for this with version three of Keystone Coming out and some previous work done in OAuth This really feels like the right time where There's a use case for this and there's folks that are able to contribute and get involved in a meaningful way So Why is Rackspace involved in this right obviously we're committed to open stack But I also just wanted to point out that this is really key to to our mission as a Service provider as well. This is sort of our high-level strategy and this isn't a corporate presentation But I'm using this slide to point out The first two items here Fit very squarely into this project, right? We're about open stack and furthering the adoption of cloud and open stack as a standard We feel like the whole community wins When that happens and that hybrid cloud is really Something where we seek to differentiate as a company The ability to federate multiple open stack clouds obviously plays into that strategy as well, so There's obviously a good commitment from us because this fits strategically with what we do And in terms of hybrid cloud this is Sort of becoming a buzzword you guys probably don't need this slide But but the analysts out there also agree that the time is is right for hybrid cloud, right? So we've got some quotes here from Forrester and Gardner and these aren't specific to Rackspace, but The idea that Users will be using multiple services from clouds. This could be SAS and IS or you know multiple SAS providers It's not just I ask but that this is the reality and this is the reality for some time to come So we feel that if we can really make open stack work together in a meaningful way in a federated way That adds tremendous value To the community to users of open stack So what can you do? You'd like to get involved in the discussion. You're at the right place, right? So There'll be lots more conversations this week. I've got a slide That has the details of some of the Keystone sessions coming up More more will follow with within Glantz and and the Nova services in the months to come I imagine I Just encourage you to be involved in the conversation Sort of the beauty of of having this be an open-source project, right? Everyone can contribute as much as they'd like to you. So we encourage you to do that the more ideas we get The better the the results will be for sure and We do have some time for questions so Fire away, and I'll leave it with this. These are the The sessions that I'm aware of there may be some others that focus in on Keystone in particular and these are the design sessions. So if you are a developer or contributing These are certainly things that that you'd want to potentially attend Yes, so sort of the time to live on the token from a security standpoint So my understanding is that as part of this work that's currently being discussed Then the time to live is one of the factors that are included in that So that yes, there is a risk. There is equally a risk in the non federated scenarios Also, and you ought to be able to tune that risk according to who you're working with The question was what other clouds would we integrate between CERN and rack space? Really, we're trying to to demonstrate Interoperability between any number of those permutations. So Currently what we have access to are The the open stack cloud that CERN Operates the one that we built there that that we sort of managed for them and then the rack space public cloud But equally there would be additional open-stat clouds that are managed by some of those tier one and tier two providers within the CERN network and Ideally if this becomes sort of part of future versions of Keystone and other services This would would work anyone anywhere that that opensack is wrong So this is the beauty of contributing upstream Within the open blueprint process, which is that as this goes up and then trickles down Then this will become available for any cloud provider using open stack who wish to take advantage of this So amongst that all the other high geophysics sites, for example, that are currently collaborating with us in the back So I think it is an important distinction between the open lab environment, which is a research collaboration between companies and CERN and CERN's public procurement model Which is a completely separated discussion? So CERN and rack space found we had common interests in this area and open lab provides a framework under which We explore those common interests However, at such time as CERN would be going out to for example look at public cloud procurement Then that will be done through the CERN procurement rules Which would follow the standard rules of public organizations and competitive tender So in this case rack space and CERN found such commonalities that we agreed to get going with the research project Hello So in SNEA or the SDC there was a cloud federation related presentation Are you also looking something similar with regards to maybe Swift and the objects store a federation Which would move objects from one cloud to another I think given that we've got the one year It's already quite ambitious to do the things that we plan on doing I think what it's useful in going through the Keystone structure first is that by doing that We potentially enable people with other use cases to take advantage of these small steps that we're doing And as I say with it being something being done through the open blueprint process Then as with other things in the community people can identify things that we haven't got within our plan of work And potentially contribute those also to the other projects Thanks So there's nothing here that would preclude you doing it with Swift But currently if we look at what we have is primarily focused on Keystone followed by Glance and a potentially followed by Compute and metering if we get that So initially when the grid model was conceived the thought was that the network would actually be the biggest problem network reliability network limitations in reality, it's been probably one of the most stable and open environments Issues such as the effort it requires to run a site software maintenance If you can imagine deploying a new version of software across 200 sites involves a lot of conference calls and So these things are becoming more of the major issues around growing which means moving to a model of independent sites Collaborating rather than a hierarchical structure is looking a lot more attractive in terms of the long-term sustainability So I think we're flattening the hierarchy in those terms even within the grid environment. It is already flattening somewhat together in terms of the authentication model changing I think from from CERN's use case it would it would be taking the active directory environment that they already manage and having that propagate I Imagine this propagates already through the grid system. Yeah, so it's it's From the user experience it wouldn't necessarily change the software that's managing that that identity Which would over time migrate to open stack and it's probably worth noting that more and really what? I think Tim had a really good point And as part of the talk about how The grid system and the cloud system aren't necessarily competitive. They are fairly complementary I think the goal one of the goals of this that it represents for CERN is the ability to to step back more and more over time from managing all that code themselves to more of a standard open stack based solution and so as we start to Enhance some of these features around Federation with an open stack they can start to peel those back from the grid system, right? So starting with authentication, but I talked about this idea of sort of workload intelligent workload management Well, that's something that's managed on an experiment by experiment basis within those portals and frameworks And so developing an open system to do that would would allow CERN to focus a lot Fewer resources on managing Individual experiment code and turn that over to to open stack to do so that's really kind of a higher level Explanation of how this evolves over time. Yes, right Okay, so in terms of how would the images be kept consistent I think that one of the aspects of doing these small steps that we're doing at the moment which is working through the identity federation questions and going on to the image Side of things is that the aim will be to then be floating some ideas of how this could be done for the image management It's not clear for example if its best thing to do is to have The replication of images across the various different glances or if the best thing to do is to be replicating pointers to To a remote glance. This is the kind of thing that we want to be working through with the community to work out the alternative approaches So I think what was the second question it was on Right so Right on the keystone side I believe that the model is one largely of a trust structure where you define a combination of attributes associated with an identity and Then a remote cloud can trust Identities that match that But probably if you want details on that they design summit sessions are the ones to come along to I would say from a private cloud to a public cloud Context which is a likely use case here probably the the private cloud identity service would be the trusted source And they would have an account with a public hub provider, but then that credential would then Be used within the public well that the how like like we mentioned the how hasn't quite been solved yet Other than they would both be keystone within this context But how that works we've written the stories for What it should look like, but the how is really what's being worked out right now, so we are at time Thank you for your questions. We'll be here to answer any more, but enjoy the rest of this