 Next up, to talk about even bigger thoughts is Tim Bell who runs infrastructure for CERN. So I'm very pleased to bring out Tim Bell. Tell us what CERN is doing. Thanks Mark. Bonjour à tous et toutes. Thanks a lot for having a chance to talk to you about CERN's experiences as we move towards an environment around OpenStack. People often describe this as a journey. Many of the user stories describe it in these terms. And it is really a cultural and technology transformation. So what is CERN? CERN is the centre of open research nuclear. We're an organisation that supports 11,000 physicists from around the world. Another worldwide collaboration taking on difficult problems. And these scientists use the facilities at CERN in order to understand the universe. So this is basic research. What is the universe made of? How does it work? There are, however, spin-offs that come from this. Many of you with smartphones will be using capacitive touch screens invented at CERN in the 1970s. The World Wide Web in the 1990s. But it's not the focus of the work we do. So what do physicists worry about when they wake up in the morning? We had a great event in 2012 in July where two of the major experiments at CERN, CMS and ATLAS, stood up and independently, without talking to each other at all, produced a fundamental particle the Higgs bows on with the same mass. This therefore accounts as a scientific confirmation and it's been described as being the equivalent of landing a man on the moon. At the same time on the personal side, Professor Higgs and Professor Englert, who in the 1960s had come up with these ideas, were able to then go to Stockholm to collect the Nobel Prize in 2013. So this is 50 years between an idea to actually confirming that as a result. However, we're not finished yet. There are some major concerns that we have around the universe, what it's made of, what does the standard models look like to describe how the particles fit together. Amongst the things we're puzzled about is why don't we have more antimatter? So the universe started off with a big bang, lots of energy. We ought to have equal amounts of matter and antimatter. Luckily, we are largely matter. There is some antimatter out there, but it's really very small. So we participate in various experiments such as this one attached onto the International Space Station, which being outside of the Earth's atmosphere can observe antimatter particles coming in from outer space. But there are other problems we're facing. We've lost 95% of the universe. So when we look at the planets and stars and how they move and how the universe expands, we know that the universe should be a certain mass. However, when we actually count the planets and the stars, we see that we've only got 5%. There's something out there, dark matter, dark energy, which has to be present to describe why the cosmos moves as it does. Looking out into future discoveries, there's some really interesting questions about gravity. We can describe three of the other forces very well with the standard model that the Higgs has confirmed as part of the jigsaw, but gravity is a real problem. We suspect there are particles called gravitons, and we suspect that these briefly appear in our part of the universe, the four dimensions that we perceive, but then could potentially be moving into other dimensions. It's like if you're on a tightrope, you saw an ant, and then the ant disappeared as it walked around the other side of the tightrope. So as we move the LHC further on, we hope to be able to discover some of these particles and understand the universe further. So when we're faced with a problem such as this, how do we solve it? The first thing to do is to bring together a large community. And then the second thing to do is to design experiments. The LHC, the Large Hadron Collider, was conceived in the 1980s. At the time, a huge number of technology problems were confronting us. But we chose to set off on a path to build a 27-kilometer ring, 100 meters underground, straddling the border between France and Switzerland, in order that we'd eventually get to a point where we could construct these experiments. If you ever get a chance to go underground, and we had around 80,000 people come over to an open day at CERN in 2013, then what you'd see is you'd see these blue pipes. They are actually surrounded by magnets, and inside there are two one-centimeter tubes. The magnets themselves are cooled down to minus 271 degrees centigrade, so two degrees above absolute zero. And the tubes inside have a vacuum which is 10 times less dense than on the moon. And in those tubes, we send round protons, hydrogen nuclei, in two directions. And at four places around the ring, we cross the beams. When we cross the beams, they're at the four detectors. These detectors can be viewed as digital cameras. The slight difference between this and your standard Instamatic is that these ones are roughly the height of Notre Dame. They weigh about the same as the Eiffel Tower, 7,000 tons. There are 100 megapixel cameras, and they take 40 million pictures a second. That creates, amongst other things, some great pictures. It also creates one petabyte a second of data. So, to handle this, we have massive computer farms, 100 meters underground, filtering this data down to levels that we can record, looking for things that we know are the patterns of the physics that we want to investigate. However, we then have to record this data and analyze it in detail. So how do we do this? In 2014, we've had the CERN Computer Center. We've got around 100 petabytes of data, primarily stored on tape, and we're currently recording our last year of running before we started the upgrade of the LHC to higher energy, around 27 petabytes a year. 11,000 servers, 75,000 disk drives. We've got some people who are pretty busy doing disk drive swapping. Looking out to 2015, we're going to be doubling the accelerator energy. This was certainly the significant increase in data rate. However, as always, we look forward further than that. And when we look at the plans for how we can use the LHC and increase the energy, we're looking at 400 petabytes a year in 2013, sorry, 2023. The compute power is likely to be around 50 times more powerful need than what we have at the moment. So with that prospect, then we clearly have to have a computer environment which is reasonably flexible to be able to address these kind of needs and to be able to perform the computing necessary for the physics. So in the Geneva Data Center, we have a nice data center. It used to have one mainframe, it used to have one cray, and it was great. It's got a raised floor you can walk around underneath. However, when you put in standard industry servers now, you can only put in a small number. People ask, when are we going to fill up the rest of the racks? And we just can't. Six kilowatts per square meter is the maximum that we can call in this environment. For those of you that are interested, actually the center is on Google Street View, so you can go down and wander around. It had about 25,000 people through as tourists last year. So it's actually a tourist attraction. Since the facilities are paid by the people of Europe, it's only fair that they should also be allowed to come along and see what we're using the money for. The center itself, we tend to call the Geneva Data Center, but actually it's in France. So we're just over the border, and when I say just, CERN is International Organization. So the data center and my office are over the border, and I then walk 50 meters over to the restaurant at the other side to get my cup of coffee, which is a Swiss cup of coffee. However, just a center in France isn't all that we're needing in order to address these requirements. Clearly, we are at a point where upgrading that data center would have been a significant investment. So instead, what we looked to do was to expand the computing facilities using other member states' facilities. So we asked the countries contributing to CERN to propose to us a data center location, and Budapest in Hungary was chosen. We have 200 gigabit line connections between the two sites. So clearly, we were rather interested in the discussions around internet tax in Hungary since that would have caused us a significant cost. So the good news is that we've got a new data center. The physicists are pleased because they've got possibility of additional computing resources. The bad news is that in today's economic times, we can't be asking for more staff. Equally, given that we want to scale out the computing, we have to make sure we use the resources we've got to the maximum. We have legacy tools that we wrote 10 years ago that we used to manage the data center. For those of you that are used to looking after 100,000 lines of Perl, it's not something you get up early in the morning for and come into work. And the user expectations are being set by public cloud services. They don't want to fill out service tickets. They don't want to wait weeks while machines are provisioned and cabled up for them. They want to click on the interface, get a cup of coffee, and come back to a working system. So how could internal IT be providing them these kind of facilities? So what we did was to challenge some of the fundamental principles of an organization such as CERN. We're a research organization from the physics point of view, but from a computing point of view, we're actually no longer leading edge. There are many other organizations running at scales beyond the size that we are. So we shouldn't need to do things that are special. We shouldn't be producing custom requirements lists that then mean we have to produce custom solutions. We need to be addressing the staffing question. People are assumed to the situation where you get double computing capacity every 18 months. People, on the other hand, it's actually quite difficult to even maintain the level as technical debt accumulates, and they're therefore needing to maintain as well as advance. At the same time, culturally, we wanted to find open source communities. It's very much in the culture of the organization to be contributing to open source, and we wanted to learn from them and equally be able to contribute back those areas that we felt were of general interest, but above all, use those communities also to execute cultural change within CERN. So what do we do? We sat down for a few weeks, we did a lot of prototyping, and we selected a tool chain around these areas, so puppet for configuration management, Kibana Elastic Search for monitoring, SEF for storage, and then the key part of this was OpenStack using the IDO distribution in order to be able to bring a flexible and agile cloud to our users. So where are we now? We started off with what was pretty much a research project in 2011 with Cactus. For those of you that tried, it was an interesting experience, but it was already clear that the rate of maturity of the software was going to exceed the speed with which we could be getting our organization ready for production. So we started the investment, started doing the tooling necessary, the training necessary, and then in July we went into production with the Grizzly release. By production, I mean when someone creates a virtual machine, we promise we will maintain an environment with that virtual machine in the future. We currently actually have four ice house clouds at CERN, so the main CERN one is actually 75,000 cores, because while I've been away travelling for the last week, the guys have installed another 5,000 cores behind me, and I didn't have time to update the slides. We've got three other instances at CERN, those large computer farms I talked about next door to the accelerator. The accelerators are being upgraded, those farms are idle, so what do the guys do? They've spun up OpenStack, 45,000 additional cores, 100 metres underground, in order to be able to deliver additional simulation capacity for the physicists. Current outlook, we have about 2,000 additional servers on order, and we'll be hitting probably 150,000 cores and total on the site between CERN and Budapest by the first quarter 2015. All code that we've written that is of any interest to the community we have submitted upstream, all the code we feel is not of interest is in GitHub under the CERN Ops Git repository. People are willing to, welcome to browse it and see if it is of interest. So the two areas that we've looked at in detail are NOVA cells. This allows us to scale to the sort of size we're talking about and also to be looking out for challenges come 2023. This allows us to build up small units of OpenStack and assemble them together into what appears to the end user to be a single homogeneous resource. And this allowed us to simplify the end user experience for the end user whilst allowing us to scale out the underlying environment. We have cells in Geneva, cells in Budapest, and we have a set of front-end servers that arrange to direct the work appropriately and schedule it. At the same time, we're facing another set of challenges, which is we're seeing a large number of OpenStack Clouds appearing. Along with the four I talked about at CERN, we're seeing 50, 60 organizations that collaborate with us deploying OpenStack as well. And we're also starting to look at use of public cloud resources outside of CERN. Clearly, if we carry on growing, we need to find ways under which to be satisfying that capacity requirement. So CERN has an organization that allows us to do collaboration with industry, and during one of the summits, we had some discussions with Rackspace where we clearly identified a common interest in solving this problem. So, in October last year, Rackspace joined the OpenLab collaboration, and then with that, we set off to solve this problem. The code for this is not research. It's now in the production release. The server came out with Icehouse, the clients came out with Juno. People can now be deploying federated identity on OpenStack. At the same time, this is clear technology change, but the cultural change in many ways is worse, is a larger problem. We have people who can accelerate very fast, understand rapidly the techniques that can be used. On the other hand, we also have people for whom the applications they're running, some of these techniques don't appear so relevant, and it brings us back to Hook's law, which is the law that says you can expand a spring to a certain level under load, and eventually it deforms. And it is key that while we focus a lot on the progress that you get on the one side of the spring, that you make sure that the tension doesn't get such that you reach that point. So, focus on the new technology, but also keep in mind that there will be some people for whom they need a little bit of persuading. So, how do we go about doing this? We assembled a small team, and it really was of this sort of size. It consisted of a mixture of experienced people who'd been running services before, and a set of new hires. Naturally, CERN has a certain rotation of staff. Many contracts are short-term. These people came in with basic Linux skills, and then leave CERN knowing Puppet, OpenStack, Elasticsearch, Kibana. They don't spend very long before they find new job opportunities. And with this team, we then went through the process of building something that we could show people. The demonstration is an extremely forceful technique. However, a number of people came up with thoughts, and to be clear, these are rather extreme examples of how people react. There were people that felt it was going a little bit too fast. They wanted to take things more slowly. And we have a fixed window. The LHC starts up again in April next year. The transformation had to be done by then. In fact, this week, while we are here, they've just started dismantling the old configuration management system that we were using. At the same time, there were the people whose services were organized in silos. This means they had the budget, the staff for the entire service. Now, to turn around to them and explain that, now, please give us your budget, and here is a quota on the cloud, is a major cultural change. We had the experienced service managers who had been running services for a long period, and were finding that the people who were joining their teams actually had been taught Puppet and OpenStack in university. And they were saying, you're doing it all wrong. You should do it this way. And actually, we had a skills inversion, where rather than you join a sitting next door to the specialists and understanding how to do the tools, we're finding the new people, explaining the more experienced ones the best way to go. And then, finally, we have the people for whom their servers are precious. They have something where what they want is a unique configuration that you can't get from anywhere else. Now, in many cases, these requirements can be justified, and it involves a large amount of consulting and discussion to work out the right balance between implementing consistent environments and implementing specialized configurations that potentially cost elsewhere in the organization. So, when I look back at how science has evolved, Newton wrote a letter to Hook, the guy that wrote the spring theory, where he talked about dwarves standing on the shoulders of giants. And science evolution has been on this basis. Each person makes a small progress, the next person takes over, builds on top of that to get further. Large collaborations, like we have around the LHC, the Atlas Collaboration is around 2,000 people, these communities get together, work on the basis of transparency, meritocracy, and a shared vision. When we look at the World Wide Web, Tim Berners-Lee started off and had a very nice text line browser. He didn't feel that embedded graphics was really something that was needed. Now, if we'd been in a situation where the only browser selected was the one that Tim wanted, we'd probably still have a basic text interface. We might still be ending up on Internet Explorer 5 as the only browser that we could use. However, what we have now is a hugely vibrant ecosystem, which is based on some solid core APIs and allowing healthy competition in other areas. So in summary, thanks to all of you for your efforts, when you do helping on an OpenStack, so when you're contributing code, finding patches in documentation, when you're working through with others on the mailing lists, when you're coming to meet up such as this, giving feedback on the user survey, just remember they're along with helping out OpenStack, you're helping us understand how the universe works and what it's made of. Thank you very much.