 Hello and welcome to the introductory talk of the DevOps track. It is given by my co-chair, Chris Butard. He is from Belgium, and he works for Inuit, and he has founded the DevOps days in Ghent in Belgium, and he will explain the rest himself. Hi. Good afternoon. I hope you guys have a good lunch. As Gerrit introduced me, I'm Chris. As far as we know, I'm not related to Dries. I know him for over a decade since we started Drupal. So my background is basically that 15 years ago, when I started my career, I was a developer. I wrote all kinds of shitty languages from Java over Pearl to PHP, and I was always the guy who was putting stuff in production. I was always the guy who was racking the machines, turning them on, putting up clusters. So I basically became an operations guy out of necessity. My role today is that I'm the chief trolling officer at Inuits. Inuits is one of the bigger open source consultancy companies in Europe. We're not a Drupal shop. We do a lot of Drupal, but we're not a Drupal shop as such. We try to glue infrastructure and application development. So I've been building clouds since before Amazon was there. I wrote some books. I wrote a lot of papers, and you can know me from my blog, Everything is a Fucking DNS Problem. But what I mostly try to do is actually stimulate people to be good at their self, good at their job, which is what I try to do. So what's this DevOps team? Over the past decade, I've been going to a lot of conferences. Strangely enough, this is my first Drupal count, but I can be seen at dozens of conferences a year. And I run into a lot of people talking always about the same problems. So we have a bunch of software developers. They write code, and then we have to put them in production. How do we do that? How do we deploy stuff? How do we scale stuff? I was running into the same bunch of people, Patrick de Bois, Gildas, Andrew Clay Schaefer, Jess Humboldt, Lindsay, a whole bunch of people talking about the same problems over and over again and sharing their experiences. And out of that, somewhere in April in 2009, Patrick and I decided that maybe we should do a conference focused especially on that part. How do you deal with integration between software development and operations? And out of that came DevOps Days. We had our first conference in Ghent in 2009, and it was a mixed open space formal program. A lot of people came back from that small conference where this was the best conference I've been to ever. This is a conference where I learned from other people's mistakes, where every participant was actually sharing his experiences. So we've done a bunch of them. We usually do a couple of them in the States, a couple of them in Europe every year. And by now, the word DevOps is something Gartner is talking about, pretty similar to what happened with Drupal five to six years ago. Small project, a couple of guys starting it, or one guy starting it, and now it's huge. So our next conference will be in Rome in Italy. I still haven't told you what DevOps is, and it's pretty difficult to say what DevOps is. It's a movement. It's a group of people that know they don't have all the answers yet, that know there's problems, and we're reaching out to different communities. Like I'm doing today, I'm talking to the Drupal community. We have been talking to the Java community. We've been talking to Ruby people. We try to figure out where the problems lie in deployment and scaling and making stuff high available and help people. And there is absolutely nothing new with the tools we use. There's nothing new with the methodologies. Only the name is new. And now people have a common word where we can say, this is DevOps. So to give you a real definition, that's the answer I was looking for. Do you happen to know who wrote this? How old is this? Any ID? It's pretty close. It's actually William Deming who wrote this in 1986 in his book Out of the Crisis. It's pretty much a summary which is on Wikipedia. And that ID, it's DevOps. How do you collaborate? How do you work together? And how do you achieve something good? Adam Jacobs turned that into DevOps is a cultural and professional movement. And that's also what it is. But let's get a bit deeper. So there's four key things into DevOps. And the first big part is culture. Culture, what is your organization looking like? How do you deal with operations? How do you deal with people? How do you deal with failure? Are you allowed to fail? Are you allowed to learn from stuff? And the second thing is automation. So you want to have an infrastructure. You want to have applications where stuff goes automatically, where humans are not involved that much so they cannot make more mistakes. Measurement is also a pretty big part. You want to know what's going on on your application level. How many users do you have? How many people are actually signing up for your service? How many people are really digging into the last page you wrote on the content part? That's metrics. And you want to talk about sharing. And that's a good summary, sharing your experiences, sharing your code. It's open source. So those four words, camps, is a good word. Obviously, there's people that call it CAM now, but we'll let them do. I used to have this slide saying, what's the real problem? And Tim Bray wrote this on his blog in 2010. But you get feedback from, yeah, but do you mean DevOps doesn't use UML and doesn't do actual quality? No, it's about the problem. The big organizations are slow. So how do you really get here? In a typical web development shop that doesn't have hosting capabilities, what we typically get is a project manager who calls at 5 AM on a Friday afternoon and he says, here's a terrible, put this code live. By the way, there's a marketing campaign. It's going live on national radio or television at 6 o'clock. So we actually need this yesterday because everything they forgot about, they had a perfectly working site on their laptop, but they forgot about, well, we need to put this onto a server somewhere. And that's painful because what's an old school operation shop then says, we don't have any machines available. Do you need a database? You're going to add how many users, where they're getting from? How are we going to do the security? Does this need to be scalable? Does this need to be clusterable? How do I install this? And you get this. You get operations people, development people fighting. So 10 days later, systems people look into the machines and they see an enormous load on their platform. And they figure out what's going on. Why is there so many memory being used? They see a disk flooding. And we wonder, hang on, is this user data? Are we storing this data? Are these logs? Why are we writing out all this stuff? Is this relevant? Is this not relevant? And you go to a developer and you ask, how many users do we have? We don't know. So why are we having 10,000 queries per second? We don't know. That's a problem. Why do you have debugging enabled in production? And you end up figuring out, who the fuck wrote this crap? Well, that's the guy who just left the building because he was sick and tired of maintaining his own traffic code and he left. So 11 days into production, Titanic. I was a state in enterprise environments. And trust me, it is that bad. So about a year ago, I was wondering how good is Drupal doing on this. And I set out to host a survey. We had a lot of questions. We had a lot of answers. There was a bunch of people who replied to the survey. So a couple of the next slides will actually have the figures of this survey. And, well, I can tell you up front, they're not pretty. But we can solve this. The thing is, we need to get operations people and development people at the first part of a project involved. We need to know that if you plan on launching a new project, if you plan on launching a new site, get everybody involved as soon as possible. Get the operations people there, get the development people there, get the marketing people there. Get everybody in the same table. So we know something is coming. So people need to talk. And we need to talk about also the non-functional requirement. We need to talk about security. If you were in the talk from Nick and Peter earlier, if you have solar setup with no authentication, everybody can destroy your site. Do you want that as a company? No. Do you need backups? What if you lose your data? How are you going to upgrade your site? How are you going to deploy it? How are you going to make it high available? How are you going to scale it? How are you going to monitor it? All those things are topics you need to think about before you start deploying a site. And it means you're going to have to break down some walls between developers and operations. It means you have to put people together in the same room and let them talk. Let's face it, both developers and operations people were all just little kids. We want to play with our toys. And isn't it much more fun if you're seeing kindergarten the really young kids? They play a part in their own corners. But once they grow up, they start playing together. And it's much more fun. So it's about opening up. It's about connecting more people. So that brings us to what is the ultimate? What do you really want? And just humble, back when he was still working at ThoughtWorks, he wrote a book called Continuous Delivery. And in his world and in my world too, it means that you are continuously updating your platform. You are continuously, when a developer puts a piece of code under version control, it goes through a number of tests and it's automatically being deployed. So you can do that continuously. Some people say how many times a day do you want to deploy it? Flickr says we do 10 deploys a day. There's companies now going on and boasting about how many deployments they can do, how many real code updates they can do in production, so how good they are. And they find that important because to them, deployment used to be pain. Upgrading a site that used to be pain. Nobody, how many of you do have a site in production where, well, you know it's a bit tricky, it's a bit old, and you need to update it? But you don't do so. How many? Lots of hands. Why? Because you're not experienced with it. You don't practice upgrading a site. And that's probably the most important part. It's not the point that you can deploy and upgrade 20 or 30 times a day. But the fact that you know that you are capable of doing so if it's needed, that's much more important than actually doing it 10 times a day. Knowing that every piece of code you have written is production ready. So how do we get there? When 10 years ago, I was a software developer, I had to teach my colleagues about using version control. I had to teach my colleagues about, look, you have to build your software. And building does not mean compiling it, it also means syntax checking. It means style quality checking. It means pattern code reuse checking. It means basically verifying if your code is good. We had some bug trackers. We had tools to do testing. And we started doing automated deployments. Systems people, up to five years ago, they were not using version control. Well, I was, but not everybody. And now we're basically telling to those people, look, you need to do the same. You need to test the infrastructure. You need to build, you need to test. You need to have some kind of issue tracking. That's practices that are needed in all parts of the community. So well, I thought developers were doing a good job on version control. Turns out that there's still 14% of Drupal developers in the survey, and those were the people that actually cared to fill in the survey. There were 14% of them that did not use any form of version control. Who's not using any form of version control here? Ouch. One guy that dares to say it. I bet there's much more. Some of you already used version control. There's already the ones that used it. 50% of them are using Git. There's still a huge bunch using Subversion. Works, it's better than doing nothing, but there's stuff. So using version control is one step. Using continuous integration. And Barry Jasmine will do a talk in continuous integration tomorrow. Basically, testing code, integrating multiple parts of your code and testing those. Testing if they're run on the actual infrastructure they're going to deploy it on. Testing if they're run to your back ends. Not with stuff back ends, but with real stuff. That's also stuff you need to do. You need to think about building a pipeline where you have a number of tests. Once code is being committed, stuff starts happening. There's a continuous integration server. There's plenty of them in the open source. Jenkins to name one, it's a really good one. And what you do with such a tool is you build a pipeline which tells you step by step, look, this piece of code has been committed. I'm going to run test one on it. Test two. If test two fails, well, don't push it to production. Don't start doing any other test. Roll back and basically alert all developers. Stop the floor. Let's fix this. Because as long as you keep on introducing new bugs, basically you're building up technical depth. You're building up a problem. And it's better to fix it once and for all and then going on. And this doesn't have to be just for the developers. You have to have a parallel set of pipelines where you test both your infrastructure and your application code. And on top of that also, and that's a more difficult part, data. How do you test data? So build pipelines where the three of them are integrated, where you do tests on relevant data. If your site has a couple of terabytes of data, which you are querying, then having a test which only queries a database with only three lines of data in there, it's irrelevant because you're not going to see the side effects. You're not going to see what happens on real data. So I asked about continuous integration in the Drupal community and, well, 72% replied that they were not doing any continuous integration at all. So I want to ask a question again. Who here is testing their code? Who here is testing it locally on their machines? Who's testing it on a platform that looks like production? OK. So the idea is that you build a platform where your development, your testing, and your acceptance and production environments are actually identical, that you don't have the same movie, but in different cities. Some people test locally on a Windows machine. Some people test locally on a Mac machine. And then they go and test. And there's some Linux flavor running. It's a Ubuntu box. They go in production, and it's a Solaris machine. How can you be sure that there won't be the same stuff? Testing locally, the local database, small one. Testing in production, well, production with a huge, massively scaled MySQL setup with replication built in. That's not the same thing. You're not testing what you're putting in production. You're going to break stuff. So sadly, when I asked the questions, there were over 50 people that said, well, development we do on my local box. And it really is a problem. People using different database schemas, people using different database engines, different versions of search engines, you're actually building up a problem, because you're not ready for production. You're not ready to run your software. There's a bunch of tools that we're using. The slides will be up. And you can actually read what's in the graph there. I'm having trouble spreading it here myself. So there's a bunch of people that were actually using Jenkins. They were actually using a bunch of tools. But it is about being able to reproduce what you do, being able to reproduce it in production. And there's tools for that. There's tools that allow you to build an environment and rebuild it for you. There's tools like Puppet, Chef, C-Evengine, that allow you to do and configure a host exactly the same you do it. If you want to start deploying 10,000s of machines, there are tools like Kickstart, Pre-Seat, Pie that allow you to deploy 10,000 identical machines. Share the code with your developers. I've come up to points where the code that we as infrastructure people write, that the developers actually start writing the RSpec tests, and they start writing the tests for our code. And that's where you get people to do review. So I'm going to mention one tool here, Vagrant. I'm going to do a buff on Vagrant tomorrow. I know there's also another talk going on Vagrant, but this is really a tool which allows you to set up an identical environment on your laptop, really lightweight. The only thing you need to keep around is not huge and huge disk images. The only thing you need to keep around are a bunch of one config file, your own code, and the code to actually deploy the machine. And if you put that into one Git repository, it's less than 50 kilobytes. It's small, it's versionable. You can use tools to reproduce those tools. And we've seen use cases where I give Vagrant boxes to developers, developers build their environment with the same code, we build the test environment, with the same code we build production. And to add to that, there's actually sales people going around with the same code on their laptops, showing customers what's going on. They just check out the latest branch, the latest version of the code that they want to show the customer, and they can just show it locally. So that basically condenses to you need to be able to have something that's installing automatically. Because if you give something to a sales guy, he's not going to be able to, oh, let me do this side install. Let me go to all the pointers. Let me point and click. You want something where your computer can install your software. And Leucanis, founder of Puppet Labs, said this, passed them in 2007. If my computer can't install this, basically, your software is broken. And I think he's right. We're talking infrastructure as code these days. We're talking about automated deployments, deploying 1,000 machines a day, without a human being part of that. We're talking about being able to reproduce this. Because if you have to deploy 1,000 machines, you don't want to do this manually. Five years ago, when I asked the question in the audience, do you want to install 20 machines? Manually, people said, yeah, sure. And we moved to 500 machines. And now it's on demand. When you deploy an Amazon instance, it has to be exactly the same as the one next to it. So think about configuration management. Think about those tools. But also, think about how am I going to deploy my site? Well, database imports are evil. You don't get the right data. You get something that evolved through a series of action to the current state of the database. Sometimes with host names in there, sometimes with IP addresses in there that point to the wrong setup in production. Also, when you're deploying a site, doing it manually means it's a human who's going to make mistakes. It hasn't been tested. And it's something that, when disaster happens, you won't be able to reproduce. It's going to be something that somebody would buy had. So from my survey, 61% were still doing it wrong. They were not doing any form of using features, using install profiles. They were just going to a site, install, click, click, click, click, click, and, bah, there was a site. It's good for a small shop. But once you start growing, you're in production. So Drupal is not the only tool with that problem. Lots of companies are having this problem. And Tom Lyman-Sally was so pissed almost two years ago. So for ACM, he wrote an article on why this matters. And this is what he told a large audience on what he wants to see in an application. He doesn't want to see GUI. He doesn't want to have to point and click left and right to figure out if stuff works. And definitely checkmark the third and the fifth options, but not a second. Well, try doing this with GUI. Using Selenium, where you only have been able to do a checkbox and not figure out if it's enabled or disabled, it's difficult. So if you add that statement and what Luke said, well, to me, if my computer can't deploy your site, it's not worth deploying. So there's a bunch of challenges. There's really big problems we need to tackle. It's data. Drupal does a really bad job of mixing configuration with actual user data. And that's something we need to figure out. What's config? What's provisioning? And what's actual user data? How do we isolate that? How do we build a tool chain where I can actually only migrate my data and anonymize that? Whether it be financial data, whether it be data that somebody wants to share, you want to be able to just isolate your data. Think about a billing application. Think about how do you deal with these things? All the billing data needs to be there. You want to get that out, stuff like that. Drupal8 is getting there. It's growing. It's doing a lot of stuff better. So what is this DevOps thing, actually? Well, there is no clear definition. You cannot be a DevOps guy. It's definitely another role. It's something you, as a group, as a team working on a platform, need to grow. You need to think, we are going to start working in a DevOps way. Actually, the only job title I think is suitable for a DevOps guy is the DevOps evangelist. There's no strict rules. I don't care. Nobody cares if you use ChefPuppetSeaVengine as long as you have some tool that does automation for you. Well, I used to say, don't write your own tool. And everybody was right with that, until both Luke, Kinese, Adam, Jacob, and Mark Burgers were in the audience, and they were the three authors of that tool. So they were allowed to write it, just as I would say, don't write your own content management platform, except for Greece. And this DevOps thing, it's definitely not new. It's old. People have been doing this for ages. And if you're not doing this already, then you're building up for disaster. You're basically allowing your company to fail. So to conclude this, Drupal is getting there. The Drupal community gets it. Otherwise, we wouldn't have the CMI initiative. Otherwise, we wouldn't have a lot of people adding code to that. And there's conversation happening. I feel that this talk is actually one or two years late on this conference. But there's still a long journey ahead. There's still a lot of people that need to learn that a site is not something you put up somewhere once, and then you forget about it. It needs to be maintained. It needs to be updated. You need to get the security updates in place. But there's definitely going into the right direction. So I have a test. And it's a test I like to do with a lot of people. It's called the 10th floor test. Who knows what it is? Exactly. Who thinks he can do that today with his infrastructure? With his Drupal sites? Well, the answer was, you pick out any random machine in your infrastructure and throw it out of the 10th floor building, the 10th floor window of your building. Does your infrastructure, does your site keeps running? Does it go unnoticed that you lose half of your data center? No, because there's the chaos monkey out there, and even Amazon goes down. But that's the goal. That's what you want to do it in. So this DevOps thing, it's a new movement. You're welcome to join it. You're welcome to join the discussion. We have a conference in Rome in October. There's DevOps Days at Oric. But remember, it's not about the tools. It's not about which ones you use. It's about changing the way you work. It's about people. It's about communication. It's about talking to each other. And it's about exactly the same of this conference, opening up, connecting both systems and people. And with that, that was basically my last slide for this presentation. So questions? Yes? Are there any mics in the audience? OK. There's huge communities growing that are publishing a lot of their code on how to deploy all of the different tools you need to set up a Drupal infrastructure automatically. There's both in the Chef and the public community. There is tools available to do deployments of Drupal, of solar, of all different components. Obviously, there is a bit of work to be done to set it up. But there's so much code you can reuse today. Five years ago, you would ask me the same question, had to say, look, you have to write a significant part of code yourself. Today, if you look in GitHub, find Puppet, whatever, any service you can name, it's going to be there. There's going to be a huge variety of people that have contributed code, that have pushed code, that is usable to build your infrastructure. You can build parts and parts together. But it's sometimes not about taking code and writing it. It's more about changing the mentality on how you manage your infrastructure. If it's a good hosting company, they know that they are managing something at scale, that they need to automate a full deployment, that they cannot afford to you picking up the phone, calling and say, look, we want you to add two web servers and this and this and this, and then going into the machines and do that manually, because it won't scale. They need to go to a point where they can say, oh, so you want to have that, that, that, and they just say include ABC commit, push, and then the whole test chain starts rolling and they start pushing out a new infrastructure exactly for you. And that's a mentality shift. That's going from the same point where software developers had to go, oh, well, we'll have them click, click, click here, here, and here and do the same with infrastructure. And that's also something that needs to change. But if I look at the goods hosting companies, they are already thinking in that mentality. And they shouldn't be charging you extra for that, because the fact that they're doing it, that's the benefit for them. They can scale. They can do stuff much faster. So it's actually bringing them in money when they do it this way, more questions. So to me, any LAMP stack can be clustered fully redundant. It depends on how you look at your data. I can do a separate talk on my SQL clustering, because that's the biggest problem, your database. And there's different approaches to doing that. One of them is caring about consistency of the data. The other one is caring about accessibility of the data. And if you want to merge them, too, it starts becoming more complex. But if your question to me is, can I build a 100% foolproof MySQL cluster? My reply is, I've done so for about five years in telco environments. So yeah, sure. That's not a problem. But it takes a bit more than just a regular MySQL setup. It takes some additional tooling to go there. And that's the part where lots of people fail. They think, oh, we're going to set up Multimaster, and it's going to take care of everything. No, you need to add some more tooling, pointing your actually application to the right instances. And that's taking a bit of work. But it is definitely possible. More questions. Which one I would say? So the question was, which tool I would advise Chef Puppet or Sea of Engine? So that's a difficult question. It's a question about the culture in the company. If you're a Fortune 500 company, if you're a bank, if you're a pharmaceutical, and you have strict rules on which software you're allowed to be deploying in your infrastructure, it's going to be pretty difficult to convince your old school IT people that you need Ruby. So then Chef and Puppet, all of the table. And you're going to look at Sea of Engine, which is, since the last version, going really into the right direction. Sea of Engine is actually the mother of the configuration management tools. Mark Burgess basically invented the field. But he's doing a really good job on where five to six years ago it was a tool where it was really a huge learning curve just to get started. Today, it's a really common tool. But if you are open to Ruby, if you're open to other tools, then the point we usually see is Puppet or Chef. And if you see a lot of Ruby shops, they prefer Chef. If you look at Europe, I haven't seen any real Chef adoption in Europe. The Puppet community in Europe is much bigger than the Chef community. And then my point of view is, look, both tools have their benefits. I'm working mostly in Europe. My customers actually ask me to do Puppet deployment. Over the five past years, I don't think we've had one single customer in Europe asking for Chef, as opposed to lots of them asking for Puppet. I know in the US it's different. There's much more Chef adoption in the US than in Europe. Both tools have their advantages. Both tools have their problems. But look at the skills you have available. If you're a Ruby shop, Chef is probably more easy to get the developers actually write code. If you have a local European shop, you're going to find much more Puppet people than Chef people. After all, it's not about which tool. It's about using one. And that's the most important one. But don't write yourself. That's shooting yourself in the foot really hard. There was another question over there. So there are two parts to this question. The first part is, if you talk about infrastructure of code, do you mean APIs? And with that, I actually have to say no. I don't mean APIs. I actually mean lines of code that reproduce what you would do manually. What you expect from a number of third-party vendors appliances and such, you expect them to have an API which you can use to talk. So there's a proprietary load banser. You're not using Nginx. You're not using Varnish and all those things. You use something big, expensive, which doesn't job equally bad as the other ones. But you're paying 5K for it, or 50K. I expect those vendors to have an API. They don't have one. So I resort to the open source tools again, because they're better. Do you want all the other developers to talk to those APIs? No. Do you want to have people that do that? Yeah, sure. With all the developers, they don't need to talk directly to the API of the load banser or to the firewall. What you want is that they, if you have a piece of code where there's a lot of configuration, you have a shared tree, you have a shared access and a shared responsibility to that code. So you can have developers actually contribute parts where they have templates, where they have code, and say, these and these files need to be in place. And I want you to have these, these knots and triggers. And they can help you there. So that's the code. It's not about the APIs. Of course, you can go a step further and say, look, we have built a dashboard. We have been front end, where you can have developers use an API to launch new machines. But that's not a full requirement. It's easy and up stuff. But the API part on the infrastructure level is, it's not a must. It's nice to have that. The most common mistakes I see are twofold. First of all, I see a lot of development teams working in Scrum. They have fixed timelines. They have a sprint two weeks. And they have a limited set of features in those sprints. And what some organizations try to do is to get the operations people involved in the sprints. Now, the nature of running an infrastructure is different. There's a hardware failure, and you need to react. There's a server going down out of memory. You need to react. You cannot say we're going to do 20 interactions in the time frame. So you need to think about different methodologies there. And one of the things we see a lot is that the operations people are using Kanban rather than sprints to actually organize not the number of days they have to fix something, but the number of resources they have to do concurrent work. And that gives them also metrics that also gives them visibility to management to say, look, we're only five people here. We can only do this and this and this together. That's an approach you can do. It doesn't mean that if you have big plan projects, you as an operations group cannot use sprints in two or three weeks to develop stuff. And if you're actually the guys that are working, the guys that are running the shop, they cannot run in the sprint. Another problem, that's the second one. I see a lot is that there still is no communication between those people. The only communication you see is an issue tracking system. Whereas what you should do is try to get, if you are doing sprints, for example, get the system people involved in the sprint planning, get them involved in the sprint demos so they know what's coming up. They know what's going to change in the application and get them also involved in the architecture boards because I've seen situations where, oh, wow, new tool, development was building something and they had, well, they wanted to do something fancy with jQuery and they wanted to have a search engine backing a pull-down menu. And they used Elasticsearch for that. And two days before the deployment, they said, look, we need Elasticsearch box. You start talking to the guys and you figure out, but why do you want this? You just need a simple key value store. Why do you use Elasticsearch for that? Oh, well, can you not use the MySQL database you already have? Oh, well, yeah. There was no communications. And when you ask me questions, they say, oh, yeah, sure. Of course. We didn't even know we could use that. So getting those people to talk by involving people in different meetings. If you are doing sprints, get systems people involved. Also, operationally, get developers involved in the daily standards of the systems people so they can add to the list of work, look, we have this problem. And by the way, this guy knows how to reproduce it. Don't make it just one central ticketing system. Get people to sit together, pair them next to each other, sit them in the next two desks next to each other. Development people, operations people, let them work together. And that solves a lot of problems. One more question there. They're manufacturing much status? Yeah, OK. So the question is, development has as a target, adding new feature, getting a side life, changing stuff and making something that evolves. Whereas traditionally, operations people say, oh, this stuff has to be stable. Don't touch it. It works. Don't break it. So that's where continuous integration comes in. That's where testing comes in. If I, as a systems guy, know that you have run a zillion tests and you could, without me being involved, going from your test setup through staging to a user acceptance platform, and all the changes you made have been tested, then what am I to worry about? It can go live to production. But if I know that you, as a developer, haven't written a single test, just did that on your laptop. And you say, well, I know organizations that don't even develop on their laptop. They just hack in production. And when stuff goes down, it's not the guy who just modified the database connection string that gets the page. No, it's the DBA that gets the page at 5 AM. He's going to be pissed. But if you know that you have built up a chain where that change has been tested, has gone through a set of validations, there's nothing to worry about. And you then are achieving the same goals. Because there's this, I don't know if you guys know ITIL, Information Technology Information Library, which has a lot of procedures, which has a change management board where you have to have a group of people deciding if a change can go through. And lots of people say, look, but DevOps in ITIL cannot go together. And that's wrong, because they can go together. The main goal of ITIL is basically making every change being auditable, monitorable, and manageable. And it's about reducing your risk. And if you have reduced your risk by knowing that every committed developer has done up to the point where everything is tested, you have reduced your risk. You are much more sure that the last commit he made is going to be good, as opposed to let them do three months of development works and then do everything in bulk. Oh, hang on, we just did three months of development work. We pushed it to production and something breaks. Which code change broke it? Nobody knows. So the fact that people think that continuous delivery is going to be much more dangerous than just big production changes, they're wrong. It's about getting people to the point where, look, we want to have new features and a stable platform. Let's make sure that all the new features we want to implement have been tested. So we know it's a stable platform. Does that reply to your question? OK, thanks. More questions. I can go and do Q&A for a couple of hours. That's why I keep a slide short, but a few guys. The next one is in Rome, October 5 and 6. Go to devopsdays.org. It is there. There is a map with two countries. And on the map, I think they're still Melbourne and Italy, unless somebody just didn't get pushed without doing tests. OK, so it's a usability thing. Well, IKEA is working in the Philip DevOps way. And they do that stuff. They do full automation. They do puppet. They do a lot of monitoring. They use New Relic as a back end and stuff. You don't work with the sysadmin, but they have built a self-service platform for you. And not only accurate it. I have built that in-house for customers. There's other vendors around that build the same. But they give you the tools to set up development environment, to move stuff to staging pretty easily, and then to move it to production. And they've taken out all the complexity to do that for you, but you're still doing that. And in a way, the fact that they give you those tools to do that, that's the communication already. One more question there. Can you repeat the question? Yes. If you write good cap files that actually have tests before that and stuff, it could be considered a continuous delivery tool. Definitely. It's just one of them. As I said, I think there's talk about Capistrano and Vagrant boxes. There definitely is a Capistrano talk later this week. So yes, that's one of the tools you could use to actually do deployments. Why not? So if you guys want to mail me, bug me, follow me on Twitter, read my blog, or call me. But please don't call me in the middle of the night. My kids already do that. Then.