 Thank you everyone, it really isn't a tech conference without some technical difficulties, so thank you for bearing with us today. So, as she mentioned, we'll be talking about how continuous delivery in DevOps really can make your IT transformation awesome. So, first, you know, Nick was hard, he wrote this article in Harvard Business Review 2003, and he didn't just say, IT doesn't matter, the title of the paper was, IT doesn't matter, so that's us, right? So many people kept thinking that just IT didn't matter, and the underlying principle and the underlying thought for the pieces of the paper was that IT was never going to help you make a difference in your business. It's going to help you keep up, you're going to have to do it, like eating your vegetables or your tea, but it was never going to make a difference, it was never going to make you awesome. Well, it turns out time to change, things are totally different now, and if we do IT the right way, with the right practices, the right processes, the right technology, IT now even just matters, it will make things awesome. It will help you leap from all your competition, and one of those things that makes it awesome is continuous delivery. So, here's what we're going to talk about today. Again, times and IT have changed, DevOps is good for IT, DevOps is good for organizations. We'll talk about what drives that change. One of it is tooling and automation, and here's continuous delivery, so we'll dig into that a little bit. Another one is using lean manager practices, those practices and processes, also having a good culture. So, back when Nicholas Carr wrote this paper, it was back when IT was just a cost center, people would buy servers, they would put them in closets, they would put them in some really pretty up lighting, which was nice, but it just wasn't enough to have some server, or they could do really, really hard things, like buy ERP systems. You could throw consultants in a problem, but you know what, your competitors could do the exact same thing. It wasn't giving you something that was a competitive advantage, and it especially wasn't doing something that we call a sustained competitive advantage. It was giving you what we call a point of parity, but not a point of distinction. Well, then something came along, and people started realizing the tech that the things that we found out in the lean manufacturing movement, in the Toyota movement, could be applied to software development delivery process. And if we embraced these types of methodologies and these types of practices, we could make our software and our technology better and awesome. We started hearing some of these stories really early, starting in about 2009. We started hearing it in 2007, but really 2009 was kind of a groundbreaking moment at the Velocity Conference. All Spa, Heron, told the story about the Dev and Ops cooperation at Flickr, doing 10 deploys per day. We've seen those numbers skyrocket ever since, at places like Etsy, places like Amazon, places like Netflix, but I like stories. But I also really, really like data. My background is research. And, you know, I'm always the skeptic in the room. I'm the one that says, oh, that story is nice, but is it actually going to apply to me? And so I started working with some of my colleagues, Gene Kim, Jess Humboldt, who works at Public Labs, and we decided to dig in and really get some data to see if this is actually applicable. Does it work everywhere? Does it generalize? Or is it just a handful of little companies that find a way to do their special sauce or take the numbers and do special math to make it work in a place that just don't apply, right? Okay, so as we start digging in here and looking at what's going on, who here is on the Dev side of the house? Who here is Dev? Okay. Who here is Ops? Okay. Who here does like project management and Agile and... Okay. So as we start talking through this, think about how it applies to you and how we can use this as we're on our own journey. So just as a refresher, Dev Ops really kind of comprises three key things. Part of it really is the tooling automation. We need some kind of technological piece, although you can't just buy something and plug it in, okay? It's not just going to be the tech because you know what? HP LaserJet firmware was made awesome. Using firmware, okay? So you can do it on old tech. People who say if you just use legacy and they can't do Dev Ops, no, they're wrong. Dev Ops is also using good practices and processes. Harkening back to the new management discipline. It's also making sure you have a really good organizational culture, okay? TIFEA research shows that things drive IT performance and organizational performance. So I mentioned this started back at 2009 velocity. John Alsbaum, Paul Hammond, 10 deployments per day. At the time, this was sort of groundbreaking, revolutionary. People didn't think it was possible, or they thought it was super, super mean for the operations people in here, right? A lot of the time when we think about an employee coming, we're not super happy, right? This is painful. I come from an ops background. A deployment meant that developers were throwing something over that proverbial wall, sometimes Friday at 5 p.m., they were going to happy hour, and we were just gonna go back. But things went up from there. So Amazon, now some of their deployment stats that they released, I believe it's 2011, they're maximum deployments in a single hour. They're seeing an employee about every 11 and a half seconds. These numbers are amazing. On average, we're seeing 10,000 hosts receiving deployments, and they're on maximum 30,000 hosts. Although people say, he's never been cheating because Amazon has more folks than God. What do we see from other places like Etsy? Check out Etsy's stats. Now, Mike Britton gave a talk in 2013 and said what once required six to 14 hours in their deployment army. Now, in 2013, took 15 minutes and a single person, they were doing 30 deployments per day. So if we can get our technology wrapped up and our practices and process in a good culture, we can make our deployments mean, efficient. In March 2014, they were up to 50 deployments per day. In April 2014, they were up to 80 and 90 deployments per day. So your processes continue to iterate, they continue to improve. So once got to the chase, DevOps is good for organizations. A lot of the time, they're trying to convince bosses management that investments in technology, that investments in tooling, that investments in people and processes are actually worth it. Here you go. When we compare high-performing organizations and low-performing organizations, we see double. So high-performing organizations are twice as likely to exceed profitability, productivity, and market share. So this is great. This number is pulled from the literature. For the second number, this has a little access next to it because I was only able to do this analysis for people that volunteer as stock ticker symbol, and I did the analysis for those that were probably traded on the New York Stock Exchange. But for those, they showed a 50% higher market cap over the previous three-year period. So we're definitely showing real promise in direct contributions to the bottom line of the entire company. IT is making a real impact. It's not just something that you do to keep up anymore. We're driving real, real, real change. So let's take a step back and see how we're driving that change. We're driving it by contributing to IT. And IT performance, when we talk about this, is three key things. Deployment frequency. This isn't just delivery and release. This isn't necessarily going all the way through to your customers. This is the ability to push code. All the way through your pipeline and have it ready to push to your customers whenever you're ready. This makes it now a business decision to release all the way to customers and not a technology limitation. This is also a mean time to recover. How long does it take you to come back from any kind of incident? This is also a mean time for changes. So the time from code commit to code deploy. Notice the first and the third are throughput metrics. It's how fast you can get these changes to your system. And also mean time to recover the stability metric. Here's something to pay attention to that's really, really, really interesting here. For years, we've been told that you can only have one and not the other. And it's usually that we're told that we have to slow down. We have to tamp down on that speed in order to get stability. This is not true. I don't see it anywhere in the data and we don't see it for years. Throughput and stability go together. And it's really exciting and really fantastic. So, what does this mean? So when you look at hyperforming demoxins, they're more agile. They get higher throughput. They can make these changes faster. We see 30 times more frequent deployments. So they're pushing code through the system. But they can make good changes. They can get fast feedback. If something is small and if something breaks the building, that's exactly what it was. 200 times faster lead times when there appears code commit to code deploy. Those exciting new fun things that you find out for your customer, you can get all the way through that pipeline. If you have compliance or regulatory changes, you can push those through the pipeline. Hyperforming demoxins are also more reliable. So they have a better change success rate. I didn't mention this previously because when we do all the fancy stats, I already signed something. So when we do all the fancy stats, it isn't a classifier or a predictor of IT performance, but it is still statistically different and is significantly different between the high and low performers. So I mentioned it here. When you do introduce a change into your code, high performers and low performers, the change is much more likely to be successful for high performers than low performers. And then look at MTTR. If you do have any kind of incident, service segregation, anything unplanned, the high performers are much more likely to recover than that very, very quickly. So as I mentioned before, demox promises and delivers more throughput, more stability in tandem. And it's without the trade-offs that iTunes told us that we're going to have just have to deal with the plan on having to deal with the trade-offs. They just aren't there. So what does that mean for us? We talked about agility. When we think about these frequent deployments and faster lead times, for those of us in the crowd that were dev, project management, even ops, okay? Think about new content delivery. When we have new content delivery, when we need to deliver value for our business, how much faster can we get that through our customers? We can save these around AD testing. How do we know which things are actually good for our customers? When we can do AD testing, we can push that AD test through our pipeline. And then we can even push through the removal of features that are value-at. Talk about speed to market and opportunity costs, any compliance with regulatory changes, any security vulnerabilities we can push through. When we talk about things that are more reliable, what do we think about value savings around reliability and uptime? Also compliance and security. And even just the reputation about compliance and security. What happens when you're the guy on the block that gets hacked? Or you're the guy on the block that gets hacked like four times? Or what happens when you're the company on the block that does not get hacked and everyone else does? That's a lot that you can say about that because you were the one that was able to address all of those vulnerabilities. Speaking about AD testing, Roniko Harvey did this fantastic experiment at Microsoft. So when we think about adding features to our products and our services, we generally think we have a pretty good idea of how our customers want, right? Who here knows what their customers want? No hands. I'd like to reflect the record. There are no hands up. My hand is up. There are no hands up. But in theory, who here has business leaders who know what their customers want and they ask us to build things? Yeah, now we got hands up. So our business leaders know what our customers want and they ask us to build things. So Microsoft was in that position, right? They're like, we know our customers. We've been making office for years. We've been doing all of our products. We know exactly what our customers want. We don't need to do AD testing. And Roniko Harvey's like, I'm 50-50 at best. Finally convinced them that they should be doing AD testing. Turns out only about a third of very well thought out, very well designed, perfectly constructed ideas deliver value. One third, net, do nothing. And one third, cost you things. Either they cost you money, they make customers go away. It's a bad thing. So imagine what you do in that situation. If you implement every single one of those things, you could be at a net zero. That sucks. All of that work for nothing. But also, if you actually implement the things that give you nothing, you've now introduced all of this complexity and technology into your code. Now you have all of this extra work you have to support. But if you can do AD testing, you now know which one third of those features are the magic one or the making you money or giving you customer engagement or making your customers happy. But without the ability to easily introduce features and easily pull back, or undo all of those features, you just don't know. We also talked about the value of reliability and uptime. Netflix is a great example of this. Are you guys familiar with the story of when AWS crashed super hard several years ago? So AWS went down. Netflix was fine. There was a big conspiracy theory that maybe Netflix was fine because they were a special customer and so they had a special account with special servers just for Netflix. Not the case at all. We found out later that what happened was Netflix had been preparing for things to randomly die. They actually waited several, several hours. I think it was five hours before they even declared a step one. Turns out it's because they had a thing called chaos monkey. Are you familiar with chaos monkey? You don't choose chaos monkey. Chaos monkey chooses you. They had been preparing for something like this. And so by the time something like this happened, they were totally prepared. They had a few services degraded, like maybe recommendations didn't work, maybe the ratings didn't work, but overall the service was still fine. By the way, chaos monkey is open source. So you can actually use it. So by building in these types of things, our own work gets better. And we thought that there were failures, right? It's like having an environment, just having an environment telling you, don't start fires. We want to have them practice on situations that are bad so that they can get better at putting out fires. We want to have our own engineers and our own systems practice on what happens when things go down so that if we have an incident or an outage or something that goes down, then it's fine. So there were a few key factors that correlated with each component. Notice that for MTTR and the time for changes deployment frequency, reverse control is pretty consistent throughout. We see monitoring is good. I mean, testing is good. Continuous delivery is important. Culture, job satisfaction kind of for learning. But I really wanted to be able to dig in here. So it was a quick, you know, mid-check. We know IT performance is comprised of throughput and stability, both are possible without trade-offs. IT performance contributes to org performance. When I say org performance, that means money, right? IT performance makes our companies better in health. It helps us make money. By the way, culture is a key predictor of both IT performance and organizational performance. We know that automation and tooling are important. But when we did the follow-on for the study in the second year or the next year when we were really kind of doing some solid science, we really wanted to figure out what drives IT organizational performance. So what we did as we dove in here, and we backed it up and we said continuous delivery. We know that continuous delivery is a good key piece, but what makes up continuous delivery? And so when we dig into the data, continuous delivery is comprised of these things, test deployment and automation, continuous integration, and all production artifacts are being converted to people. Now, in the data, what does this mean? So, for the first year of iteration, this is what we took a look at. What is test automation? Test automation are these things, okay? Developers primarily create and maintain acceptance tests. Your developers need to be doing this, not QA, not someone else. The people that are actually doing the development need to be creating and maintaining your acceptance tests. When your automated tests pass, your developers need to be confident that the software is releasable. Test failures actually need to indicate a real defect. These can be easy for developers to fix accepted test failures. Developers should share a common pool of test servers where you can use acceptance test failures. And developers should use their own development environment to reproduce accepted test failures. Those are some of the key things that really helped define what was test automation. When we looked at deployment automation, we asked what percentage of deployments were automated. When we looked at continuous integration, these were the things that were key here. Oh, sorry, that first one. That, sorry. I'll get back to you on that one. I have a copy and paste here. When we took it with inversion control, there's a key here in inversion control. And the key here is that it's inversion control for all of our production artifacts. So it's that our application code is an inversion control system, our system configurations, our application configurations, and our scripts for automating build configurations. So it's not just our application code. We need to have code for several things in all of our inversion control systems. What it does is it gives us the ability to replicate environments, to spin up environments. If we have something happen, we need to spin up an environment that allows us to do that. It also allows us to spin up environments, probably environments for dev and for test. Throughout that whole chain. Now, here's the exciting thing. So this is a structure equation model. Statsy. Where you see an arrow, impacts, drives. These are more than just correlations. These are predictions. So continuous delivery makes our work better. It drives lower change failure rates. It predicts IT performance. We see this great quote from Yahoo. We never had testability before. We have it now. We have this experience. We know stuff is working. It's working with controls. They were able to drastically gain efficiencies in deploying their infrastructure and improve their time on passing infrastructure. They can now pass their entire infrastructure within six hours of the path being made available. So it makes your work better. It also makes your work feel better. Which I think is just as important. So continuous delivery makes your work better. It also decreases feelings of burnout. Modern tech teams. And it decreases our deployment pain. The other awesome thing is going through to increase organizational performance. So it does still help our companies make money. Yes, it is worth the investment in continuous delivery. But what else drives IT performance? Name management. And these are the key practices here. With limits to drive improvement. Visualizations to monitor our work. And monitoring to make business decisions. Now, we'll go over these details really, really quickly. So with limits. As a team, we're good at limiting our work in process. We strive to look at our whip. And we have a process in place to do so. Our whip limits make obstacles to higher flow visible. At least to process improvement. And there is a way to increase throughput. This is actually one of those. With limits, by the way. One of those really interesting things that are leading indicators. Not just a lagging indicator. So it helps us predict what's about to come and what's really really good. It does a lot of really good things for our developers. What is visualizations to monitor work? We can use visual boards to share information. Sharks showing defect rates. We have a visual mode of organizing work. And we can have these on dashboards as well, right? It doesn't just have to be a physical board we can see. So this can work for distributed teams as well. But it needs to be readily available. It should be displayed at work stations. We need to be able to see what's actually happening on our teams. And then the last one. The key here is that it's monitoring. And it's not just monitoring at the page to wake us up in the middle of the night. That's not fun, right? Because then we're going to turn our pages off. Hide them across the house. So this is monitoring to make business decisions. We use data for monitoring tools to make business decisions daily. We use data to make infrastructure monitoring tools. And we also have a place to auto scale capacity. So the great thing here is that Lead Management drives IT performance. A good example here is Etsy. Etsy uses a lot of graphing tools to make decisions. Here the vertical line represents some kind of employer code push. If the graph drastically changes, the person knows who pushed the code. They know that they should go take a look back at their code. On the odd chance that I have done the work that I had time to do for the day, I can look at the graphs that exist. And if someone pushed code that didn't make an immediate change to a graph, but that changed the built-up layer, I can go take a look at a graph and prioritize work and decide what my next work task should be based on the visualizations that I have. Available to me. The other nice thing about Lead Management, it also makes work feel better. It decreases feelings of burnout and improves organizational culture. A great quote from Julia Wester who is a professor of broadcasting. She said, I was trying to figure out all my team was working themselves to death but not getting anything done. By implementing whip limits, we were able to focus on our work. Finishing work feels better than sprinting and feeling like a hero in the moment because that's only a moment. She actually found that by implementing whip limits, they were getting more things completed and delinquent instead of having a whole lot of things that were always in progress, making some investments in visualizations, some monitoring tools, setting whip limits, training your team in whip limits. It does pay off in the end. Now, I mentioned organizational culture. Take a minute and read through this a little bit. This comes from a researcher named Ron Westrom. When we did our study, we adapted this tool to become a measure for organizational culture. So as we read through this, who here works or has a friend who works in a pathological cultural organization? Who here has a friend who works in a bureaucratic organization? I've seen very few. Who here works or has a friend who works in a generative cultural organization? Who here has a friend that does not want to vote? I found this that about half the people work in a bureaucratic organization, about a third in generative, about 15% in pathological. We do find that a strong organizational culture, as is mentioned by this, prioritizes information flow, trust in health care and aviation outcomes, it tends to fix safety outcomes, but in technology, it does predict IT performance and organizational performance. It's very, very strong. High IT performers perform the best in this, but like I said, I'm just glad to start over quickly, but I think it's important. It predicts IT performance and organizational performance, because it does tend to highlight information flow and high trust organizations. And it's funny, because some people are like, yeah, yeah, you do all the tech, you're a culture girl, there's something that's interesting. They decided to take in the teams. They're theory, when they started looking at this, was that they would find the perfect algorithm to define the team based on skills. They'd take like a no jazz programmer or an R programmer, and a person with a certain sense of skills, not at all. What comes down to making the perfect high functioning team is dynamics. And the number one thing is psychological safety. Can I trust my team? Can I take risks around my team? The next one is dependability. Can I depend on those of my team to get things done? Structure and clarity. Do I know what my work is? Meaning, is my work meaningful to me? And an impact. Is the work that I do impactful to the business? And will it actually create a change? So culture is important. The founder of Intuit has said by installing Rapid Innovation Culture, we perform 165 experiments in the peak three months of tax season. Our business result according to the website is up 50%. So that's money right there. Employee result, everyone loves it because their new ideas can make it to market. Which I think is really, really exciting. And I love the thought that founder pointed out not only that he's making money and it makes customers happy, but also that the employees are happy and excited about it as well. That's really important. I really love this quote from Great Lyndon. So Great Lyndon wrote the recommender software at Amazon. You can check out. It brings up recommenders. It recommends things that you might want to buy. He walked into his senior vice president's office with his great idea. He had prototyped it out. He had coded it. Because as we go to the grocery store, you know, there's always things that you can buy at checkout. And who here has bought something on their way out of the store? They really had a plan on buying. It happens to me all the time. So he's like, this is a great idea. The SVP is like, no, absolutely not. Because in a store physically, we can't escape. We can't run away without walking past the register, without having to pay for something online. I could just close the web browser. I could get distracted. I could go somewhere else. So he walked out of this SVP's office, coded up an AV test, and pushed it into broad. He made it live. What? Went back a few weeks later and showed that it made a measurable impact on revenue. He didn't do that. And not build his hands. And not get fired. Right? But he says, I think building this culture is the key to innovation. Creativity must flow from everywhere. Whether you're a summer intern or the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate. So his SVP, probably as a thrill, was like, you're right. You made a lot of money. Let's push it in. Let's absolutely implement this. Okay. Really quickly. Job satisfaction is the number one critical of organization performance. We've known it for years in other literature. For some reason, not everyone's convinced that this would be true in technology. Shocker. Totally true. Job satisfaction is important. So just to wrap this up. IT does matter. The value, we create not just competitive advantage, but sustainable competitive advantage now. If we do the right mix of technology, process, and strong culture to tie that together. Because remember, now suddenly we're working with more than just the IT team that hangs out in a corner somewhere. We're working with the entire volume strength. So, if you'd like to learn more and receive exclusive invite to a DevOps benchmarking tool, there's a chance for a personalized analysis of your results with me, Gene Kim and Jess Hummel. A copy of this presentation and a copy of a metrics guidance white paper from the DevOps forum. You can send an email to NicoleFB at centerslides.com subject as DevOps. Again, if you want an invite to the benchmarking tool that I'm developing with Gene Kim and Jess Hummel, copy of the presentation and a copy of a metrics guidance white paper, you can send an email to NicoleFB at centerslides.com. Thank you so much for your time this morning. And I hope you enjoy the rest of the conference. Okay, turn to your questions. And there is a mic or if you want to yell, I can... Thank you, Nicole. Great talk. I'm wondering if you can give some advice to organizations that would like to go to continuous delivery and are not sure which thing they should do first. If there's one thing they could start doing right away, put them on that path. You see several different paths to success and every company point would be different. So it depends, which I know is like the hardest answer and the worst answer. Definitely having a strong culture is going to be really important because you're going to be including so many different people in that analysis or in that conversation rather. Making sure you have good coding practices, making sure you have things in version control, I would say is definitely a strong first step because it's so important and so predictive of so many different things across the value chain. And I say version control, make sure you have everything in version control. So not just your application code, your scripts, everything else. And then maybe the next step that I tend to see that ends up being really important is a good, strong automated test suite because then you can start integrating more things early on and you have a pretty good... You can feel pretty good that as you start integrating more and more things you have a pretty good, more successful as you start integrating more and really get that back. Myself Pushan here, this was a very good presentation. My query is on... Yeah, DevOps really works for e-commerce kind of a business and how actually we can link that to a highly regulated industry like an aerospace industry like aerospace industries? So the question is, this appears like it works for Web properties, how can we automate... how can we regulate industries like aerospace and the pharmaceuticals and all the gas industries? How exactly this works? So we actually do see this working quite well in highly regulated industries like aerospace. I was just chatting with CISO of American Airlines the other day so aerospace uses this quite a bit. It actually works really, really well for them because by using things in code, by putting things like regulatory changes in code, compliance in code, infrastructure in code it makes a lot of their work much easier because their auditing processes are much more straightforward now. They can see everything in code. It makes so much of their work more straightforward and very transparent instead of things being hidden in books or in papers or in PDFs or in people's heads. It kind of surfaces a lot of the process and it makes it much more consistent, reliable and really increases the value of the function of what they're doing and it makes things like security and patch management much faster and much more it helps you kind of roll it out to your entire infrastructure a lot faster and we're seeing that quite a bit right now particularly in the financial industry which is also highly regulated. Yes. I just have a reply to that as well because I'm just humble to be making a lot of these practices. To send a service in the US government line, there's over 4,000 pages of regulation you have to read in order to be able to comply that that service can be sent live and what they're doing at the moment is actually creating a set of automated scripts that you can run to validate those compliance regulations that are actually implemented by the service so the ability to actually take these regulations and turn them into tests that you can run against your systems is very, very powerful and they may be able to cut down times to go live to get this authorization from nine months to days basically by including a lot of these ideas. So it's actually very, very powerful particularly in areas where you have high performance. We're also seeing people implement it or automate certain things if you're in a highly regulated environment where people are really, really worried about this we're seeing people roll it out with a high risk profile. So sometimes that can be one really good way to start introducing this is is introducing automation for things that have a really low risk profile and then start increasing it to things with a higher risk profile as you go although in some ways you may argue that for automation where there's a higher risk profile because then people who chat live, she or constantly are involved but you can also do things based on risk profiles. I have a question about devops and delivery. The question is do we need to bring it up front with any practices like even testing and if organization is embarking towards continuous integration and delivery and devops culture how should we face or should we start up front with any practices like automation and testing and run the devops culture or progress in parallel or we can face it out in two phases like in practices earlier and then go for the integration and delivery aspect. So you're asking if you should do the tooling piece first or the practices piece first? Again I see what I tend to see is companies do the tooling piece first because you just buy it that seems to be easier I would say I'm looking at a lot of people because that's what I tend to see is like you just buy the thing and do it but generally when you start implementing a new tool I would also take that as an opportunity to start implementing one or two like agile practices or lean practices at the same time because you're going to get that sea change and that excitement and that culture change at the same time and I would take advantage of getting everyone excited doing not every single thing all at once but take one or two things. Thank you.