 I'm Nicole Fortran, I'm director of organization performance and analytics at JEP. I'm also a researcher at the University which is mostly a fun way of saying I don't like data. And so I'm going to be talking about how DevOps is a key to organizational and IT performance because I really love stories but the plural of anecdote is the data. I love stories, I love hearing about the transformations that I see in the industry over the place but it's really hard when like we go to executives and managers and we tell them about this amazing thing that I heard at this company, it's other place they're like super cool, super fucking working here, who has done that before? Yeah, right? And some of this comes back to this amazing article that was written by Nicole's car in 2003. It was a Harvard Business Review and the title of this paper was IT Doesn't Matter. This sucked because every executive read like read a business review and they're like super IT doesn't matter. So suddenly IT's a cost that are who cares for this? Everyone, right? And we've all heard it way too many times and so suddenly it was like technology doesn't matter, right? Like we're gonna deal with it because we need to deal with it and that's it. So this is where we're going, this whole point in this talk is we're gonna talk about why IT does matter and we're gonna talk about how times have changed and IT has changed, okay? DevOps is good for organizations, DevOps is good for IT and then we'll take into some of the detail lines and talk about what times has changed and we need to have three key components to really make a difference here. We need to have technical practices, we need to have some process around it and we need to have culture but we have to have all three of them in order to make this work and in order to make it stick. If we only have one and not the other two it just doesn't work, right? If we only implement technology and not the other two this is going to feel and look a lot like the 80s and 90s when like companies were just implementing like massive ERP systems and not doing anything else and they were surprised when it did some like massively change things, right? This probably sounds familiar or if we only do the process and suddenly have our meetings standing up instead of sitting down and nothing else changes, doesn't do anything or if we only fix the culture and we have like pizza on Fridays but nothing else changes that also doesn't make a difference, okay? We need to have all three. We'll talk about why. So we know that if we only implement technology it just doesn't make a big difference in organizations and we know this you know Nicholas Carr said you know IT doesn't matter and it's because for years and for decades organizations spent money on IT but it was because you know I can buy IT, plug it in, I can throw up a closet, I can give it really pretty up lighting, by the way this is Facebook, right? This is Facebook's cross-center but my competition to do the exact same thing, right? My competitors can buy the same server or competitor server and they can do the same thing. It gives us a point of parity but out of point of distinction so that's what they call it, right? It helps us keep up, it doesn't help us get ahead. So we have to have all three components in order to really make a difference, right? We have to include the whole organization, we have to include the value chain, we have to do this with an eye toward business value and then it suddenly makes a difference and so I'll be talking about the research that I did over the last several years with Gene Kim and Jess Humble and we're looking at data points with technology teams around the world. We have over 20,000 data points, in some cases 25,000 data points from several teams and so if you go back to your organization, if you go back to your managers and they say, oh, this just doesn't work for me, it totally does. So we're looking at industry verticals across several places, right? Retail, enterprise, technology, financial services, this really holds. This has also been published in Pureview so we see evidence in several places. So when people ask me what I mean when I talk about DevOps, what I mean when I talk about DevOps, I'm talking about technical practices that focus in on continuous delivery and I'll talk about what I mean when I say continuous delivery, management practices seen in lean management principles, and organizational culture and identity and research says that these drive IT performance and organizational performance. Okay, so let's go ahead and cut to the chase. DevOps is good for organizations and when I say that, I mean high-performing organizations, so when I like throw the data in the pot and I say, what do I mean when I say high IT performing organizations? I'll get to IT performance in just a second, but I say when I compare the high performers to the low performers, the high performers are twice as likely to exceed profitability, productivity and market share. And for those that are publicly traded on the New York Stock Exchange, they see 50% higher market cap growth over the previous three years. This came from the 2014 study. The 2015 and the 2016 study didn't have high enough data rates for me to do statistical analysis, but this is showing up in the data for the overall organization. So, pick this car. Good second, right? IT does matter at the overall organizational level around the world. So, this just has the asterisk because this was only for publicly traded companies, but the 2X more likely to exceed profitability, productivity, and market share. This holds true for all companies. So, that's awesome. So, for anyone who just sees the only needed proof that it matters for companies is out. That's awesome. Okay. So, also, DevOps is good for IT. What do I mean when I say IT performance? I'm talking about throughput and stability. This is how I measure it. And there are a few reasons that I count these two measures. So, throughput and stability. When I talk about throughput, I'm talking about deploy frequency and lead time changes. I include two different measures for stability. MTTR, mean time to recover, and change failure rate. Now, there's a couple different reasons I use these two measures. One is that it speaks really well to the two different groups that we tend to speak about here, right? We talk about the traditional ops crowd, right? Their role in life is to introduce changes and code as quickly as possible. Does it sound familiar? I'm seeing some heads. Now, the other group that we tend to talk about here is the ops crowd, right? They want to keep the infrastructure up, right? They want stability. They want to limit code and changes, right? So, MTTR and change failure rate. So, these measures are good because it speaks to the two crowds, right? They're also really nice because these two sets of measures are a tension to each other. It's really hard to gain one set of metrics without gaming the other, okay? So, these are also really good here. So, when I talked about high performers versus low performers, I threw all of these metrics, like, just in a bucket, right? I threw them all into the screen. I said, like, let them all fall. How do they fall into groups? The high performers tend to be very, very good at all of them. So, they all load together. They all cluster together, cluster analysis. They're all similar to each other and dissimilar to the other groups. There was also a medium performer group that were all similar to each other and different from other groups. The low performers were all similar to each other and dissimilar from the other groups. It fell into three groups. Oh, I'm super creative. So, I call them high performers, medium performers and low performers. You'll notice that I said the high performers were all really good at all of them. The low performers were all really bad at all of them. And the medium performers were all hanging out in the middle at all of them. This is an important point. For years, we've kind of been told that in order to be stable, you have to kind of slow down. I don't see it anywhere in the data. Three years now, over 20,000 teams, I don't see trade-offs. I just don't see it. So, that's also super cool, I think, but I like data. So, kind of weird. So, here's what we see. High performing DevOps teams. So, it's high performers compared to those low performers. High performers are more agile. I see 200 times more frequent deployments and they move faster. Over 2,500 shorter lead times. Code commit to code deploy. So, when you think about moving fast, making changes, getting code to market, being able to pivot, this is a compelling, compelling story. Also, when we talk about being more reliable, those high performers compared to those low performers, I see 24 times faster recovery from failure. And three times lower change failure. So, when I introduce any type of change to the system, any kind of service innocent outage, not even a full failure of the system, any kind of service innocent, it's three times lower change failure. So, DevOps promises and actually delivers more throughput, more stability and it's in tandem without the trade-offs that ITIL implies or calls for. We don't have to have these change windows. We don't have to have these two week waits. So, let's talk about what it means for us, right? So, we're more agile. We're faster. So, think about what this means for us and all of our teams and our businesses. These are really compelling when we go to our executives and we go to our managers. What does it mean for new content delivery? Being our competitors to market. Value and savings around EV testing. Value and speed to market. Now, let's consider a world where we don't have competitors. There are times when I did this talk with people, there's someone back and it's always a guy. Some guy in the back is like, I don't have a competitor. I work for a government and I don't have a competitor. Like, hey, maybe you don't have a competitor but you always have to deal with regulatory situations, security regulations. There's always something we have to deal with. Being able to address that quickly will always serve you well. So, Ronnie Cahoggy is at Microsoft. He was at Google for years. And speaking of AB testing and being able to do experiments, because we always think we have the solution. We know exactly what our customers need, especially the people in suits at our companies, suits and dresses. They always know exactly what our customers need and what our customers want. Turns out we are not very good at knowing what our customers want. So the ability to test and understand what our customers want and pivot quickly serves us very well. Ronnie Cahoggy said evaluating well-designed, executed experiments that were designed to improve a key metric, only about one third were successful in improving that key metric. So if we have the ability to test that quickly in our code and then pivot and redeploy code that can address that, it serves us very, very well. So let's also talk about reliability in our code. If our code is more reliable, what does this mean for valuing and saving around reliability and uptime? We also have compliance and security or even just reputation around uptime, compliance, security. Mary this morning spoke about Southwest and Delta going down. In the United States, that was a big deal. They just went down for a few hours and they impacted flights around the world for days. Several years ago, Amazon went down and it affected everyone for hours. Are you guys familiar with this story? And Netflix did not go down. There was a big conspiracy theory that said that Netflix didn't go down because they were the biggest customer. So they had special cash, they had special servers that were only for Netflix. They did have some degraded servers, and you couldn't rate your favorite movies. There was some limited search functionality. Turns out they did go down for Netflix. Netflix didn't even notice for a few hours because of Chaos Monkey. They had been prepared. They were ready. They didn't even declare a second incident for five hours. When Heartlead came out, the two most affected properties were the Canadian Tax Agency and a major hospital in North America. An executive at the hospital said fixing it as soon as possible or having compensating controls in place days before could have saved this entire breach from occurring in the first place. Notice he said days before. They knew about this two weeks before it happened. But they couldn't respond in time. So up to this point, we know that IT performance is throughput and stability, both are possible with that trade-offs. We know that IT performance contributes to organizational performance. So the big question is, what drives IT organization performance? I'm sure we all want to know how to like, do the thing. How do we fix this? So, state of another day, another way. We know what IT performance is. Step one. Step two, question mark. Step three, profit. I love you guys. So here's what we found. Continuous delivery makes our work better. This is what we found in the 2015 study. The way we can read this model is the factors on the left are what make up continues delivery. And this was the first iteration of the study. So we know that comprehensive fast reliable test and deployment automation. So test automation deployment automation is an important part. Also test based development and continuous integration is an important part. And putting all of our production artifacts in version control, all of these are important pieces. From here, we know that continuous delivery leads to higher IT performance, which in turn contributes to higher organizational performance. We also know that this decreases deployment thing. I don't know about you, but like push and deploy and having that panic into your gut is not fun. So it's important that continuous delivery makes our work better, but it also makes our work feel better. Here's an example from Yahoo. We never had testability before. We have it now. We have this experience. We know it's working and it's working with controls. They have automation automated configuration deployment of 250,000 nodes. They can deploy up to 140,000 node configs in eight hours. They can patch the entire infrastructure within six hours of the patch being made available. By the way, here's without a map. This is from a peer review paper. We were also able to find and add a path to burnout. So we also see decreases in burnout and added a path to organizational culture, which I'll talk about in just a minute. I was really excited when I bumped into someone from Microsoft who had a fantastic example of implementing continuous delivery and finding a really good example of improving that decrease in burnout and that decrease in or that increase kind of in work life balance. So they just came from a talk of from Dev Ops 2011 2016. They implemented CD and what they found was so you can't quite see it in the little screen cap there, but before CD their work life scores were 48% and after CD it jumped up to 75%. So really does make your life better. It makes it feel better. So we we iterated on this made a little bit better. Right? So research kind of take big steps through time. So 2016 study, we added effective test data management. We added Trump based development. And we added incorporating security and so shifting left on security. Does this actually help? Some of these were actually pretty controversial. A lot of people just didn't believe that Trump based development is going to be an important part. And they really didn't think that shifting left on security would help. They know the security is important. They were worried if it slowed down the process. Turns out it doesn't. It can do it the right way. What this does, it needs to less rework, which is important. Several of the other things that we found, lower levels of deployment pain improves our organizational culture, which you know, I'll get to in just a second. Higher IT performance identifying a stronger organization to work for, which again, I'll chat about in just a second. And several of these things again contribute to higher organizational performance, which our businesses really, really care about. So what else drives IT performance are lean management practices. So in 2015, we focused on lean management core practices of using effective whip limits, which by the way is one of the rare leading indicators in software development delivery. There aren't very many here. So that's a really good one. Another one was the use of visual displays to monitor quality, productivity and whip. And another one is the use of monitoring tools to make business decisions. The key there is making business decisions. So this is not like using pagers to wake you up in the middle of the night. This contributes to higher IT performance, decrease burnout and making a better organizational culture at work. A good example of visual displays here is Kezi. Mike from Etsy at the time was VP of operations at Etsy. And he said if it was graphite, they have some fantastic tools there that contribute to this great culture where every time you see a vertical line there is a time that they pushed code. So you can push a code, you can take a look at a graph and you can see if the graph moved. And if it moved in a way that was expected, you can go on with your day. If it moved in a way that isn't expected, you can go back, take a look, make sure you haven't done something different. Another thing you can do is when you're done with your work for the day, you can go take a look at this wall of graphs, or at least the portion of the wall of graphs that pertains to the work that you do. And if something is looking a little different, because we know that sometimes the work builds up, something like a memory leak problem won't show up in your graph immediately after you push code, you can go make a business decision or a decision to work on code based on the graphs on the wall. Another great example about Wyplovitz came from Julia Wester. I was chatting with her at DevOps Enterprise Summit one day and she said, I was trying to figure out how my team was working themselves to death but not getting anything done. By implementing Wyplovitz, we were able to focus on our work. Finishing work feels better than sprinting and feeling like a hero in the moment because it's only a moment. Now in 2016, we decided to go even further up the stack. Kind of taking a look at the puzzle front end. So here, we took a look at gathering, broadcasting, and implementing customer feedback. So how many times we work on things and wonder where it came from, who actually cares about this, did this come from the customer, do we understand why we're doing what we're doing? We also took a look at splitting work into small batches. So decomposing that work into small batches pieces that we can work at, that enable that small single piece flow through the system and then making that visible all the way through the process. We've called lean product management. All of these contribute to these things. Identify, again, strongly with the organization we work for, a strong organizational culture, and a higher level of IT performance. So, I keep alluding to this organizational culture. Take a look at this. This is based on the work of Ron Westrow, he's a sociologist who actually works in areas like healthcare and nuclear power, areas that are high risk, very complex, and he suggests it's a typology where high trust and information flows are important and it predicts what happens when things go wrong, which is perfect for technology. So, we found that this is very predictive of IT performance and organizational performance of tech. So read through these. Who here has a friend that works in a pathological organization? Low cooperation, again, just your friend, who here knows somebody, has heard of someone that might have heard somewhere that's typified by low cooperation, messengers of shot, responsibilities, or sheriffs, bridging is discouraged, failure leads to scapegoating and everything is crushed. A few hands. Who here knows someone, or has heard of someone that works in a bureaucratic organization? Modest cooperation, messengers are neglected, narrowly defined responsibilities, bridging is tolerated, failure leads to justice, and novelty can lead to problems. Who here knows someone or is lucky enough to work in an organization, teams with high cooperation, messengers are trained, risks are shared, bridging is encouraged, failure leads to inquiry, and novelty is implemented. A few hands. We find significant differences among high, medium, and low performing teams, you can probably guess which way those go. We also see about one third of respondents across the data set in generative organizations or generative teams. 55% in bureaucratic and 50% in pathological. Now, in 2016, oh by the way that's cross ball three years 2014, 2015, 2016, we've kind of stuck pretty constant to that. You can measure that with a six to seven question study, and I see several teams measuring that quarterly or or twice a year right now to measure their teams because we do know that as as teams start fighting, as culture starts breaking, technology starts breaking as well. So 2016 we moved on to identity as well because we do know that identity is a pretty good proxy for or is at least a really good indicator and a good predictor of how you will continue to contribute to your team and how loyal you are likely to be to your organization, which is really important nowadays, particularly because recruiting for engineering talent is so difficult right now. So these were some of the questions that we used. It's been adapted from an MISQ paper. So this is also pretty close in a quarterly really highly employee NPS score because of ever heard of NPS. By the way, there was an HBR article that showed that NPS is really predictive of revenue as well. So that's another reason we wanted to take a look at this and it's pretty similar to these two constructs together, culture and identity is really similar to some of the Google culture findings as well. These two combined map really really well to the Google culture findings. So I'm glad I chose to work for your organization rather than the company. I talk to this organization is a great company to work for, willing to put in a great deal of effort beyond what's normally expected. I find that my values and my organization's values are very similar. In general, the people employed by my organization are working toward my goal and I feel my organization cares about me. The other nice thing about this is it talks about goals, values, what my organization does. This maps pretty well to things that are likely to fight against burnout. That's the other reason we put in this initial study. Oh look, Google team performance. So Google studied, they've studied their managers for years. They have about 37,000 people they've studied in Canary and for years they only studied the manager so it's more than a thousand people. They decided to study their IC, their individual contributors, that was 36,000 people. They expected to find like the magic mix of what would make the perfect team. They thought I'll find like a Node.js programmer and an R programmer and a project manager. It's going to be perfect. But they found not that. It didn't matter. It all came down to team dynamics. The number one thing aren't above the psychological safety. So we actually included in the 2016 study, we included, we turned this into survey questions, we included a latent construct based on Google team performance. Turns out it didn't work and it didn't work because it split in half. The top two items loaded really strongly in the Western culture. It like statistically that kind of like hangs out with the Western, the culture items and the bottom three items hang out really really strongly with identity. So turns out we were capturing, we were finding the same things that Google was finding. Just split in half with Western which is trust, safety, like hanging out with those people and the bottom three structure and clarity meaning and impact. I want to keep working with the people I'm working with and I know that my work is contributing back. But also there's no magic formula for what makes the perfect team. It's understanding that I'm safe, I depend, I can depend on the people I work with and I know that the work that I do has made it. I think that's really exciting and really important and fascinating as well. And that comes back to, you know, culture. So into its founder Scott Cook has a really great quote here. He said, by installing a rampant innovation culture we perform 165 experiments on the peak three months of tax season. Our business approval result conversion rate of the website is up 50 percent. Employee result, everyone loves it because they're new ideas can make it to market. I really love this for a few reasons. First of all he calls out the culture, right? He said they perform 165 experiments in the peak three months of tax season. That's, that's risky, right? I mean how many people here perform a lot of experiments in peak high season? There are many executives that do that but what else are you going to do, right? I mean you don't, you can't perform experiments when it's not tax season and if you do those are not the people you want to do experiments on. Like that's not your key, those are not your key customers, that's not the behavior you want to be experimenting on anyway. Not at all. The other thing that I really love that love about this is he highlights the conversion rate, right? So he talks about like it worked, we made money, this is awesome. The other thing that I super love though is that he immediately goes to the employee result. We made the company money, okay, like made my shareholders happy, this is great. I also made my employees really happy. I did write by my employees. People are happy, this is fantastic. Everyone loves it because they're new ideas can make it to market, which is great. Now there's another great example here that I think is really interesting from Amazon. Who here, I'm sure we've all shopped on Amazon, right? Have we all seen the little recommender thing on the shopping cart as we check out? This is a great story. So as we go to like the grocery store or convenience store and check out there's like magazine, gum and sweets which is awesome and awful because I get sweets every time, but it works because we're trapped, right? I can't be like, distracted, walk out because I'm trapped. Amazon though, not so because I get distracted on a website, close, close my browser or just like squirrel, walk away, happens. So Greg Linden had this idea who's like this is amazing, I'm gonna write a recommender system as we check out. Code it up the prototype. Walked into his SVP's office, SVP, right? Seen your vice president. Walked into his office and got this great idea. His SVP was like, cool idea. No. Super no. Because if you get distracted, we are going to lose shoppers. Greg's like, but no. This is how it works in my head, right? Walks out of his office. His SVP's office goes back to his own office, codes up an AV test pushes to production, for real, leaves a few weeks ish, I think, for my time, finds out that it increases revenue by a few percent, which at Amazon scale is millions of dollars. Goes back to his SVP's office. The SVP isn't like super happy, but also millions of dollars. So he's like, okay, do the thing. And this is what Greg writes. I think building this culture is key to innovation. Creativity must flow from everywhere. Whether you're a summer intern of the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate. Don't raise hands. Who here would be able to code something up, push to production, when your SVP just said no? And who has that culture? But also, who here has the technology staff that would allow you to push to production in a couple hours and know that if it didn't work out, you could push a JK to production in a couple hours? DevOps lets you do that. It provides the technology, it gives you the practice, it gives you the culture to enable all of these things. And it gives you the telemetry, it gives you the data to be able to capture this and understand and know what the impacts are. That's power. So here's what we've covered today. Times and IT have changed. We know that DevOps is good for organizations. We have the data to prove it. And it's not just me. Now we're starting to see studies come out of Harvard and MIT that are showing us. And it's interesting because, like, so many of us in this room, we know this, right? Like, yeah, it's totally made sense. This flies in the face of decades of research. So this is really, really exciting. We're at exciting times right now. It's good for organizations. DevOps is good for IT. Now we know what drives this change. It's technical practices, it's management practices, and it's culture and identity. But we have to have all three. If we do any one of these without the other, it's going to look a lot like the 70s, the 80s, and the 90s. And that's how we get so much, you know, funny faces from our execs when we go to them and we have that magic bullet because it's not the magic bullet, right? It's going to be hard work. But now we know what it takes. If you want any more information, I've got an ROI paper coming, and you can get peer reviewed research for any of the state of DevOps reports, links to them at devopsdeskresearch.com. And we've got some time for Q&A now. So thank you so much for your time. I appreciate it. And we'll have a story about what they're working on now. Sean, of course, I'm interested in the culture aspect for DevOps. I hear that talked about a lot. Is that something that's often already in the organization? I mean they're kind of, they've been in that vein, or is that something that, as they've made this transition to DevOps, or has it come with ads over agility first? Or is it something that's just kind of purely coming in as a, you know, adopt the DevOps capability, I guess? So all of the above. So sometimes you walk into an organization and the culture's already there. Sometimes you step into an organization that's trying to undergo this transformation and you realize they're way about all the pathological end. And there are several pieces of culture that are important, right? So there's the western one that I talked about which is high-trust information flow. There's the climate for learning which values learning and realizes learnings and investment and not the time-seek collaboration and realizing that, or at least an acknowledgement, that interactions between dev and ops and info-sucking testing QA needs to be the win. There's overall job satisfaction and enabling and equipping people with the correct resources and taking advantage of the skills and opportunities to do their job, right? Those are really kind of the key public core, I would say. It's some of everything, right? So if if you're working in an organization where like, so those things that I just said kind of resonate and you're like, yeah we're, we have a little bit of improvement but we're kind of on the right track, you're probably pretty good, keep fostering that. But if what I just said, you like, not at all, work on building that, right? You really need to be working on things like cliche buzzwords ahead, right? Breaking down silos, fostering communication, finding ways where taking risks is and some failure is okay as long as it's like within bounds and everyone trusts each other and you understand that like when you took those risks and when you failed it was because you really thought it was the right move. What does that make sense? Organizations, what's the high curve I'm just interested in where are we? Garten has a high curve and you know it takes a while for organizations to adopt. So DevOps has been around for a number of years now, so does your research tell you what kind of percentage of organization you're actually achieving those kind of results that you're talking about? So I, I don't do research in where the high curve is. Gartner says that we're at the peak of a high curve which I sort of love you Gartner. I take that a little bit with grain salt. I think it depends on where you are. There are some organizations I talk to right now when I work on assessments and and they, you know, some of them have not. This is nowhere on their radar, right? Like they're still way back in legacy code days and they don't know where to go with culture and they are nowhere near continuous integration let alone continuous delivery and they don't know what it, they've never heard of like drug-based development, right? Like they have not heard of the games product. Like this is nowhere near their radar. There are some that like don't, you can come to the room, totally want your help with this. Don't say the word DevOps. Like we're completely incinerated. Like the execs don't want to hear the word DevOps. So say to transformation and technology initiative, but you can't say DevOps. And then, so I'm also, I also have an academic app where I publish academically and I still, you know, I was tenure-tracked a few years ago and in academia, like they're barely starting to use, they're barely starting to understand that DevOps is the thing and that's in the last two years. So that's completely an answer. It depends on who you're talking to. But it's, I think, I don't think we're at the top of the head curve like Garver says. I think we're still a couple years old. So you mentioned a couple of times that the psychological safety and most elements correlated with Western culture aspects. Are they're similarly effective, but different things that you see in other cultures? Or is this particular element that correlates with these successful teams? Or what does that look like as you start to look at it on Western countries, on Western teams? So I haven't, I haven't measured national identity culture items yet, but I haven't seen significant differences by country yet. But I also, most of my data sample right now is predominantly United States and EMEA. Are you mine? Just a couple minutes left. Have you seen DevOps work in a highly-requited industry since the point? Yes, quite a bit. So I I was just, I've just been working with a large financial services firm. Actually, a couple. So I've seen it work quite well in a couple baking areas and telecom, which is another highly-regulated regulated environment. Obviously, the clash between the regulation and innovation. So there isn't necessarily a clash between innovation. There are there are certain times where it can, it can be perceived as a hurdle, but there are also other times where this, this is really a benefit. So there are like things like soft supply and some PCI, where the thing is that you decide as an organization how you're going to comply and then you, and then you do what you decided you were going to do. So sometimes it just means going back and re-deciding as, as organization, like re-studying out, re-establishing, re-finding what it is you're going to do. The other thing that's really nice is that by, by automating as much as you can, you have then created audible systems and audible records that faster and enable more, more innovation and more speed. Because the more manual processes you have, the more change approval orders you have, the more it slows down the process and the more manual stuff you have, the more opportunity you have for error and the less, the less audible one you actually are. So we actually see several, several financial services firms, several telecom companies being really innovative in this space because they, they realize the opportunities that are there and the better they can take advantage of this and leverage this. The, the better opportunity they have to get ahead of all of their competitors. Really enjoyed your talk. I'm curious if you see any correlation between sort of why DevOps becomes really effective and related to company size. There's always this question, you know, hey I think that DevOps is really great but I don't think that's really necessary until I get to X number of employees or engineers, a team of the size. Have you seen anything in the data that you've done? So I, the last time I took a really, really in-depth look was the 2014 data and there was no specific significance at all. I just recently took a very quick look at the 2016 data and it does look like there is, there is some difference where, now I don't want to say that there's, so I'm just saying, I don't want to say that there's, like you don't need to pay to do it until you get to a certain size, but it does look like larger organizations are less good at IT performance. Is that the right way to say it? There, there's a higher likelihood of larger organizations at enterprises being low IT performers in the, in the latest dataset. I will also say that between 2015 and 2016, like, like, apparently this way, because we were reading about left to right, it was like, like, people were trying to log 2014, 2015, 2016. Like the high IT performers took off 2014, 2015, kind of got better. 2015, 2016, skyrocket. So that's the other thing, right? Like, if you want to be getting better, you need to keep getting better, because there is no, like, I'm going to hang out, I'm going to take your off. It'll be fine. Everyone's just hanging out. Superno. So I'm not entirely sure, because I haven't had time to take a huge look at it this year, but 2016 was the first year where I've, and I just took a super quick look at the last couple of weeks because someone shot me an email. So I was like, I haven't looked yet. Holy cow. Because my first email to them was like, I don't know, I haven't looked, but I know the last time I looked, there was really a big difference. Let me check, let me check. It's like, holy cow, there's a difference now. And it could be because the 2015 to 2016 year job was, was huge. One more. And then time? One more than time. So last question. Yeah, sorry. And then you can find the actor on the way to lunch. So on the topic of platform with organizations, are there, do you see any emerging organizational patterns or technology patterns that may accept the platform? I don't have data on that. I mean, we can chat, we can chat story. We can talk story, but I don't have data. Okay. Thanks, everyone.