 Okay, so this is quantifying the true business value of DevOps, and I am Ronald Zadrian. Hi, and you did not drink too much last night, or maybe you did, but the screen really is jittering a little. Sorry about that. Yeah, that's true. If you have any questions, given that we're two and we're going to be checking Twitter, just tag them DevOpsBiz. So Adrian and I work for BlueSpark. BlueSpark is a distributor company. That's a good chunk of BlueSpark at Austin, and DrupalCons is usually where most of us meet. And we do, we're a Drupal shop, like I guess most of us here are, and we do websites for various clients. In addition, and to a certain extent, why DevOps is really interesting for us from a number of different perspectives, we also work on a product, which is RumiFi, which launched last week, actually, with like its first offering, and if you do anything with travel, hotels, you'll be interested in this RumiFi.us, check it out. So we do a product, and we have to worry about how we get changes, and we launch new versions of that product, and how we get to market quicker. And we also manage a magazine, like an online magazine, which has its own set of challenges. So when we think about DevOps, we think about it from a number of different perspectives. What can we do for our clients? What do they need us to do? What can we do for product development? And what can we do for kind of an online site that we run entirely and we need to worry about all its different aspects? So the main objective of this presentation is hopefully to start a debate about what's the best way to convince non-developers that DevOps is something to invest in. I think we all largely agree that this is a good thing, and we'll talk about what exactly we mean when we think about DevOps operations. But how do you go to business and say, look, this is a good thing because, and it's not because there's some cool tool or something really exciting technologically, because it's going to change something for the business. Okay, so before we get into that, let's just briefly talk about what is DevOps. And very quickly, DevOps is essentially the realization that, at least the way I understand it, that nothing is done in isolation, right? So any part of software development relates at some point to something else, and unless we start seeing all the different parts together and how they connect, it's not going to be an efficient flow from design user experience all the way to actually releasing it into the world. And the great thing is that this is something that other disciplines have had to come to terms with and deal with long time ago, especially manufacturing. And there's a great book called The Phoenix Project. How many have read that book? Look it up. It's a bizarre book because it talks about dev operations, but it's written from the perspective of just this guy that wakes up at home and has a wife and kids and needs to go to work and there's the politics of work going on. And it's just, it's literature, right? It's a story. It's a story that only developers will get because then it becomes quite technical. So it's this weird mixture where you're reading an actual book, like literature and narrative, but it's very technical and it's a great way to get across what the problems are, what the issues, challenges and how they fix them. And one of the main points that The Phoenix Project brings up is that you need to look at other disciplines. They talk about manufacturing. So think of that process as going from one type of discipline onto another and they need to interact and they need to think through where the bottlenecks are and then it describes how they solve all that. Okay, so that from a very high level perspective is how we think about DevOps. Now Adrian is going to talk a bit more about the specifics and how we break that into layers and then we'll talk about the analysis we did. And so we were thinking about this and thinking what really are the different parts of DevOps. How do you segment this and how do you break it down so that you can analyze it? And so we're just trying to give you the background for what we're actually talking about when we're talking about putting some sort of framework together to quantify what DevOps gives you, what it saves you. And so we found it's helpful to break it down into chunks but it's also about realizing that it's not just, oh, we do DevOps at the shop or we don't do DevOps, it's too hard, it's too time consuming, expensive or whatever but how much DevOps do we do? Now in a month, in a year. And so sort of the first phase is literally the basics of what you build upon and that's what we're calling create your world and that's the things using tools like configuration management and we don't get into specific tools in this talk. It's almost missing the point in getting into debates about what's best but it's just about let's use a tool. And so if we're automating things, you have a lot of benefits. You've got happy developers, they're not sitting there and logging in, they can brush up, get, pull the rest of it. And so you've got a stable, repeatable, you might even say, an important base upon which the rest can be built. And so the next place we go is monitoring and that's really just knowing what's going on and having effective monitoring really can prevent a lot of embarrassment in your life. Don't ask me how I know. But anyway, it really is hugely important and effective monitoring is even more so. It's great to have tons of notifications when your server loads to over two but it sort of all gets into a big wash of data and isn't useful and you end up losing sleep and getting mad and just shutting down Nagyos, which isn't really helpful. So there are two big pieces to talk about. One is notifications for things you need to know about and you need to know about right now. If you know what's going on then you can save a bad situation before the rest of the world knows about it. And then the second is to make sense of the huge amount of data an application can spit out and there are a ton of tools that exist now that were really just getting started or did not exist eight or ten years ago and it's really great to have things like Splunk and Greylog and things like that that let you quickly distill down and you're sitting here writing shell scripts to figure out what the hell happened last night. And it's almost unfair to group the latter with the former but it's all of a piece with knowing what your application is doing right now and why it's doing it. So the next thing we're talking about, we've created the world and we're monitoring it and we know what's going on and so now we're going to try and improve it. And so, you know, you've done a lot of hard work to define your world to create it rather to define the environment, the OS, the static parameters, logging and so many more things but if it's just in one place then you've got to redo that work each time you're going somewhere else. And so it's really about using a provisioning tools and being able to know that, okay, my development environment other than perhaps logging parameters matches my staging environment which other than perhaps content matches my production environment I can make a change one place and know that it's going to work everywhere and so it will save you time and money if you get it right and by the way some of these quotes that we are seeing on the slides are coming from a survey we sent out and greatly appreciate the responses if any of you are here that did that. So the next thing is testing. I think the image is a bit current. I think everyone knows what kind of phone that is and it's really important to know things like this before you say spend trillions of dollars to push them out into the world. And so we've got an awesome environment. We're monitoring it carefully. We can deploy it to data centers worldwide at the touch of a button maybe even while drinking a beer but does it really matter any of it if the application itself is utter crap? And so I actually come from the sysadmin side from the ops side of this originally I'm going to put on my old sysadmin hat and say, nope, server's still up, my job's done. You guys need to fix your application. But hang on a minute, that's not how we do things anymore that's the whole point of DevOps, right? We're breaking down the silos, we're working in a team and so it's not time right now to pop the top and catch up on the latest bastard operator from Hell column. The job has just begun. And so testing is really, it's a whole world into itself we're not going to get into the details or start a war over which a test suite is the best or how to do testing but it's just, I'll just say a couple of things and one is that any test is better than no test. That's Howard yesterday who's giving a presentation on the culture of DevOps and he's saying if you take one thing away from this session yesterday he said just write a test, any test, get started that's like literally the most valuable thing you can do. And one of the biggest benefits is not even a completely quantifiable thing in our experience but it's that it really makes clients feel good about what you're doing, the development that you're doing, the application, and mistakes and bugs happen people understand that but if you can say look we didn't anticipate this, we're human but we wrote a test and this one is not going to happen again it really, it's emotionally pretty big to people in our experience. So moving on then I'm just talking about scaling and it's not just the sort of slash dot effect idea that if all of a sudden your site goes viral and you've got thousands of hits per second instead of per hour a day that you can survive it but it's about right sizing your world, right? That the resources and use are going to match your application's needs and then it will happen automatically, you don't have to go in and make it happen or forget to scale turn off four servers after the big event get hit with a thousand euro hosting bill a month later and so the benefits should be pretty obvious to most if not all, it's little things like cost savings and not having the site go down is slightly important and so the final phase and we've sort of had six phases in that security and the biggest thing here our takeaway is that with consistent environment you're going to be able to react with assurance if you know that your environments match each other then you can add and test a patch with confidence you're not trying to do this in production and hope there's not a chain reaction that takes everything down and once you've done the work once you're close to done, you didn't have to write down the steps because you've got your code as documentation so those are the phases of DevOps and so we've talked about the implementation obviously in a sort of vague do it kind of way we're not going to do live demos and things like that but we're going to zoom even further out and discuss a little bit about our philosophy of DevOps and this isn't the only way they're the only way to think about it but this is what's worked for us and been a useful framework and so it really gets summed up by what you're looking at now it's not all or nothing if you even just do one thing you've got automated test running or you've got configuration management going now you're an organization that has DevOps in its culture and a little really does go a long way and it's about saying we're doing this, this is we're an organization that does DevOps and this is our ideal that it's going to be holistic and so along those lines just go ahead and get started don't get overwhelmed by it dip your toes in the water maybe at the beginning of your next project when the budget decisions are easier then you're going to say okay this is when we install Jenkins and it's not that you feel like you have to have it all to do any of it and it's really not about the latest cool tools oh well first we were on puppet but then Chef came out and that's even better so we're going to install that you can have excellent DevOps with a set of bash scripts and it's all about what works for your organization and your people and so you need first the culture change just the idea that no one's excluded again from Howard's presentation yesterday Howard Tyson, he said one thing that I really liked and he said DevOps is not a department it's not this old idea if you hear the developers and hear the operators over here in a silo and the idea is that to me what that says is that DevOps is everyone's job and everyone's contributing even if it's just saying it would save me half an hour a week if this was automated and getting that started so this is wonderful we've talked about a lot of things I think probably most of you in the room are on the stage like this is great this is wonderful but you need buy-in right someone's got to pay for this and so in the final analysis how do we get clients and or leadership to sign off and write the check for time logs that look like Grunit with some vagrants for a while or played with puppets puppet that could be a real time log boss chef around varnish the application almost like a language and culture disconnect that we're trying to bridge and really just get that buy-in and get leadership interested if leadership is especially in larger organizations where leadership is not necessarily close to the developers and operators and so now we're trying to get to what you came for which is how do you relate tests like this to something the customer understands and values is one thing that I think really sticks out to a lot of people that's time and money and so we have so making that case for the business of value the business value of DevOps then we've really seen that increased efficiency lets us focus on quality our team really is happier when they're not dealing with my new show on the command line and that really has made our clients happier and so I really like this from Greg Nattison who's now at card.com he's another colorad and he says we're able to provide new features faster and more reliably and we haven't had a white screen after adding automated tests after having them about four major events a year before that and so when we're talking about the benefits for the client we think about having more robust systems you can do more to them without being worried that a house of cards is going to fall over time to market new features and the increased confidence in the process and what you're delivering is what I was talking about earlier and so they're really just we found that this assurance that clients get is not just something that we've experienced and so so far we've had some hand waving and we knew this is so but now we're going to really dive into the figures and because I was a music major I'm going to let Ron deal with the actual numbers okay okay so I think we largely agree DevOps is a good idea it's something we should be doing measuring what you do is very important and when you have which is all the things that Adrian talked about which are really very vague and very hard to attach a value to how can you go to management and say look we're going to need 100k, 200k invested specifically to set up this infrastructure to train everyone to use it when the only things that you have to talk about are very hand waving and there's no direct connection so we tried to come up with a framework that would allow us to quantify some of the gains very directly now there's all sorts of problems with this actually and I shouldn't be saying this before presenting the framework but it's very hard because there's so many differences between different organizations so one organization's saving is not going to be the same as another and the way we deal with that is we try to have some parameters in there so that you can play around with it but it's a bit like tests being able to point to a number and have a number and say this is what we expect the gain to be and then you can argue about how valid that number is and you can move it up or down playing around with the parameters is much better than any specific argument and because we're all doing pretty much the same thing we're building websites first we're building Drupal websites in particular it's actually much more transferable from the experience of one agency to another so what's our methodology Adrian talked about the different layers so we have six layers we're going to try and identify savings that can come across when you implement DevOps in each one of these layers we're going to allow the model to vary and we're going to do it across different projects and we'll see what comes out that's really the spirit with which we started this said okay let's see what makes sense, let's see what the numbers tell us and there's some interesting outcomes so the first layer is creating your world now an organization that doesn't use DevOps is there could be any number of things it did before how it looked at BlueSpark is you'd start by literally figuring out passwords or thinking about how do I get my key on this server so that I can actually log in and do things myself you have to go get the repository from somewhere you have to figure out where the database is how you access that, there's all this pieces of information that you needed to collect and if an organization doesn't have a way to share this pieces of information it becomes very hard and especially in our case where we distribute the organization you know maybe I live in Italy so I work in the morning and I do something and then in the afternoon I'm out and Adrian is in Colorado is like 7 hours difference so he wakes up and I'm already gone so unless we have a system whereby we hand over information it becomes very difficult so set up a wiki and you know a lot of if you go to any DevOps session it's not going to be how to set up a wiki that's not exciting but that's part of DevOps you set up a wiki and it has instructions you've already done something you can get a bit fancier and have like proper access control LDAP or whatever you know excites you what we are moving towards on our projects now is actually having a vagrant machine that has the entire project there and the only thing a developer has to do is download that definition, run it and it does everything from them it will go get the database it will go get the code it will set up any permissions and they can just start coding so this is where it's going to get complicated it's you know it's advanced math I hope you'll be able to follow so how do you find the cost of this so there's an upfront cost of defining an environment we said before you need to create your world then you have such like a fixed cost and then you have how many developers are actually going to work on this project and how much time is it going to take each developer to set up their environment so we took three of our projects and pulled some numbers first project we have 11 developers second one is 6 and the third one although it's a smaller project actually more developers worked on it and I should point out and this is probably, well I don't know I think it's maybe more specific to BlueSpark the way we work is people tend to specialize so they move between projects quite a lot so rather than having two back end developers and a front end developer and that's it people move across projects based on what the task is because they're the person that can do that the best okay so average time to set up individually this is without DevOps it depends on the complexity of the project the first one there's like language definitions it's a multilingual there's domains to set up there's thrush scripts that need to run with Cron so you find if people don't have a consistent environment they're messing around with just getting it to work and that's a huge annoyance but it's inevitable if you have complex sites it's literally half a day yeah I can now log in what should I do now and half a day went by we all know that can take place if it's a really simple project like the last one it's just going to be a case of figuring out where the information is and getting your Drupal site running and that's it if you have a vaguer machine you standardize it across all projects and really it becomes a question of how quickly can you download things so the bottleneck is literally your connection so I'm the slowest person in the company because I live in Sicily and not a good idea if you want to do web development did not think that through there are compensations well yeah the food is good the weather is fine so the savings overall are 46 hours across the three projects and you know you can argue about the specifics endlessly the way we go to these figures is just talking to different developers knowing what happened because it's projects we worked with and this is pretty much it but there is something there that we can point at and say you know you're going to have take project A 32 extra hours to do something else because you implemented DevOps well now you can translate that to money and say divide that across five different projects of similar size or similar number of devs to onboard and we can actually plan to invest that much and this is just on creating our world just getting set up we haven't done anything else yet monitoring our world and this is a really bad one if you don't have any system in place because it's essentially you have to get on two logs and start gripping around and you have people that love it like Adrian here there's all sorts of fancy things and there's people that are like I don't know and usually you know if it's very annoying to go check what's going on you're going to clear cash and hope that's going to fix it and then you're going to try various things that have worked in the past not being sure if they're the right thing or the wrong thing because you don't actually have any information but you can do things now and it's really easy just search it and there's actually a lot that have done it within the Drupal world you can have proper faceted search of your logs you can pay for something like New Relic and we'll see the numbers you can actually go and say this is why we should be paying maybe a thousand or two thousand dollars a month for New Relic or any similar service we not really advertising New Relic although it's very good we use it a lot so it's just being honest okay so the way we go to number is we looked at the tickets associated with the different projects and tickets that are set up as marked as bugs then we reduced that number of tickets by what we're calling the misrepresentation miscategorization factor which also translates to the client doesn't really know what's a bug and what's a new feature and but it was kind of an easy way to say okay and you'll see the numbers we don't really have that many bugs it's an easy way to say this is what probably were the actual issues that led to us having to dig through logs and then you multiply by the time it takes to dig in logs right so these are the three projects and again project A and project B are huge projects with think close to thousand or more ticketage so we have we reduced that significantly because that's a better representation of what are actual bugs although it's probably still not reduced enough to represent just real bugs bugs maybe then you a very optimistic time it would take to pinpoint issues we just said 20 minutes if you don't have a system in place and 6 minutes if you do that's really optimistic I think if you don't have a system in place it's easy to go to an hour 2 hours and so on and so forth but still even with this you can see that again you get on reasonably sized projects you get huge savings that's an extra 2 weeks of your life to do something interesting as opposed to trolling through logs what you don't like to say I do not and again a number that you can take and say this is why we should be paying for some service and that's we actually we had this argument with a client and it's a really it's a huge client but sometimes the way organizations are set up they can for example they can justify developer hours they can say yeah spend 8 hours on this and I don't know let's say it's 100 dollars an hour it's going to cost you 800 dollars right and you go to them and you say well if we set up a tool that tool is going to cost 400 dollars a month and it's going to save X amount of hours unless you show them the specific numbers they have difficulties dealing with that because setting up a tool is some different accounting department and it's not a line item they can pay for and it gets confusing so actually the easiest thing to do is you pay for it and somehow you pass on the savings to them having actual numbers is going to help a lot with that okay so that's monitoring improving your world and this essentially means sending new features building your world and then fixing it and sending new features and so on and that's what we used to do we would click click click click click click click click click click click done and now do it again actually on production or you know you'd have features and you do click click click click click export send it up click a bit less you're still clicking maybe in Drupal 8 we might stop clicking or or you actually click click click and then you do your feature and send it to GitHub and then you have Jenkins or whatever tool you have that runs jobs for you that is going to actually take that code send it to staging or production clear the cache do revert the features for you and all of that and you've saved a huge amount of time so the way we quantify this is to look at the number of commits we have group those commits because you don't actually do this on every commit right and then there's different styles of developers so some developers are going to work quite a bit and then send everything up others will send every single change but you can kind of group it by a certain factor and say every 10 or every 20 commits that's an actual deployment that's an actual sending code to staging or production where the developer is actively aware of it because obviously if you have Jenkins on every single commit you are deploying but what we wanted to identify is developer actually stopping and saying I need to go make sure this is all fine and if I didn't have Jenkins this is the point where I would actually go and send the things and test them so you get the number of deployments and then you check what we are quantifying is how much time it would take you to check that things are actually where they are supposed to be and they are largely working and this are the numbers we got so the commits are 3600 for the first one 2500 for the next one and so on and again very very optimistic saying that in order to check validity it would take you 8 minutes without DevOps and just 2 minutes with DevOps and again the idea here is to take this framework and plug in numbers that are realistic for you and see how much time you save and we just got another week and a half of time because we did some simple things at the level of improving the world so you get the idea I will go through the rest a bit quicker testing yeah what we do for testing now in most of the projects not all of them is BHA tests for key scenarios so it is not exhaustive so there is a lot of space for improvement and the way we quantify it is to say how many tests you have the average time to perform the test and the number of deployments now the figures here are really interesting and they just prove a specific point very clearly so the assumption we made as we were thinking of the framework is let's say that you are really committed you just don't have the tools but you are really committed and you are going to test anyway even if you don't have the tools so you define all your different scenarios you send some code if you are really about quality assurance you are going to go in and do all your tests manually because you have no other way or you have a tool to run those tests for you and the point it proves very easily is if you don't have an automated tool you are just you are definitely not ever testing unless you are spending 2,000 hours if you have 57 tests or 6,500 hours if you have 108 tests on a project you don't fool yourself that you are doing anything close to QA or testing or anything like that you are just not doing it and this is on the basis that it will only take you 10 minutes to go through each test and you know especially in the second case those are complex flows that e-commerce related and there is a lot of different interacting components so if you don't automate these things you are just not doing it ok then scaling the the last two are scaling and security and these two are really they can impact very much but it is very difficult to quantify them it is truly unless you are keeping a very specific track of logs and you spend a lot of time to analyze the projects it is very hard to identify how much time you spend on this or unless you have a huge issue the way you would quantify this how many performance issues did I have and how much time did it take me to react and we just plugged in some very basic numbers and came up with results essentially we are saying it is going to take you about 3 hours without DevOps faster with them because you are probably doing auto scaling or you have a way to set up new servers just by running a specific script securing again one that is really hard this is our world and security is one that there is nothing you could have done for hard lead or shell shock but at least if you have something in place you can look cool while you are dealing with the inevitable problems I apologize it was his idea and the way we quantified again number of security issues average time to react don't focus on the actual numbers this is just something for you to take away and then plug in your numbers for your projects that you know much better okay so you say you did all that what would be the result what comes out and we have the world view so how much time you would save overall and again this raises a very specific issue testing and it's no surprise that the whole move to Jenkins to start with especially in the Drupal world started when people really were testing because there is just no other way to do testing I mean without DevOps it's 9,000 hours so we actually took testing out to get something a bit more useful because essentially as we said testing without DevOps you're just not doing it so you take that out and you see that you can save over three projects of which one was really small the others were let's say larger medium to large projects you have with very optimistic and super efficient developers they're very committed and actually click real quick you've still saved 260 hours now 260 hours you know perfectly well what value that has for you so you can take that number and say just over three projects I'm going to save 260 hours is it worth me investing to have DevOps in place and if you take this to management and they're still not convinced get new management yeah pretty much okay so to sorry to add another dimension to this we also did a small survey and send it out and ask people to give us some feedback about the challenges they've had and so on we'll pass it back to the musician to do this I'll cop to that and so we just like to share a few of the most interesting results of the survey and it's actually it's been really great to have some of the long form responses that's kind of validated some of the thoughts we had while putting this together and so it's obviously it's not scientific it's not a controlled random sample we send it out we put it on the Drupal DevOps group and we send it out in our twitter account so it's pretty narrowly targeted at people who already have some interest in DevOps but with that said then we find that the results are interesting in spite of that and perhaps even because they come from a group of people that are already interested in DevOps and what we see within that group and so we find that only about a third of respondents are actually using a Drupal specific hosting service you know we're thinking of things like Aquia hosting, Pantheon the commerce guys platform things like that and so I suppose a lot of people they're thinking the market is open anyway over three quarters of our respondents are using some form of DevOps practice so it's really out there it's there I mean if you put that in contrast next to the results we have with our quantification framework then you might you can say as a shop if we're not doing DevOps then we're getting behind the curve and it's a competitive disadvantage and so when we're talking about whether you do it or not then certain shops are just saying top down that's it we asked about people using third party continuous integration services and that was quite low less than 15% of our respondents are that said those that are are pretty bought in completely changed their workflow for managing the deployment of Drupal applications and I'm sure we moved a lot of sort of boring time from their lives we asked you know did you have trouble getting DevOps implemented did you face pushback and people said really not many did again and this is a narrow sample if you broaden this to anyone in Drupal that's tried to do something with DevOps and that number that percentage probably grows a bit but I mean it's not that scary if you're there saying thinking oh I really want to convince our leadership to take this on then maybe it's not as hard as you think again from Greg Nannison he said it went all the way up to the CEO and so he was resistant to automated testing and now accepts that really does get him what he truly wants even though the line item doesn't look like what he wants and then other respondents said that that they've really seen a significant percentage savings in troubleshooting issues having things like we talked about log creation and tools that make it easy to quickly narrow down to your real problem and finally we asked people about what were the biggest wins they saw what really gave them the most bang for the buck when they were looking at different phases of DevOps and different tools and the things that really stuck out that we heard again and again were configuration management automated deployment and quick easy scaling and easy in the sense of not that it's necessarily easy to set up but once it's there you don't have to tinker with it and so if you're wondering where to get started and then these could be a good place to begin and start building institutional confidence in your process and what you're doing and so that's really it for the survey thanks a whole lot for being here and we've got a few minutes for questions if anyone has anything they'd like to ask us and stunned you to silence I guess so thanks so much we'd really appreciate any evaluations that help us get better it's up there on the Drupalcon site so we appreciate it thank you tell us a little bit about what you think is going to change in Drupal 8 yeah absolutely the biggest thing I see coming down the pike with Drupal 8 configuration management initiative features it's it's been a godsend in the sense that without it then you have a whole lot of trouble really deploying anything configuration wise with the Drupal site but it's not what it was built for and it has some pretty significant limitations one of the biggest being that contributed modules have to take the effort to integrate with it so it's not baked into the product and so that's going to make a really big difference anything else in your mind I think the one thing that is probably unexpected that will change with Drupal 8 is the way people build websites with Drupal period and that's not going to come immediately but because Drupal 8 allows you to decouple one of the content management framework really from the UI that runs on top of it and it's the Drupal we know I think you're going to see that in one or two years after it's released people are going to have many more examples and not be scared to build very different looking websites and that's obviously going to have an impact on how you test and how you manage this applications because they're actually going to be decoupled and easier to manage on their own and I'll say from a deployment perspective if all of a sudden you're thinking about things like headless Drupal and it's not you know here's this neat trick I did with Drupal 7 the dog standing on his head all of a sudden it's really basic that you can build applications that are more modular and so you might be easily thinking of doing things like putting Angular apps that are out there and distributed data centers around the world and you're really seeing big improvements in your response times because the code is closer to the customer things like that and like Ron said I think people haven't really started to think through and imagine a lot of that stuff yet so there are big changes coming do you mind stepping up to the microphone so it's on Lea, thanks what do you think about having the developers, Drupal developers doing the DevOps or having a dedicated person to do that or persons I really think it should be all of the above and as I was saying a bit earlier then the biggest piece we see is that it should be a culture and it's not and you know maybe you have one person who's really interested in it that's the thing and they do a lot of the implementation but it should be a mindset that anyone's looking for areas to improve things that, oh I just did the exact same thing ten times in a row this is stupid, you know let's change that and so that's what we really encourage is for everyone to have investment did you look at any other monitoring tools or did you just pick them up specifically New Relic yeah no again no relationship happy customers we looked at a few but I'm advertising it again it's your fault it integrates lots of different pieces of information really well so it just made a lot of sense well I can I can say that the alternative to it that we know about and this could be poor research on our part but it's sort of getting a lot of tools so like gray log would be an open source log aggregation but that's just logs and then sort of a good MySQL monitoring tool you'd need one that can do stack level analysis that shows you sort of response times what's taking the most time through the stack in different pieces that it's it's a good way to pay for a lot of things at once basically cool thank you everyone thanks so much