 Ladies and gentlemen our next and final speaker in the system track is spike Marelli who will be Doing a talk entitled. I'm going mad. Thank you very much Is it working? Okay. All right. Sorry about it mess up. I've gone mad already So I'm going mad. It's uh, I wanted to share what what I've gone through in the past Six years from being a completely a system administrator to do in development and then from being a full-time developer Going back to do system administration and how that worked out for me How many developers are in the audience? Operations folk Do we have any QA people? So all right So I Can say that this will work for you, but it has worked for me pretty well In in my various jobs, so I'm going mad. It's monitoring aided development and One point that I start like to start with is that we all Code and And even when you start with your one liner on a on a bash shell you are coding You are effectively coding and then maybe your one lines and become not enough and you move to a bash script And then you can go farther and start to do pearl Python Ruby anything that you want But you all code Even if you are an admin your coding if you are a Q a person testing automation is becoming bigger and bigger and Testing automation requires a lot of coding and Also, we've seen a huge uptake with a configuration management lately when you have a configuration management either if you use Pappet or chef So at yourself writing a ruby you're still coding. So we're still talking about code But somehow especially for people in operations where I where I come from you don't feel like a developer you're not treat as a developer and This is this is hurtful Because this is one of the reason the fact that operations and development use to complete different languages And don't recognize each other. It's part of reasons why in organizations. We have these kind of problems and These are snippet of a map of Pappet manifest and as you can see it looks a lot like a piece of code So if we all code We should try to talk to each other more than we do and we should try to learn from each other More than we currently do and it's a it's weird as a as a thing of not feeling as a developer as a system as a system person Why don't I feel as a as a developer? Why are my scripts not treat as it if the world? code When when everybody when people start Developing and they were the first hello world. How is there anymore? It's kind of be about complexity because a low world is not more complex than your automation screen and nonetheless You are kind of full and feel as if you were a developer when you're writing your first hello world People in operations are still the people that I know and met in my in my career. Do not feel ever as developers They feel as something completely different Now monitoring metrics is a lifetime love for me. I've since I started doing a My my career is a system at sys admin the fact that I had nagius and then nagius checked all my systems For me it was also the first day that someone showed me that nagius. I was wow Now something is checking things for me now. It's I can sleep much better than I used to and then this this huge uptake there is when from monitoring I discovered metrics and metrics is a something that it kind of is if It gets talking of monitoring talking in metrics some people use one term Rather than the other They all kind of belong to the same world, but effectively they have two completely different meanings Metrics tell you how a system is doing monitoring tells you much more like is the system okay is the system broken and If you if you think about medicine I Think about the metaphor of a doctor Doctors for our in our Western culture they they come in when you get sick So when you get sick you go and see the doctor and the doctor try to backtrack what happened I you were doing in in the last ten months and figure out what might have gone wrong and When I went to China and I and I had kind of a I got to see how all Chinese doctors used to used to work A Chinese doctor always follows you it follows you when you are Okay, and you pay them to keep you healthy and when you get sick That's when you don't pay them because they failed and in this way metrics to me are a much better way of Understanding looking at a system because you kind of follow the system rather than just getting to know when something is broken or something works and I had them I spent some time in in data warehousing and in data warehousing I met this guy that was a Pure data geek at its best It has this huge piles of data heaps and heaps of data and it could pull out incredible things out of these heaps of data and that's where I got my my love for metric metrics comes from Comes from these experience and this guy had Had a great message and one day he told me there is no such thing as too much data only data You don't know how to make sense of and this is very important because Obviously, there is a cost in having all this data and having all these metrics. There is a data analytics Dev room which was really interesting and they they talk a lot about how you do data mining and what you can get out of of that data and Certainly, it isn't a simple task but that task has huge payoffs and So sometimes you come across people that says well, why would I want to keep certain metrics? Why would I want to you know? I just keep CPU or I just keep memory There's few things what's the point of keeping more than what's the point of keeping them for longer, you know They're gonna take all this space and then this noise How am I gonna find all the good informations that are hidden in all this data? So why the but why there is a cost then that point obviously is is partially true There is a costly, but it's important to recognize the benefits and So certainly you have to mind information overflow There are no free lunches. You cannot expect that by starting collecting data Instantly you'll be able to figure out everything about your systems that you didn't know before and There is certainly is a Risk that you will be misled by all those metrics so I Didn't I didn't test my code as a system guy when I started I did no testing whatsoever Obviously, I would run and when I'm saying testing I'm kind of talking about unit testing to begin with But mostly I would run my scripts and test them maybe on at that time that barely were VMs Maybe some send beginning up send And you would do some of that and then you would have a box in the office to run some of this stuff on and that was Mildly right, but you know you need testing nothing because that's not how we roll in in hops apparently and Then I I met so I changed company a change job and I went to the company where they were Better developers and not just because there were better developers and better. Maybe is a is a I'm abusing the term but they were More friendly toward operations and so they really believed in testing and they wanted to talk about it And so they were doing like talks doing lunch and various things to which we ended up going as an operations department and inspiring me I got inspired by how they were approaching softer development How much value they were putting on testing and how much they were getting out of this testing? They had metrics to prove that testing for them worked really well They produce better softwares fewer bugs. They had less problems in production And so I started to kind of wanted to do this testing, but then I still couldn't because Systems don't do testing at all But then I moved on as I started my own business and I and I wanted to actually make something And I guess it's the same it's happening the same even if we freeze after projects all the new free software projects are coming out these days They all have tests and they actually go on how much coverage will we get to it? They get on the tests so I started to do something and the amazing thing is that as I started to do testing I realized that actually in a way as a system person. I was doing testing already and When we started to do TDD, I realized that I used to do TDD with Nagios Because you would bring up a service in Nagios and a service check in Nagios and then you would bring up Nagios as a monitoring system and then you would bring up the system and you will see the check pass Which is what people do in test driven development to write the test first They see the test fail in then they write the code to satisfy the test and they see the test pass and that gives them a certain amount of certainty that that code is correct and Test driven development is It's really interesting for for many other reasons one is security. I've realized that sometimes developers don't care too much about security depending on what they're doing and these are there are always fights between operations and and Development because developers maybe don't care as much as operations would like them to do about security TDD is great for security because you can write tests With in mind like user input So if you have a function that validates email, you can write a test that checks for escape carters in your email So TDD is a great way to do testing and so when I start applying testing I also started to record the success rate on my test. How was that in testing and I had this kind of thing coming up. This is more the beginning of a project and This graph tells a story that you will not be able to see if you just looked at some numbers at any given time And what this graph is telling me is telling me that I didn't do as much testing in the beginning where I was Introducing unit testing and then I got to tag zero one where I wanted to kind of do a release And I actually matched and all my tests Passed and then I diverged again. I was at the test but some of them were failing I wasn't really carrying and then again another release and I matched it again And now there is an interesting thing happening up there I'm not diverging anymore and I'm not diverging anymore because I've introduced TDD and So I'm writing the test before writing my code and I'm not forgetting When I when I you can set up in in github mercure or whatever you use You can set up your hooks to run your tests your unit tests When you do a commit and reject a commit if the test don't pass Now as you get your tests and you think okay, well, I'm adding this test. I'm doing well But then you have the question of how much does this cover of my of my code base? Because it's it's useless if you're adding a lot of tests for just one small portion of your code and Then you can start to measure this as well and graph it And you can get something like this and something like this at a first glance again it says something and You can see that you didn't have coverage You've gone up in the beginning and then you kind of got up to the 80 and then you've got down from 80 So you've lost coverage probably because you had the code and you added some tests But this test weren't covering your code and then you go up to a top and you will you're reaching a hundred percent coverage? And it's interesting that you've reached a hundred percent coverage With Tago too, but you've added almost no test just a few tests So that is an interesting thing and we'll come back to it in a minute And then you start thinking well, I've got this coverage of what I've got this test. Are there any other interesting metrics? How big is my code base? And you could say it's lines of code Well lines of code is a really bad metric because how big is your actual error code is not really what you're interested in But it's a good starting point. So you kind of go. Thanks, but no, thanks If you start to graph it nonetheless It says something interesting again because as you can see you got a number of tests and your coverage and then your lines of code going out And now do you remember from the from the previous light that I had achieved a hundred percent coverage on Tago too, even though I didn't have many tests Look what happened lines of code it went down Why it went down? Because I refactored So in this graph having these metrics recorded and having them graphed Tells me something that I would not necessarily catch if I didn't have something like this So at this point, I know and if you if you think about you know over the span of a year You could pinpoint every time you refactor every time you change something big in your code But obviously as you said Linus code is not a good metric What you're really interesting is complexity, which is your enemy. You don't want complex code Because you're more likely to reduce but the same way that you know one complex systems because they're more prone to failure But how do you measure complexity and the things that I started thinking with is Something some ideas that I got from from spam assassin and spamman Simple scoring so you begin to think, you know, how do I assess? complexity and what metrics can I use to address complexity and So one thing that you can start to do is call graphs Call graphs so you can use there are many many libraries that will scan your code And it will tell you a call graph So which function is called in which function and it will build a tree and so by there You can see the amount of nesting now if you have a lot of nesting where you get Functions, then maybe you get like a four five Subseen six ten layers of nesting that is really bad that counts That is a metric that you can use to score your code as a complex code Number and size of functions. There are people that will limit that will impose Abnormal limits on the length of your functions Because they say long functions are hard to read and again. It's easier to introduce bugs So you can use that as a metric to judge the complexity of your code The other thing is called closure now Cook closure is something I came across recently with Michael Fedez. He made a post on his blog and He was saying if your code is good code You tend to not change the same files many times You tend to add new files or add classes if you think in terms of object Programming and so the idea that you extend your classes You don't necessarily will attach classes that you've written before and so he graphed his his commits on github and You could see that a lot of the fires were never touched again He will introduce a new fires or other functions and then he had a couple of areas Where he had like hundreds of changes and so he targeted those places for refactoring So it's again a metric that helps you write in better code which then in turn It works better in production and keeps your operation happier And again, then you have Complexity of the build system if you have a building a complex application You generally end up with a complex build system So the complexity of a build system is a good indicator and can be used for scoring to judge the complexity of your code and again here a graph of those things and you can see that complexity and this line is Resuring because it tells me that despite several changes my code complexity has not gone out And also it's important to do it with style style can also be a source of metrics How good is your code? so for example if you use stuff like lintian which is a It's a checker that will run through your code and have a lot of I'll tell you a lot of things about your code If you're naming the way you're naming for your variable makes sense If you the length of your lines all sorts of things that you can pick up and there are specific ones like pap 8 I use a lot of Python. So I'm using pap 8 and Then reward beautiful code. This is another thing that I found really to be really important. This works Works both for operations and for development Especially for operations. This is really important This is the thing that I picked up in development Developers tend to reward beautiful code. They tend to reward good code. So they put Value on doing it right Operations if you're doing if the operations doesn't write nobody notices Culturally that is how it's expected and that is all is also harmful because nobody is supposed to notice that and Nobody get gives value to what operations is done directly and this creates conflict between Operations and and development because operations gone. Well, they get all the credit We don't get any credit and so you create these these animosity now The thing with metrics is a People complain the biggest complaint I've heard when I talked about metrics is that as soon as you introduce a metric people will start gaming it and All the companies that they try to introduce math metrics, for example to judge Developers how good they were doing how good the stuff was have more or less to a certain extent failed because Developers will start gaming Those metrics just to get a raise at the end of the year or something so There are problems with metric again. There are no free lunches It's not just the cost of analyzing the cost of rising the reason a mindset that has to be changed In order to make good use of metrics But they are extremely powerful and then of course all the stuff that I was doing and I was doing it manually Then you shouldn't be doing it manually Really and so I started to use a CI and I stopped exporting data What I was doing commits or looking at my commits and scanning my code And I started to use in Hudson that has been renamed Jenkins Build bot is great as well, but anything that is continuous integration it is really useful and continuous integration is also being picked up by by operations For for system scripts. There are people that use continuous integration to run through the Papa to chef manifest again all these configuration management stuff has changed a lot the stuff in operations But in doing these I kind of forgot where I was coming from Because I kind of got really excited about the metrics and about the development And where I came from is this and this is really bad and if you haven't been on call You probably don't realize what it is to be on call when I wasn't on call in my first year of system administration I didn't really know what I was talking about the first time that I went on call it really shook me and When I had this I had a specific ringtone on the phone that the company gave me and we couldn't change the ringtone After I left the company when I was in a public place and someone would have the same ringtone. I would twitch So you kind of have these things that you know it really gets into your life It's really difficult to to kind of it being on call really sucks big time and so you want to avoid it and One thing that you can do to avoid it and one way one path that has worked for me is to Introduce metrics because how is greater than if and It is never too early to start monitoring your applications behavior and this is key. This is where Operations and development can start to collaborate much more Operations can bring to development and this is happening to a certain extent in a new development environments Put monitor in those development environments add more than track CPU usage memory usage and add those metrics And those metrics can help developers because again We do TDD to catch for example TDD is good for a factoring because you know that if you have your tests and You refactor and you break something you will know Now how about using metrics for CPU usage or memory usage to figure out that when you're a factored you actually Introduce a loop in your code or a memory leak Wouldn't that be useful wouldn't it be useful to figure out that you've introduced a memory leak before it hits production? Before you finish the sprint if you're doing agile or whatever you're doing and you get Two months later a month later to run Maybe you are still testing and you're still doing load testing and you figure it out before production But it happens later and you have to go back and kind of figure it out how it went wrong where it went wrong So monitoring and having metrics from day one Can be really really helpful and it has helped me directly in in some of the code that I've written because I'm still not a great developer and I made a lot of mistakes and Testing I saved my life in many cases and having metrics that saved me in many many cases I don't think that can be done is to write code that it might or if friendly This is another thing that developers can help can kind of come together a point of contact between development and operation in the in the sort of DevOps kind of Couch would change and here guys. It's a small flash cap and as you can see I've got a slash month slash status as Lash month slash self-test and as Lash month slash metrics So if you have stuff like that, I can point my Nagios my monitoring system to those kind of things and I can get very easily very simply a Status of an application or the performances for example if it's a web app you can store the last a hundred written code in a in a memcache and this is happening a good deal in In system tools like if you think of memcache For example memcache as you can tell it to the port and you can run a status command And you get out a list of all of the current status of memcache Which is really useful to judge How your caches are doing? And the bottom line is that ops is changing operations is changing Configuration management has made a huge difference in how operations has been moving in the last couple of years three years and We're closing into something that looks much more what developers are used to and this is really important because again in These conflict between operations and development in many organizations have One of the big problems is the language that both parties are speaking and the fact that both parties can speak code Is greatly helping to reduce that divide? Configuration management also has given birth to these infrastructures code sort of thing Which basically means that since now my how my systems are set up is Close to writing a piece of code now my infrastructure really can be represented with code Which can help in in this process and then you have behavior-driven development, which is something fairly new But it's something that developers love there are a lot of developers They really like to do behavior doing development and they use you the cucumber or rubber framework Those are applications that will allow you to write in In native languages like in plain English or whatever is your language in natural language You write your test and you say something like when I connect to such and such page I expect such and such output and those when and expect a Keywords that tools like a combo rubber framework or lettuce if you do Python will know how to interpret and Convert into a test. So now you have something that developers are really happy with they love to do BDD and Now operations can use it. There is a plug-in called Nagios cucumber. They allows you to run Tests written in natural language win Nagios to monitor your application. So now there is no longer this kind of a Developers write their own tests and then they pass it to operation Which have to rewrite the test into something else to fit whatever monitor infrastructure they're using the two groups Can use the same language and the same tools and Then continuous integration as I was saying it's already happening in ops and having these things in Common can greatly help with the dialogue And so you can help but how how do you help if you are not? Realize and accept that you code Don't think that because you are in operations the fact that your operations justifies No testing not using unit testing or using the serve paradigms that Developers are using learn from your developers Understand how they do it why they do it what they do and adopt those kind of techniques. There's lots of good stuff and Advertise your achievements. I touched on these earlier developers are generally Identifies as the ones that produce the features produced of what is sold to customer So that is what is visible operations are never visible and so start to advertise your achievements start to talk about it and Engage your developers in my experience when I start to To actually go to developers and ask well, how could I test? Like my my scripts there were more than happy to talk to me. So really it's not that they don't care is just that They're speaking a different language and there is a hurdle in in getting over that that diverse So if you ask for things that they recognize as familiar, there will be more than happy to help you If you are a developer Three ops as developer understand that they're writing code and recognize that Share the knowledge how you're doing again do the opposite what ops are supposed to do in terms of getting in contact Code applications. They're easier to monitor like we went through and learn from operations like Type type into the knowledge about metrics and monitoring because you can be really useful And the most important metric this is something that I stole from from Patrick Stokes Patrick the boy gave talk about Dev ops in in London last week. It was a great talk and it was all about cultural no tools and His stock was about trust And trust is the most important metric Trust is the most important method because if you're trying to get these two groups together to talk to each other There is a there is a gain to be made there because if you if you have like 10 people and each one of those people can in theory Produce 10 of whatever unit of work and then one of those people doesn't trust One of the developer doesn't trust one of the ops people Then they will hold their work Waiting for the ops of the group that they were like or maybe they will install their applications on the road They become a bottleneck. So your production goes down Because you're not trusting each other and to close. So don't let uncertainty drive you insane. Go med Thanks everybody Questions no Thank you no one well, no, there you go Okay So there's a little part of your present edition I didn't catch because what is TDD? What what is the theme TDD? Oh TDD. Sorry test driven development is the fact that you develop your tests Before you develop your code. So right then then write in your code and then you go on and says well What really should this code look like how it should behave and then write in a test? You write your test first and then you write the code then make that test pass That is much more likely to guarantee that you will have all your tests in place So all your tests will cover all your code Does that make sense? hi could you speak to the automation of tests in a deployment kind of scenario So the automation in test and deployment sure with The thing that most people have seen doing and I've done myself Is done with virtual machines? so What you end up doing is a Generally try to spin up and create with virtual machines environments from scratch Which is what sort QA is used to and then you deploy your code So at some can do can deploy your code to to any machine that you want to so could be bought So the idea is that you use something like Even at some for example can spin up an instance if you want so you can tell it to Create an instance with saying KVM or in the cloud if you want Deploy your code to it and then run a script and that script can run all your tests and give you all the things if you're doing other things like We Continuous we continuous integration you could run metrics on the same box The outside or whatever is the continuous integration that you use runs on that is perfectly fine It is not that good to do integration testing. It works. Well like to get metrics from unit tests Not so well for integration testing from integration testing really you would want either develop the environments They're on demand or environments that you can at least clean out between runs because you don't want to reuse the same environment twice Does that clear the question? Good Yeah, I think we're good Wow My question might sound a bit cynical, but do you think it's really possible for developers and operations to trust each other You said you spoke to some developers when we're in operations And they were really interested in helping you Making their applications more testable and everything, but my experience is exactly the opposite What the operations and most developers don't want to talk to operations want to just cold and do nothing else And they have their own idea how it should work Which is really and it's really hard to get to them and to explain them how the real life works Well, so the thing is that I'm not I'm not saying that Developers should behave and take on responsibility of operation. I totally appreciate that as a developer I don't want to know in fact when I start coding I'm bothered by the fact that I have to install something or take care of something because it breaks I wouldn't want to do that. So I appreciate that but I'm what I'm saying is that in terms of How you handle certain problems That kind of thing can happen transparently. So think for example continuous integration. It's a good. It's a good thing where Both operations and development contribute to That system and then without having like so say as an operation I contribute to the fact that when your continuous integration spins up an instance on Xenon KVM it installs monitoring it installs monitoring that monitors everything that's happening on that box as a developer then I Deploy my application on on that box where I have my application end up on the box And I don't have to know that those metrics have been collected and how all I care is there I get that feedback So I don't require the developer to be involved with creating that environment I'm saying that developer has a as an advantage if we take Those metrics he looks at those metrics and takes what operation can give him So there is a collaboration that is possible without requiring either party to actually learn the in the details of how the other the other Side is working You shouldn't need to to know the in the details of how to set up nagius that is irrelevant But if you think like for example in we be a behavioral driven development and writing a test in natural language In natural language, there is a good example the developer doesn't have to learn anything about how nagius work He writes his test with cucumber with Robert framework and then he passes the test to operation So there operations and development they they give to each other they gain from each other without actually having to learn anything in terms of The underlying details does that make sense? No, go ahead. Why doesn't why doesn't it make sense? because in the end From what I've seen developers live in a different world Can you speak a bit louder, please? Sorry, so from what I've seen developers live in a different world and they Sure, but is that good? No, right. Okay. So if we agree that is not good. What is the simplest thing that you can do? to try and fix that and In my from my experience the simplest thing that you can do It's kind of to congregate around testing and congregate around common tools Because as the fact that you're using completely different terminology and things it Contributes to the fact that as a developer. I don't want to know about all that stuff So your idea is that the two worlds of developers and obstacles Can come around exactly around testing can use that as a intermediate language. Yeah You use you use use testing as a as an excuse as a common language to talk about What has to eventually happen in production? Is it better now? cool, I Think there's one that just a practical question when you talk about The scripts that you write and the testing that you apply over them and all of that What's the which are the languages that you use which are the? Tools that you used to unit test those Scripts or those well pieces of code or whatever you want to call them Sorry, I catch I didn't catch the beginning of your question when I spoke about what? When you when you talk about the code that you write the core graph Yeah, the code that you write so in which language do you write that code? What which kind of things? Does that code? Do and what what problems that does it address and how you test it so which? Which framework or whatever it do you use to unit test? That and to even to record those those unit tests the results sure, so I developed mostly in Python So the tools and the things that I use are pretty much all around Python and a bit of Ruby these days But still so for the core graph and that kind of thing I use pie core graph Which works really well and now puts in in dot format so you can even graph it with graph bits That works that works really well. I mentioned pp8 to do the and lintian to do the style checker on my code and I used to use build bot for the continuous integration I ran in problems configuring build bot and At that time I couldn't really be bothered with figuring them out And so I tried that son and that son seems to do all it in a in an easy way So I switch to that and the other thing that I use I I use a KVM on my box to There's Python lipvert to drive lipvert is an abstraction on top of KVM Well on top of every possible known visualization visualization system accept arbitrary ones So you can from Python you can drive for example creating a new virtual machine and then deploying getting absent to deploy code to it and then run all your checks in there for the behavioral driven development I use Lactus, which is a Python clone of a cucumber Robo framework is also very interesting and it's probably More known than Lactus on sort of the big level you can it is more powerful, but for for simpler stuff I would say Lactus is much more approachable. So I would recommend that at least to start with What else is there to store metrics I've done a few of use a few different things I'll try to use SQL light because I didn't need anything big and it was mostly just me So I use SQL light to store all the metrics in in the beginning and then For other metrics and system metrics. I used to use RAD tool and then I moved a lot of stuff to graphite just because it's simply the problem with them We grow with RAD tool is that it expects Things to be in this in the exact time series and if you miss a certain certain times lots It will give you troubles graphite is more tolerant in sense sporadic events Which is what kind of you end up doing when when you just sends metrics when you do commits and similar things I Think that's that's about it. That's pretty much all the tools that I'm using Oh, if we unit testing I use the built-in unit test framework. So Yeah, the unit test that that comes with with the standard late Pi test is a school as well They have a bunch of up a bunch of advantages the other thing you might want to look at its talks to your acts Which allows you to To set up different environments would even different versions of Python So you can run like concurrently you can test on 2.4 2.5 2.6 and 2.7 and 3 all of them and test all your code And that is also interesting because you can collect metrics It's in our case where metrics can be very useful You can collect methods and see how your code performs on different versions of Python So that can be interesting because you might say well, I focus on my development 2.6 because they get better performances Yeah, that's all I think we're done done. Yes No Are there still some more questions? Otherwise, I would say thank you There's one more Hi Have to implement application monitoring in a quite heavy corporate environment And I'm facing the the position of I would say or the reluctance of the operations people to send so much event so much Data into their their system Do you have any comment on that? So obviously if you have a lot of data is problematic It can be it can be a pain all this stuff I've been doing it mostly for myself. So I had as more necessity. I've used to deal in the back Couple of jobs ago with with a lot of data and we had like huge minuscule clusters or HDFS Also used that it works pretty well. There is a there is a project called Open Open HDMS which has been launched by The what's this called? Not to eat at people D and I dig one of these big company now I'm forgetting name which is basically our D. So you had you get kind of the the sort of thing that run robbing databases do and An RDD tool does in terms of you know giving you graphs and all of that and storing metrics bar on HDFS So that's quite interesting. It gives you a back-end as a file storage that scales really well And at the same time you get you still have an interface That it's that you would manage like you would not kind of normally interface the RAD The other interesting things that you can do if you want to use Sort of our D base tools now our D supports our D cash D Rd cash D Caches your reads and writes and you can chain them So you can have different boxes the store different slices you sort of partition your data and then you have sort of you ask all the Rd cash D and you cash that and And So you have like one master node in chain to all the others and that kind of makes you make scaling Rd Easier and I'm fairly possible fairly feasible Otherwise it depends it really depends also on what kind of data I mean for some things you could think of using for stuff like this probably a key value store will work really well because at the end at the end really Metrics we're really talking about a label and a value and a point in time You know you have three variables. You just have you know the point I'm so using something like a redis Works really well ready some Cassandra and mango Depending what you're doing for for metrics. I will probably use redis Over mango and the thing with Cassandra. It works really nicely But it comes with a kind of a heavy luggage Because you have to get old thrift and other things coming out of Facebook, which are really nice But then you have to maintain them and so you read this in that sense is a be a more lightweight Anything else? No, thank you very much then