 Hello, everyone. I guess we start. I still see people coming in, but they will catch up. We are Swiss, so we are used to punctual starts. So first, who am I? I'm Bastion. I go by Dasrecht on Twitter and every medium I can find. If you want to know more about me, you can find me on bastionbidmer.ch. In the Drupal words, I work in the modernizing test bot initiative, Drupal CI. I help there out whenever, whatever tasks I can do. They told me that I need to write down my job title list and I'm the chief, you only live once operations evangelist. So I'm that person who tells your customer not to use Git or something and I also tell them that you deploy on Friday afternoon right before you go off for weekend. That's my job. Of course, I'm not doing that, but it's like, that's what we do. So we have a quite dense program today. I start with a little introduction. Then we go over to the ELK stack. I cover the architecture of the ELK stack a little bit. Then I head over to tools because there are many tools you can chain together and if you get started with the elastic search, it's really a good start and I try to make it as easy as possible for you. There's also some automation I'll cover and I invented the number P22N for performance optimization just because it's too bothersome to write it down every time I just say P22N. So visualizing lock files, why? Michael who sits in the front row here told me like, hey, can you check the errors from yesterday between 1502 and 1507? It's like, okay. Are you kidding me? The second action I had was that. I got all the lock files together and started to grab and chain tools together on the bash and I got a file and then I just removed the stuff I don't want to see and it was like, yeah, I had a file but was it really what I was searching for? And the next question would be like, oh, yeah, and can you go two minutes before what you just did and then the whole process started again. So what I say is visualization is better than plain text, right? Right. So that's what we deployed a patch some weeks ago and basically that graph you see is the error rate of our web servers all the time and you see exactly when I deployed a patch, the error rate goes down, everybody is happy and we know that we fixed an issue. So in my opinion, visualization is much better than plain text. Actually it should look like this. Who of you is grabbing through lock files sometimes? Who hates it? Oh, yeah. So welcome to my session. So do you lock your watchdog messages to the database? Who does it? That's good. That's okay if you have one site. If you have 70 sites logging a lot of data all the time to the database, you get a lot of database traffic which is basically unnecessary. And what happens if you want to have a lock file which was already, watchdog messages which is already truncated. It's like, yeah, okay, it's gone. So for those use cases or for elastic search, there are a lot of use cases. For example, client calls you and say, hey, you deleted page whatever on my website and then you say, okay, let me just fire up elastic search and you do an audit trail. Who deleted which page and you see, oh, it's your user which deleted it. So I'm sorry, it was you. You just click delete. It's also good to know when was a certain module enabled or disabled or when did it yield an error. Also what I told you earlier is the drop in the errors, you can just see things without like counting, okay, we had an hour ago, we had 1,000 messages and now we have 500. That means it's like, it gets more tangible to see what you're doing. Billing is also a use case. For our customers, we build them by hits. So if you D-dust them, we get more money. It's easier just to have all the data in one system and say, give me all the hit data from one month and give me a number. So that's what we are trying to do. You can also do quite fancy things and that's something I saw last week at the elastic search meetup. They inject in their error logs like application speed or they do deep inspection on the IP addresses. So they have somehow a service which checks if it's a Tor node which was connecting or a normal node or they can distinguish at some point with some JavaScript if it's a bot which is visiting the website or if it's a human. And they just pass it along to the server access log and then they can run data analytics on it. It was a job website so they are keen to know how people access it. That's also why they check if people are accessing it via Tor. So in our office it looks like this. We have a video ball and one screen is totally dedicated to showing the stats of the server like the total count of errors. You see when there is a sudden spike, you just see it. It's like you know that the usual error rate is around 200 messages and if you see a sudden spike, you just go in and check and you see there was a deployment, there was a Drush CCL running at that time and that yielded some errors or there was a crunch up which failed. It's just more tangible just to look at something and it's easier for the human brain to interpret the graph rather than some text. So heading over to elastic search. Who is using it? Quite a few. Who plans to use it soon? Great. Wonderful. So the ELK stack is the short name for elastic search, Logstash and Kibana. One small side note first, things move still fast. That's a slide I did earlier when in the beginnings of elastic search with like if you just didn't catch up two weeks, you just missed five versions of elastic search and it was like okay, now I need to update and you read through everything so try to keep up with the speed of the project and it will never fail you. Also updates got quite a lot easier and I'll cover that. So if you start now, you don't have the steep learning curve I had some years ago when you just, it was just three different products and you could put them on a server and lace it together with duct tape and shoelaces. So first elastic search. Elastic search is a Java application which now fears like 50% of the room and the other 50% say okay, that's nice. It does search and indexing and it's a distributed system. So you can say I want to have copies and you can say you want to have shorts. I'll cover that later on. It's also made to be clustered so there is the send discovery or send disco how they call it which is you just say the server when you want to do a cluster you just say hey, your cluster 01 and you just drop in another node which is also cluster 01 and they just discover each other by shouting into the network I'm cluster one who wants to be in the cluster and they just start to do their thing then. It's really cool it's also really painful if just somebody else starts up a server in the same network and has the same cluster identification in a lab for example and then your cluster is doing strange things and then shards are replicated and he takes it down or goes home after work and then your cluster is broken and you're like why? Yeah, that's the fun thing about it. The whole thing elastic search you can call it over a JSON API and it has a really good rest interface so you can do everything over curve requests and you can automate a lot of things like also failovers and stuff. The indexing part is done by the apache Lucene project so they use that library and one interesting thing which you should all be aware of it does disk based shard allocation so if you have more than one server it always tries to move the data to a server which has the least disk used so it tries to save it from yourself that you just under provision your disks and it tries to just move data around. If you crash it with a lot of data it will still fail so you need to do something about it. I talked about indexes replica and shards so an index you can envision it like a database and a replica is you just say I want to have one database and I want to have four copies of it and elastic search then says okay I have just one server and it creates one index and it tries to create four replicas but it says then okay I'm just one server I can't like do whatever you told me and it will then just show the cluster state red unless you drop in four more servers and then it will allocate the shards and sync everything sorry allocate the replicas and then it will say okay I'm green again I'll cover that later and the shard is an instance of Apache Lucene so if you want to index data you need to have some shards running if you see the data isn't indexed as fast as you want to have it you can just add more shards. The important thing about this is when you create an index database you need to say how many replicas you want that's something you can change afterwards if you want to change the shards you can't do it you can just say once how many shards you want to have. Since we are doing daily rollovers which I also take on later it's not a big deal if you see okay it's not okay today you just ramp up the shards tomorrow and it will then rollover. So how does elastic search look like? When you request elastic search you just start the char and it says okay I'm running on port 9200 and you just type that in it looks like this it's like okay I'm the elastic search instance number three I'm on cluster one and it says which build this is and the tagline you know for search. That's like okay that's the first impression you get and it's like okay what can I do with it. So there are a lot of front ends which are also just basically it's a bunch of HTML and JavaScript and they can do rest request rest API requests to elastic search. So that's one plugin and you can see I have several databases like lockstash with a date and you see the green boxes are actually the shards. So the green ones are the active one the lighter green ones are the inactive ones so if you just pull a server out it tries to rebalance unless it can do that successfully. So talking about this this is an elastic search plugin. The elastic search plugin system came in lately not that lately but it was early in the days you just copied it somewhere you had some files and it was not bundled with elastic search. Now you can just say bin plugin install and then run the plugin name and it will install it and give you back the name so it's much easier because it's with elastic search and elastic search has the plugins bundled. So speaking of that who run a elastic search instance publicly. It was fun. So speak with me. I will hereby solemnly swear I will never ever expose an elastic search to elastic search server to public and never ever. Just don't do it. Well it has been fixed issue but back in the days you could load binaries. And yeah we had like I'm a bad example for talking about security and elastic search because our server did a lot of damage in the internet. So basically somebody could upload a batch script and just run it and it was a botnet and yeah or ethernet link on that server was saturated quite a bit. So they took our server down and we were like yeah why and you just don't get that they come in over elastic search because there is no error logging. So that feature has been defaulting to disable now since some versions. There is elastic search shield which is a security layer for elastic search. It's a paid subscription feature so I'm not covering that because I don't get a license from them. I asked them and said hey I would love to do a talk but they said no. So elastic search security on the cheap. Run it, bound to local host or in an internal network and just do an SSH term. Easy as that. So that's how most of our developers are working. Some have the VPN which they can access it directly but that works pretty well. Never had issues with it. You can also if you have the screens you saw earlier you can just create a SSH session on boot and then you're in. That's just the easy way to a third convent that you need to have it publicly. You can also do a really easy thing. You can use NGINX and some authentication. Probably you authenticate against some authentication over in your company and you set the cookie and then you just let traffic go if you have the authentication cookie. That's also not a way but that works out for us. Another feature, that's just thank me later. There is if you get up late at night because your elastic search server doesn't have any disk space, you just delete some indexes and I did the start thingy and suddenly I had quite a lot of space. So that was fun. Well, all the data was gone too. So you can disable the really destructive calls like the globbing that you can perform actions on all indexes. You can disable them and I strongly suggest to do so. Another feature they have or elastic is providing is Marvel. So if you run a really big distributed system of elastic search instances you should look into it. It's also currently free because they are developing it but as soon as it's finished they will just make you pay for it. So I didn't look that much into it but it looks, it's like stats you can just look at it and it's nice to see how you just enter data and you see where the data flows through. So that's fun but that's all. So the next thing is lock stash. So one question is did the Catalan citizens invent lock stash because I saw something like that this morning. So you see next slide that's the lock stash. So they look pretty, you see? They look pretty same. So no they didn't. Lock stash itself you can think about it like multiple input, multiple output system. It can take nearly everything that is text or is a buffer or a file, something you can just feed into it then you can do a lot of processing and you can just throw it out again. That's the easy way of understanding what lock stash is. You can collect data, you can parse data, you can also say like I don't want to have this data. If you have a debug flag in your log line you can just say debug and if you go into lock stash you can just drop it. And you can store it or you can forward it. So it's either an end point in your system or it's a throughput system. I'll go a bit more into detail now. So the life of an event. Something is written to for example watchdog. Watchdog sends that request to syslog and syslog sends it to the input of our lock stash. And then we do some filtering because it's just a log line. We need to break it up. We don't need to have the system name where it was relayed through. We want to see what's the client website's name, what's the error, which user was triggering the error because that's a really neat information to have when you know customer calls and says I can't save node whatever. And you can just say okay give me everything from this user and you see errors. That's something really cool. You can run it by codec. So codec changes how some data is represented. So you can for example take codec which is Ruby debug and you have a log line and it creates a readable JSON file for you or a JSON output. And you have the output. So you can feed your data into a file or send it to Redis or send it to S3 for example. So lock stash is J Ruby. If you really want to know why it's J Ruby there is a link down there and I will distribute this later on. Shorten Zissel who created lock stash goes there. It was a GitHub issue and he just wrote up a really big gist why he's using J Ruby. So don't worry about that. Sight note here since everything later than version 1.4 we know at 1.7.1 it's just a flat chart. It just can run the chart. Earlier it was like chart bar and run classes and stuff which was not that much fun as it is now. Now you can just run bin lock stash. There are a lot of country plugins and you can also write your own. If you fancy so but you then need to go every step and if something changes you need to do all the hard work. I never had the issue that we said okay it can't do exactly that thing we need to write it on our own. You don't need to do that. And here come the daily indices. So on every midnight it rolls over to a new index and that's really cool because it's like a log file you log rotate every night and you have your data really clearly separated. We were talking about inputs. You can have a file as an input. You can say lock stash listen on the syslog server port or you can also say hey I feed all my data into Redis which we call also later. Or you can get data from lock stash forwarder which was lumberjack for those which use it since quite a while. Covering some filters you can use grok which is basically reg accent steroids. You can have a grok parser which goes through your log line and just name every part of it and after naming them they will be available as indexed entities of your event. So that's really cool. You can mutate things. You can remove, replace parts of your message. You can drop things like if you have just log messages or one system is sending all the other things you can find a way to say okay that's now there is like a debug flag or something in it and just drop them. You can clone them if you want to split them up in several directions and you can do GUIP based on IP addresses. So you can when you feed in all your access lock you can see in near real time where your traffic is coming from. That's really fun. If you think about it when you have day and night and have a customer base on a website which is going to the site 24-7 you see how the traffic is shifting. So that's really fun. And so also makes a big impression on the customer if you can say oh do you see the traffic or servers are handling like now they are coming from China and now they are coming from the states and stuff like that. There are also several outputs. So elastic search that's why we're here. You can save it to a file which is really really cool if you want to have long term storage because you don't need to have any all your data in elastic search all the time. You can drop it and import it later on. It just adds. I think you can easily compress a file and store it then you can save 600 elastic search indexes so it makes it easier. There's also a neat thing the S3 output. So you define like I want to save 50 max chunks to S3 and every time it hits the 50 max limit it will flush it to a new file and those 50 max it will just upload them to S3 which is cool. Depends on your use case if you want to have daily data you probably might go with a file. You can also move the data to graphite or stats D. So you can take out for example performance like a page load metric or a PHP render metric or you render like a call or you measure a call to a backend system and then you just save that data and send just the time state, the time it took to do the request to graphite or stats D which is also cool. So looking at the configuration file they all look like this. So you define an input for example standard in you define an output with a code at Ruby debunk and when you start lockstache you can say very important lock message and it will output that. That's not doing a lot of things currently but it gives you an impression how the lock or the configuration part is done. You can also say okay I want to have it outputted to elastic search which listens on your local host and you also want to split it to standard out just for debugging reasons. Do that. That's a time saver. If you have like if you start to part things like through error messages and it goes to multi lines or something like that you definitely need that and you will try like just copy the watchdog error message and paste it in your terminal and see what it does. There are also tools I showcase later on which help you in doing that. Yeah I have another, there is another snippet that you can say I have the syslog files and the start position is beginning so every time you start lockstache it will go to the top and run through. It also under if you there is another start position which saves where it was and when it starts again it will just continue where it was which is also really cool. So Kibana. Who knows Kibana since the early days? Who loved it in the early days? Well okay. So Kibana now looks like this which is awesome but looking at the history of programming languages it was written in looks like somebody had a lot of fun. So they started with Ruby then he said okay let's write it in PHP then he wrote it in just JavaScript which was like awesome you can just get cloned it and you're gone you're done you can use it which was awesome. Now it's a Node Rep server and yeah you need to install a lot of things again and if you have a PHP setup it's not that easy as just having something. The not so easy part comes with a lot of benefits which is it's based on D3.js which adds a lot of fanciness to your graphs so that's cool. There is a more complex back end. It needs to know something about your data and if you started the first time it asks you some questions how are you indexes called and when are you rolling them over just to let the Node application can talk better to Elasticsearch. You have a much better flexibility which is when you come from Kibana 3 a little bit overkill because you just with Kibana 3 you could just like do everything click it together and you are good and now you need to like write some filters and think about it in a more detailed way than you did. But you can do a lot of analytics and you can also aggregate stuff. You can just have more facets in your search and you can do those nice pie charts with like two charts overlapping which looks really cool but I didn't find any idea how to use them yet so that's something. So enough of Kibana let's go to the architecture part. Architecture is pretty simple so what we use currently we use syslog on every web server to ship data to a so called broker. Broker is in our setup the log stash. Log stash parses the data like breaks the log line up into several fields and sends it to Elasticsearch which then does the indexing and the save on the storage of the file. But bashing to say that's nice but how are we going to make that highly available because if something breaks down here it's like you just lose data because your syslog is happily sending out data and it just vanishes somewhere because it's a UDP package and they are not tracked in any way. So let's go to the real deal. To do that we need another tool so the whole thing is a big puzzle of simple small tools which do exactly what you want which is a really unixie way of doing so but we need log stash forward. It's written in go just to add some spice to your setup but it's really lightweight and it's just a thing that you point to a file and say ship it to my log stash instance. It really really uses not a lot of resources if you compare it to having a full blown log stash running on your server so it's just you don't think about it. It uses TLS SSL encryption and you need to have a real certificate. If you self sign it you will probably go to Stack Overflow then you will find some Reddit posts and then you will swear and drink some coffee and be sad and you just need a right certificate or you need a CA running which it can authenticate to. So that's something I was working on is like okay let's just use our company certificate because it's easier and it works. So the architecture now looks like this. We start with the same we have the log stash forwarder instead of SUS lock and then we have several log stash instances which take on the data. We save this data to Redis and from Redis it goes to log stash which does the indexing and saves it to Elasticsearch. Done. Okay now from here it can go crazy. So actually it looks like this. It's the same in the beginning. You just send the data from log stash forwarder to log stash and log stash just sends the data further to Redis. So if you just take out one indexer node the other one will take over which is neat but if you have been to the session about moving mollum to AWS I guess who was there? Well that's mostly of your, yeah it's all of them. So we weren't a lot of people so we did a small exercise of how to do highly available things. So the log stash will just take over because one Redis is filling up and that's the way. So done. But bastion. So we're going to set really the solution of all problems. Simple answer? No. It isn't. So if you're with me we spice it up a little bit more because the thing is log stash forwarder can deal with okay I can't send data now. I just stash it and know where I, what I could send successfully the last time. So we're safe on that side. For the shipper part we use keep a life D which behind keep a life D we have to HAProxies running in TCP mode. Which then just load balance the data to our log stash behind. So if one dies we just move the IP over to the other one and we're good. So every part of the system can take over if we need to do that. So and the indexer part does the same thing as we did earlier. So if you just restart Redis the data will go through the other one or we will have checks. We have checks which ensure that Redis runs behind and just remove it from the rotation. So that's what you can do. It's a really cool setup but it's for a use case where you would really want to have that. If you don't care about losing five minutes of data when you just restart the server fine just go with a really easy setup and don't do that because it adds a lot of complexity. So you've been warned you can use that. Let's talk about tools. You saw a lot of screenshots earlier. So there is one called elastic search head which was the go to tool when I started which was I see I see indexes I see shards it can close and open them. It looked awesome. Below you see the command how to install it but stop here. Don't write this down. There is if you translate head which is English obviously into German then it says cop and suddenly the plugin is elastic search cop. People I don't know they do things and that's really cool. So it basically does the same thing but you can do more. If you want to move a shard to another server you just click on it and then you have a little tick there and you can just move the data to another shard and in real time how your data is moved across servers which is really awesome. You have also a console you can type commands and do stuff. So that's really the tool you want to use. If you have an issue with load you also see the red thing there that's the load which it shouldn't be red it should be green but I had quite some load on it screenshot so yeah. And you can also have an overview of how your cluster does and if you want to do some maintenance you usually move data away from this cluster from this cluster node and then reboot instance so you can at one website you see how your setup is doing. That's one tool I really tell you to use. Another thing is curator. So since everything is scriptable by an API you probably want to clean up because having a lot of data in your system and you probably don't want to have everything present like we just have the data accessible for the last seven days and if you want to have more we just open indexes it again because open indexes also need some RAM and file system space. So yeah that's the thing. Curator helps you in doing so so you can say I want to close indexes I want to delete indexes you can also say I want to delete everything that's older than 10 days or everything that's bigger than 100 gigabytes. That's really cool. You can disable the bloom filter which nobody tells you to do so but I do. The bloom filter will use a lot of RAM for indexes which are open because it's a filter which is used when you index data. As we have time series indexes you won't use that filter on every index older than a day because you won't enter data there so you can easily disable them. You can optimize it or in elastic search terms say do a force merge. I did it once and it just used a lot of RAM and a lot of CPU and yeah it didn't change a lot so I don't use it. It's all in GitHub. It's a script you can run and it runs really nice outputs. So Curator is perfect for time series indexes. Don't try to build your own because it does everything out of the box. That's how I run like close indexes older than 14 days. Delete everything that's older than 30 days. You just run it and you're good. Or the second one is close the indexes which are older than 14 days. Disable the bloom filter which is older than two days because you have time zones and it gets creepy at some point and delete everything that's older than 30 days. Of course it creates a log which you could also feed into your setup and do the elastic subception. So that's cool. Another thing if you're really into Java is Big Desk. It gives you a lot of detailed information about how the Java processes are doing. Bear with me. I love it sometimes. We talked about grog filters. They look like this. That's an ugly one. That's basically I guess a production filter. No, no it isn't. But you will start writing those filters and you will start swearing. And there is a really easy solution to that which is called the grog debugger. So you copy paste a log line and you copy paste your grog filter you wrote and it will output what grog can do. Which is awesome because when you do it like I started I was writing my config file, saving it, starting lockstash, typing something, seeing that it fails, breaking down the Java process, doing the process again and it was like tiring. And here it's just like okay I need that and I need that. And you just lace it together and in the end it will work fine. It doesn't do everything perfectly but it helps you to save time. If you're into reading, buy the lockstash book. I didn't write it unfortunately but it's the best ten dollars you can pay for a book on that matter. It has all the HA stuff in so it goes from having one node to having a dozen of nodes and how you can scale it up and it goes really into detail. Elasticsearch self or elastic made a really huge effort. They wrote a small book, the definitive guide to Elasticsearch which is awesome because you have on every single nitty gritty detail of Elasticsearch you can find a big write up what it does. Sometimes it's okay I just wanted to know if I should set it to true or false but thank you for all the information. Another topic I want to cover is the performance optimization short P22N. Remember guys it's Java. So Java needs memory and it needs IO and it needs files. So since we have a lot of files stored on the disk you need a lot of file descriptors to have it open. So it can happen that when you say okay I want to have data from the last 60 days and I want to have this filter and you just hit enter that your server just says I can't do that. So be sure that you raise the file descriptors. Also be sure that you have enough memory but not more than 30.2 gigabytes. There is a link down there which explains why not. Just don't go over 30.2. And try to leverage the file system cache. Because everything, because it's based on a Pashilu scene it leverages the file system cache. So the rule of thumb is that you just use half of the memory you have on the server for the Elasticsearch and the other half will be file system cache which they also write down in exact detail how you should do it. That's the short overview how you can do performance. There are a lot of blog posts and I will also publish a small blog post tomorrow on my blog with all the references I took in here. So it's definitely worth the read. Automation. Who does puppet? Okay. I'm sorry I just checked for puppet because now I'm doing puppet so there wasn't really no reason to check ansible and salt and whatever. So if you do puppet there are three really good. Okay. Two really good things. To install Elasticsearch use that module. It's awesome. It just works out of the box and you don't have to think about a lot of things. Logstash forward also works good. Logstash itself it's like yeah you need to define parts of your config file and get them together. I didn't like that too much. So that's a short write up how I provisioned my test lab so you just say give me an Elasticsearch with the correct repo. Say you want to manage the repo. Also say install Java because if you set it to false it will just say okay can't start. Yeah. You can easily define the cluster and the data here and then you can go. It's really easy with those things. If you want to have more instances you can just copy the second part of it and just have more instances on one server which it doesn't speak against it but it's like there is no reason to do so. So it should be a separate server. For testing it works. You can run several instances on one machine. Yeah. So what can you take home from my talk? So centralized logging saves a lot of time because you don't write a lot of shell things together to get the data your boss wants from you. So that helps a lot. It's fun with the Elasticsearch stack because in the early days you need to put everything together and now it's like it's a solid block of one tool which every tool you use has a really easy way to use it and you can it's a dense package of tools but you can use it in a pretty awesome way. It gives you graphs to interpret which are really much easier than checking log files and the question about the log files between two after three and seven after three gets a lot easier to do and if people say okay now I want to have it like from yesterday and from yesterday and whatever and a month before you can just do it when you have the data. So for starting I suggest to use the getting started with lock stash. I see there is a tie point it should be 172 but it's redirecting you probably to the right version. So there is a really good guide written by the lock stash people which show you how to get to the end setup we discussed. So all I say thank you for having me here. It's been a pleasure talking to you and I'm happy to answer questions if there are some. Yeah sure. Yeah so thank you for that question. So Redis does the buffering of your files. So you could use any system which is able to buffer files like rabbit mq for example as a queuing system. Since we had Redis already in a setup it was an easy guess to say okay we have Redis we can reuse that part and not trying to spin up a rabbit mq. So it's the part where you just you have a bucket you can just fill data in and you have lock stashes who take the data out and parse them. So that's the part where you just buffer data if you have a lot of data. You could send it directly but then if you have a spike in traffic for example if you parse the access files probably your elastic no your lock stash processes get overloaded and can't deal with the traffic. That's an edge case but Redis is memory speed so you can go pretty fast. So yeah okay I just rephrase it for you recording. So the question was how are you detecting that you need to scale up your Redis because you get a lot of data. Well honestly I didn't yet reach that case but you should monitor the usage of Redis like how many space you have left because Redis behaves pretty strange when it reaches the end of its memory so yeah I don't know I would just monitor it and send an alert if you reach the end or having a setup which can like spawn another lock stash process to process more data out of Redis. But currently the old setup we have just runs on one lock stash process running as a sys lock and we never had issues. Not even in times when we were analyzing data from the Swiss television which have quite a lot of traffic at some points when they have live shows so we never had that issue. It scales fairly well so to say. Does that answer your question? Thanks. Yes I didn't. Well the question was if I try to use it with Java 8 I didn't. The last time I tried it I was just saying the puppet recipe to install Java so I'm not really sure which version came with it. So that's definitely something to try out. You have. Okay. Well okay so basically we were discussing about using Java 7 or 8 so they run into issues with Java 7 indexing a lot of data and so I was just switching over to Java 8 solve everything so it's not yet recommended by Elastic. So probably that's one that's probably one tip they say if you have the paying subscription like oh you have issues just upgrade to Java 8 so yeah. I guess they will support it sooner or later. But thank you for that. Other questions? Yeah exactly. Yeah so the question is basically how do you deal with changing applications that report to it and being able to process data which you probably didn't think about when you started out. That's correct right. So one thing I heard last week is like data you don't have hurts the most. So what I do I when we started out with doing elastic search and logging we were dropping everything that's older than 10 days. Then people found out that it's pretty handy to tell the customer that they deleted their stuff and not we did it. So the first case was like 14 days and they said like okay yeah can you please do the check again and I was like no I dropped data after 10 days because it's a work site project and it's not like by any means meant to be productive. So then I said okay we just stored the data for 300 days which also means when you have changes and you want to have something analyze it creates a lot of pain. So what I would suggest is that you split it in a file and archive the file and if you just add things or you just think about things you will need you just add them already and you can parse them afterwards you can just re-index the file again. So that would be an idea or just try to think about use cases and if you want to add it just add it proactively. But there is no way of unless you dump a whole chunk of data of specifics about your application in it you can save them proactively but it will just use a lot of power. So it's really hard to say what to take in. I can also post a link to some slides I was attending the elastic search meetup last week and they have a really good example of what specifics of their application day track and they also say if we forget something or if we add something we just add it then. But they have like a really big set of around 10 to 15 metrics day track but they say we don't need it yet but if we want to have it we have it ready. So that's the best advice I can give. Does that help? Wonderful. Other questions? Okay. One last thing. There are sprints on Friday and you're really, really welcome to join as you heard Dries. We want to try to get that Drupal 8 out of the door soon. So feel free to join. There will be people helping you. Thank you again for attending.