 Before I start, I would like to point out that this presentation was prepared for begging an audience in mind. So if you guys have been doing Docker, running it production for a while, it's probably not for you. So please feel free, leave me any time, I will not take it personally. If you always wanted to try run, log stage, sorry, I said Docker. Log stage and give this a try but didn't know how you're in the right room. Okay, let me introduce myself. My name is Margie, I'm a systems engineer working for a small triple company in Australia based in Sydney called Morved. It's small but we were lucky enough to have a few enterprise customers from the publishing industry, pharma industry and local and government, Australian government. So that gives us good opportunity to do a lot of DevOps at big projects which is really good fun. I have been a fan of DevOps, following DevOps for the last five, six years I would say. And I think I became a system in 1999 when my colleague at a web development company left us and I was suddenly the only responsible for the production web server running actually web pages of a political party which was in the government at the time. So that was really good start of my Sysadmin career. But it was fun. Okay, okay, so to give you, get you an idea. I used to work for a couple of ISPs in Australia as well. If you are wondering is this Australian accent, it is not, of course not. I'm originally from the Czech Republic. I have lived in Australia for 12 years. So I worked for a few big ISPs in Australia and you get a ticket which says customer says they get randomly redirected while browsing their website. There is no more information. So what can I do, right? So this is way back. So what do you do? You log into the web server and start gripping logs, right? So this is an Apache. So you see like gripping for 30X in Apache log and trying to filter out the lines which I don't wanna see, so I'll get rid of Googlebot. And you know, this is a mess, right? This is how it used to be. It can be even worse if you happen to have more than one web server, like behind a load balancer, you might need to jump on all of them and do that on each of them or maybe download the logs on your machine and do it there. I actually do remember working for a big telco in Australia that we had a colleague of mine wrote a little robot which was periodically r-syncing Apache logs or Tomcat logs from 24 production web heads to like a big thumb per storage box just to be able to have it locally and parse it. You know, like that was desperate but that's how people used to do it. So what if there was a new way? What if you just go to a console and type in, hey, give me log type Apache and it belongs to this site and the server response was between 301 and 304? Well, there is such a thing. It's called Kibana and you can see that here is the query line and after hitting enter there, you get 12 hits back over the last 15 minutes. If you can see it in the right-top corner which basically are exactly what they are looking for and you can start going into and you can see these fields on the left which are extracted from the lines. Then you can go and create a visualization, a little chart with all the response codes over the time or maybe even better, like a data histogram which shows you more of what's happening and what codes you are getting. So what is it we have just seen? So this was Kibana. I will explain what Kibana is and we just, as you could see, we just executed one query and created two visualizations to help us look at the data. But what's under the hood and where is the lockstash I'm supposed to talk about? Well, ta-da, ladies and gentlemen, let me introduce you to the ELK stack. ELK stands for Elasticsearch, Lockstash and Kibana. These are three open source projects maintained by a company called Elastic which has had a query in Amsterdam. And I go regularly to their meetups because this really, you know, everybody, like many companies these days use Elasticsearch for storing documents, searching. So I go to their meetups, so a little bit of interesting development on the ELK name. So I think early this year, they realized it's not ELK anymore. Maybe, you know, like there, it's like a B and ELK together. You can see that they tried. That's because there is a new component which came to the stack. It's called Beats. So we gotta call it Belk. It's probably the accurate name for today. And also I think the Elastic announced that, at least since the new version, like there is a version five of all these four components coming within a month or two, they will call it Elasticstack. So that's actually the name, the next marketing name of the Elasticstack. So Elasticstack is the old Elasticstack. And these are like new logos of the products. So what's the goal of the stack? It's basically designed to take data from any source in any format, process it, transform it and enrich it, and then store it so you can search, analyze and visualize it in real time. So that's what the stack is supposed to provide you. As I said, we can look at it as like, it has four components at the moment. So I will just walk you through these briefly. Elastic search is where the data is stored in the Elasticstack. It's actually a full text search analytic engine. You might be similar with Apache Solar. They actually share the same code base. Both of them were forked from Apache Lucene. But this one has like a high availability, like built in and horizontal scalability as well. You have a cluster of several Elasticsearch nodes. You are running out of space. You just add another Elasticsearch node. It automatically joins the cluster, rebalances the data, tries to basically rebalance itself. So it's designed for high availability and scalability. The next tool is lockstash. That's the tool which collects, process and forwards events. And basically for data collection, enrichment and transformation, it has many input, outputs and plugins. And this actually shows you a little bit more. This is the old lockstash logo before the new one was introduced, which kind of, you can see here what's happening. You have a lot of data sources coming inside lockstash. You can even pull. It doesn't have to be pushed. And then you process it somehow and then you can output it to different endpoints, whatever you need to. Just to say a little bit with lockstash, so as I said, there are input plugins, output plugins and filter plugins. So for inputs, just a few, like it can be file, like a log file you can read from the disk. It can be a socket, TCP, UDP or web socket. It can read from syslog. You can read from message queue, from Microsoft Windows event log, also from DrupalDB lock. As I said, you don't have to only, you know, like receive something you can actually pull. So if I was reading from DrupalDB lock, I would be pulling from my skill database. And the beats, I put the beats on the first, that's the new thing, as you will learn in a minute. There are also dozens of output plugins. So after you process the data in lockstash, you can store it somewhere else in a file again. Maybe you just pre-processed it, changed the structure. You can send it somewhere else via TCP, UDP, web socket, store it in syslog, put it in a message queue, send it to your metrics storage system or, you know, create a ticket in Jira Redmine, you know, create a Twitter tweet, store it in S3O, or store it in Elasticsearch. That's what the Elastic stack does. And there are a few filters lockstash uses for manipulating and reaching the data, which we will cover later as well. Kibana, that's the interface. You use your browser to run Kibana, and that's open source data visualization platform, which allows you to interact with data through powerful graphics. I'm reading that this is really nice. Brings data to life with visualization. I like that definition, that's what Kibana is. And the last addition, that's the fourth component, that's where Elk became Belk. Elastic came up with the concept of beats, which are kind of like lightweight data shippers. So think of it as something you can install on your server where the locks are. So maybe you have like a few Apache servers, so you can just put a little file beat there, tailing the Apache lock and sending it to lockstash, or you can install, I think it's called top beat or packet beat, sniffing packets, sending it to lockstash. So these are like a lock collectors, designed to be lightweight. So that's the, and there are more and more of them getting developed. To show you the flow in the Belk stack. So we said that there is elastic search when we are storing, I'm going to store the data, the processed locks. Kibana connects to elastic search and gives you the visualization, like to be able to query it, visualize it, analyze it. Then you have the many data sources, that's the input. You somehow need to get the locks from the data source into elastic search. So that's the lockstash, lockstash does that. And as you can see, one data source might use a beat to get data to the lockstash. And our data source maybe uses a different path. Maybe it's actually running locally on the lockstash server. So the lockstash might use just like a file input. There is no need for any transmission. And maybe, as I said, the new beats kind of support because they are lightweight, they might do some kind of pre-processing as well. And maybe they are capable of sending the locks straight to the elastic search because there is no processing needed anymore. You have done the little of processing you need on the source side. And then you can just store it straight away. No need to go through the whole pipeline. This is just to show you the input, output and filter plugin which lockstash consists of. So before I show you how to run that, I would like to say a few words about Docker. I don't think I have to introduce it too much because this has been such a popular tool, overlast at least one, but maybe two years. When like basically developers these days run Docker on their machines, just to be able to run stacks very quickly. I noticed enterprises even like old style, like telco enterprises have Docker in production these days. So it was really a breaking technology. Me as a sysadmin, I saw it these two, three years ago when it came as a lightweight virtualization platform, a way of virtualizing things. But actually that's not how the Docker developers sell it. They say that Docker is an open source platform for developers and sysadmin to build, ship and run distributed applications. So they kind of look at it as a packaging system. You somehow build your application, inside you create a Docker image, then you can ship it, you store the image somewhere and then when you need to run it and it would be on your notebook, it might be in production, it might be on staging, it might be on different operating system. You just get copy of that image, it's already built, it's a package and you execute it. It's a beautiful way of thinking of application packaging system which might run anywhere. This is kind of a younger definition. Docker is a tool that can package an application and its dependencies in a virtual container that can run on any Linux server. There is a dependency on Linux server, but you probably know if you run it on your laptops that there was boot to Docker, there was a newer version when it kind of installs virtual machine, virtual box for you so you can run it on Windows or Mac. I think Docker, was it last month, released a beta of what they call Docker for Mac and Docker for Windows and they cut off the need for virtual box by using the native hypervisor. Under Mac it uses the hypervisor which is built in Mac. On Windows it uses hypervisor which is in Windows so there is no more, it still runs Linux kernel on that hypervisor but it does not use the heavy lifted like a virtual box to provide it. So it's even closer, you can think of it as a native tool. So I wanna just show you quickly how I run Hello World LogStash by one line command. So I'm going to Docker run a LogStash 2.3 image. Why 2.3 because I know that version five is going to be released this month or next month and if you don't specify the number it tries to download the latest so I'm locking the number here and I'm configuring that it runs LogStash program within the container and this input and output plugin. So input plugin will be standard input and output will be standard output using the rabbit debug codec. So let's see how that looks like. I have a little cheat file. So my Hello World is here. It's starting. So it's one liner. If I say Hello World, that's the standard input it's waiting for. I'm getting the same thing back but it's structured. You can see I got like a JSON. The message is Hello World, it edit a version, it edit a timestamp, it edit the host name of the server it was processed. That's actually the name of the Docker container it's running right now. So you see this is just reading standard input processing it. So that was my Hello World application. So I just run LogStash by running one command. What I need to do, I just need to have Docker on my machine. I just showed you how to run Hello World in Docker LogStash. So I killed that container. So I have no containers running. No containers stopped. So there's nothing. And we can go and do the same test but this time we also add a filter plugin. So we had standard input before and standard output but now we are also define a filter using the grog plugin and saying try to match the message you are getting in using the combine Apache log pattern. You can see here in the middle. So that's a pattern it's like regex it's defined and it will basically try to parse the line and break it based on the definition of the combine Apache log which is a standard and I will pass a Apache line to the standard input and let's see what we will get. That's my cheat file again. So I'm running Docker version 2.3, sorry, LogStash 2.3 in Docker and I'm configuring it from the command line providing input, plugin, filter plugin and our plugin starting. And now I take this Apache line and paste it to standard input. We know it. And it used the combine Apache log definition, pattern definition and parsed this line I pasting on input. So you can see that this get got here in the verb field. This 200 got to response field. This IP address got here into client IP. So this is how I like LogStash parsed an Apache log line. So that was just a little play. I will stop this LogStash with no Docker running. I can go back. So let's try to run this. This is something you can run in production. It's not high available, but you can actually do that on your little server. So as you can see, we have three components here. Elastic search when the data will be stored. Kibana, this is how we will look at the data. LogStash is what will process the data and we need some kind of source. So I will run three containers. Elastic search with this line. And I will run Kibana linking to the elastic search. And then I will run LogStash giving it a configuration file in this directory and mounting and giving it a Apache log file which I downloaded from a production server last night. I'll show you what it is. So let's start elastic search. And once again, if you have Docker installed and you execute this command, you are running elastic search right now. This is how easy it is to use Docker. So elastic search is running. You can see it's registered here. Now I start Kibana saying run Kibana 4.5, expose port 5601, which is standard port Kibana listens to. So instead of, I want this 5601 to be available on all my interfaces on my local machine. Link it to the elastic search and name it my Kibana. So I start that. I'm running elastic search in Kibana. So if this works, I should be able to go to localhost 5601 and slow it in Kibana. And it's asking me to configure an index and it's telling me that there is unable to fetch mapping. Well, that's because elastic search is empty. Let's expect to try it. So let's give it some data. So the last thing we need to run is the Docker lockstash. And I will show you what I'm giving it. So I have a lockstash directory. I'm just, I'm not cheating. You see that I have one config file and one Apache access. So this will, I will pass this as a config and I will pass this in as a flock file. Let's just have a look at the file. So it has 34,000 lines. And you can see it's Apache lock. And let's look at the config. I will cover that, but as you say, there's input file. That will be the Apache we are passing in. We will be using grog filter for combined Apache lock gain and we will write this to elastic search. So let's execute that and starting. These who are closer to me can probably hear my fan just started spinning. That's because it's processing these 30,000 lines should take only a few seconds. So now when I go back, I'm running three Docker containers. Elastic search, Kibana, I'm looking at and the lockstash which just process the lock file and stored it in elastic search. So if I refresh this page, I should see some data. So I'm saying, hey, I can see timestamp field. Should I use it as index? Okay, please. So now it analyzed data and found this data inside the lock file like these fields inside the lock file. So we can look at it. Oh, there's no data. Why is that? Well, because this is last 15 minutes in the right top corner. That's the default in lockstash. It actually burnt me at least twice when I found something's broken and like just don't have any data for last 15 minutes. This file is from the last night. So I need to go back and say, show me maybe the last 24 hours. And you see I'm getting something here. And these are, these are Apache lines, but they're a little bit enriched as I said. So it's not, you know, like because you can see I have GeoIP Czech Republic. I took it from a Czech server because I'm not afraid of the customer being angry at me. It's a, it's a Drupal issue. I built years back. So I know I can do this. So it got enriched by GeoIP fields, but it's still the Apache lock, you know, the agent, the client IP, the HTTP version request and server response. So usually keep analogs like this. Unfortunately, this resolution is pretty low. So if I go, you usually see this field on the left and you can kind of filter through that. So you can see if I don't see just agent. So now I'm, I'm just highlighting agent or you can see I don't wanna see agent. I wanna see client IP. So that's each line, but the highlighting you did a client IP and HTTP version for example. So this is how we can do your data. So, but let's go and do a little bit of a investigation. So we were talking about a response code. So let's, let's create a few visualization. So let's start with the most catchy one. So we create a new visualization from, and we tell it to use terms, which terms, well, let's, let's use the response codes. So if I can see it here, I can see the field and use. So I'm looking at last 24 hours of locks and this was my distribution of status codes. So I will save this, call it locks. So this is high response. Now I create a little bit graph, which is more despair with me. I need to create four to show you three to show you something. So I'll create this vertical part chart. It's a good one to go. Oh, it will be, no, sorry. Let's start again from a new search and it will be data histogram, split bars and add the terms and use the field response. The same thing, but shows us as it was. So I will save that as well. And this one is called maybe. And the last one, geo IP, that's always very funny, very good topic. Geo coordinates, it will automatically offer me geo IP location. And you can see where I'm getting most traffic from. So I will save that one as well. That's good. And that, I want to show you something. It's really cool. Now when I have like three saved, I have saved visualization. I can create a dashboard. So I go to dashboard and I say, okay, so I have this pie response. I have the pie response and I have the geo IP. And I put them somehow, it's really difficult on this resolution, but I'm trying my best. So I will resize them to see them, hopefully at the same time. The geo challenging, but I will, yeah, good. So, and now I can do things like, oh, maybe I wanna look for the last seven days instead. And then when I go and zoom in, you see that all the visualization are changing at the same time. So if I click on 200 here, I can reverse the 200, so I will invert it. So this is everything but 200, the five most frequently seen response codes. I can pin that. And I can start investigation and I can maybe zoom in. As you see, I'm zooming in. It's changing all the graphs. The geo IP is changing. So I have like awesome correlation without actually just playing with the data. I haven't saved anything. I'm just carrying elastic search via Kibana. I just like go back. So I don't get lost. So let's have a look at the last 24 hours. And maybe try to, I have a few minutes left. Maybe try to, so when I'm looking at the visualization here and I pin this response code, not being 200, I can go to discover and I'm looking at the lines which are what I can see in the graphs. So I can go and go a bit more. But hey, we were saying, so what's actually interesting here? Like there is 404 a lot. Maybe you have a look what's actually causing it. So I can look 404 and it's going mostly from one IP address. Like, hey man, like how about I look at this. I pin it as well. Here I maybe wanna see not the client IP, but the agent. And I realize that all of these are crawl requests by doing this. And I can see that it's actually me preparing for previous, I presented the short version of this at a different meetup at a meetup. And I was just trying to create some anomalies in the log. So these 404 are caused by me, but you can see I just noticed just by going through graphs. So I don't wanna waste too much time here because time is flying. So it was, so what we have just seen. So we had log stash reading input lines from an Apache log, 33,000. We used the, we filter it using the combine Apache log pattern. We store it in Elasticsearch. And then we use Kibana to basically go through the data located in trying to look, we could see use geo IP as well. I could spend an hour here. I have to admit I'm not a Kibana expert. I'm interested in getting the data in, storing them and reaching them. And then somebody else, a developer can go and, you know, like go crazy. I'm not that gifted in Kibana, but I'm just trying to show you how it can help you as a developer. And I used basically as you could see, I didn't, my config file wasn't a bit. I just like said, hey, hey, use a source access log. You know, start from beginning. Why do you need to start from beginning? Because log stash just reads from the point it started. So if there is an old file, it wouldn't touch it. So you actually have to force it to reread it from the beginning because it was from yesterday. Otherwise it would be waiting for the file to get new lines and only these will get processed. Store the output in Elasticsearch. And the filter, like so if type is Apache and we set the type here in this, reading this file, we set the type is Apache. It's just like a my keyword type is Apache. I set up this keyword type is Apache. So if type is Apache, use this grog pattern and match it for combine Apache log. And that was the grog plugin, filter plugin. Also apply GOIP on the client IP. The client IP, I get client IP from the previous grog match from the combine Apache log. Client IP is one of its field. So then I applied the GOIP plugin. It gave me the, so I can then show you the map, GOIP map. And also do something with date. Like because if I did not tell LogStash what to do with the date, every single log in LogStash would have the timestamp of the time it was processed by LogStash. So if I'm importing 30,000 log lines they would have timestamp now. So that's why I'm saying, hey, actually there is a timestamp field with this regular pattern on that for the date of the event. So LogStash comes with a lot of patterns. We used combine Apache log, there is also like username as you can see letters and numbers, positive integer, sys log. So it's, and this is how combine Apache log looks like. So it's always IP, it's type IP host store it as client IP, pattern user, store it as ident, pattern number, store it as HTTP version. So it's a regular expression definition of the, so here is actually example of Apache log. So this first IP address will get stored in client IP if the match is successful of course. So where are we? I described the components, the Bellach stack uses. I have run a local instance of the stack by running three commands. We processed and stored an analyzed, we processed stored and analyzed an Apache log and found an awkward four or four just by using combine to visualize that. And I hope that each of you could do the same. You don't need too much to start. Before I jump to the last part of the presentation, I would like to go, if you have your tablet or phone, can you please go to this address and pick your poison? It's just a little live demo. Let's see whether this will fail or not. I will go there as well, just to prove that's not nothing to be ashamed of, ashamed of. Belk.site-showcase.com, just that I didn't have any better domain, neutral domain too, to use. So if I go to Belk, site-showcase.com, it's a little triple eight side I put together. Runs on Acquia in Sydney. Being assistant admin, just if you can pick one of the poison for me, which talks to you. So I'll pick Ansible. Okay, so let's go back to the presentation. I hope I got a few clicks. It's just to mention central logging. I thought I was done with my presentation, then I re-read my description and say, oh, there is one more paragraph I have to cover. So like I did that. So centralized logging, why do you need centralized login? I try to demonstrate that by the example in the beginning, it's basically to get logs to one space, so you can one place, so you can analyze it, you can do whatever you want. You can, it's convenient. I put secure in the brackets here because it has security implications. Of course, if you have all your applications and production servers pumping logs to one central place and somebody compromises that server, you know, like they have perfectly very good view of your infrastructure and it's like a security, security risk breach. So be careful if you do something like that. Think about security definitively first. It is not a new thing. I remember R-SysLog reading, like trying to configure R-SysLog to stream SysLogs from several servers to one central SysLog server at least 15 years ago. So I'm pretty sure that already 15 years ago, SysLog could stream remotely via TCP or UDP. UDP first, probably TCP maybe after. The more servers you have, the more you need this. You know, like if you run cluster with, you know, like one web server and via SQL, that's probably all right to jump and see what's on the common line was happening on, but if you have it high available, you need it. You might also need to archive that for audits. And if you have auto-scaling environment, say these days, we have like a load balancer with auto-scaled farm of web servers to be ready for your commercial running in the TV tomorrow when a lot of customers hit your site, you automatically provision more servers when this is gone, the peak is gone, you destroy them, and the day after there is no server to download your logs from. So you actually have to stream them to the log server if you wanna have any visibility of what was happening yesterday. There are many options. GreyLog has been around for a while. Elastic, Stag, Elk has been popular for the last two years. Splunk is a commercial version that had a lot of popularity. Six, five years ago, I remember working for a telco company, Expensive, but very, very good. And Elastic kind of tried to do something similar in open source way and cheaper way. There are many hosted solutions. Like that, Elok is very popular. Like many companies these days, one or two systems, they cannot hire somebody full-time to develop them a centralized log solution. You can just open an account with any of these you trust. It's always like, whom do you trust? Like, are you comfortable with your logs being with third party data log? Very popular log, the New Relic, we probably know it. Summologic, Splunk, and Elastic Cloud offers some kind of hosting, hosted solution as well. My choice is this one. You have seen it before. I would actually, this is just a little detour, but doing this, I would make it high available because here, as you can see, you are getting all of your logs into one LogStage instance. What if there is a lot of processing being done? You know, like your CPU is running at 100%. You cannot cope with the traffic. You start losing your data. So you need to kind of split it in two and need to use a message queue. So I would just use a load balancer and put at least two LogStage behind that load balancer. And these LogStage, I call them shippers. I think it's an official terminology. So they are shippers because they don't really process anything, they are just there to store the data in a message queue. If you remember that LogStage picture in the beginning, like take data from one side, store it somewhere. So this is what it's doing. It's just receiving your logs, whatever channel you want to get it into and outputs it to message queue. Maybe it's Amazon SQS because you don't know how to do that yourself. How do you store your logs? You know, like, so that's the beauty. But this LogStage shipper is very lightweight. It's just there to, you might need like really not beefy instance, very low CPU, just to put in queue. And I have two, so if I'm upgrading one of these or I need to reboot it because there is a server kernel patch, you know, I never have a downtown. So that's why I have two. And then this is the rest we have seen before. But this time I have more LogStage servers like pulling the messages, debating messages from the message queue. And I keep reading how long the message queue is. It happened to me that I had like two millions of unprocessed logs in the message queue because I didn't have alerted LogStage being done. So ideally you have the message queue being monitored and if it becomes too long, you provision another LogStage just to get it processed and then you can destroy it again. And if you store it again to Ballistic Search and you skip on it. So I just created the last thing, I'm just watching it now, yeah. Just created one little Elk demo this time. I put it on US Host Linout. It's very similar, it runs like a few dockers, the same way we just ran them here on my local machine. But it actually gets logs from a few sources. It's getting lamp logs from Japan-based Linout. It's getting lamp logs via beats from Germany-based Linout. That's where the little eShop is hosted, Bookshop. From Australia AWS Instance as well and from Australia-based Acquia Subscriptions. Just like to show you the variety. You know, like I have central LogStage solution just to play with but I can get logs from all around the world. And by using beats, which are designed to ship that, maybe the line is not reliable all the time but that's the beats it's about. It just tries to deliver it. I'm gonna show you. So I will just quickly show you how Drupal watch dock logs look like in Kibana. Maybe if you have a look at the varnish log, then I have a little teaser how you can use server metrics. And there will be maybe something else. So let's have a quick look where this works. This is the log instance we don't want. So this is my, so what did I say I will show you first on. Yeah, let's look at Drupal watch dock. So I am looking for log type and it will be here. It's a little bit, the fields you use often usually end up here at the beginning. So I'm saying that I want a log type. And if you see that it's just a quick analysis, there is log type varnish, log type Apache log type, log type nginx, log type watch dock. And I can pin it from here. So I'm saying click. I wanna see only log type watch dock. And this is what I, it already got filtered. So now I'm just looking at Drupal watch dock messages over the last one hour. And we can see what it looks like. So this is the message which came in and it got parsed. The Drupal action was page not found. Drupal request URL was this one. Drupal message was this one. So this is applying like a Drupal filter on Drupal watch dock messages. Then maybe we wanna see varnish. So I remove this, type is varnish. Varnish looks very similar to Apache. I can see that this one came from Acquia. JoIP got applied on that as well. And the last one there is, let me remind myself what it was. Yeah, I have a beat on one server which is called top beat. And when you load top beat into your Kibana and install top beat on your server, it gives you this dashboard which I just load in second, top beat dashboard. And this is metrics from one of the web servers. Over the last 15 minutes. So I can see this time was spent on Java, MySQL, PHP, top beat itself, processes, CPU usage, average memory. You can have more service like that. So it gives you some matrix very cheaply as well. So how did I get the logs inside a central alloc instance? I just used install file bit package. It's RPM, it's tap. I configured it just by saying, hey, stream these paths to this log stack. That's how easy it is. Of course, I would also use TLS certificates, but this is just to show you how quickly it's possible. Install file bit and stream the log from these three files to the log stack server. Also, I showed you some Drupal logs. The production way of storing logs is using the core syslog modules, which basically stores the watchdog events on the local server syslog facility, which you can configure like here I did. I said, hey, I created a little config file and said store these events into our log Drupal log. So I'm just using syslog module, write it to, and syslog writes it to our dedicated file. And then as we could see here in the previous slide, I'm just streaming this Drupal log file via the file bit. So it's a nice way how to get it to log stack, save it for syslog, syslog saves it as a file and then file bit streams it to central syslog. You can also use the Drupal DB log input plugin and it's more for development or when you run it locally, which you actually can see how you configure the input. You have to tell it, like, this is my database, this is my credentials and pull it every one minute. It has only a minute, it doesn't do it faster. So it's like pulling every minute straight from the database. It's another option, but not for production. And Acquia streams, Acquia has this nice, if you guys have seen Acquia subscriptions, they have their login done really nice. You can kinda say, I wanna watch PHP log, varnish log, and it keeps streaming you. They also created a log stream gem, which is publicly available. If you just search for log stream gem, you can see that it's like in the gem repository. I don't understand Ruby. But I took the gem, wrapped it in Docker container, and then I start the Docker container and you can see what it does. It just runs the log stream gem towards my Acquia subscription and saves it to a file. And then again, I just say log station, I'll read it from this file. So I'm actually getting close to real time data from Acquia into my log station. I talked to Pantheon and platform message this morning. They apparently have some way of exposing their logs remotely in the roadmap. They said, you can SSH to the server and get it or arcing it, but there is no way how to stream it at the moment. But something they know about and working on. So the last thing, last two minutes, let's see whether we have got anything of your clicks at the showcase. So, so you can say, look for everything which is request page for the last 15 minutes, looking for everything, we just sold Puppet and Siblechef in Franco. So there are some and look at the request page. And can you see what's happening here? Like this line is full, fully qualified URL, and this is just the path. Like it does the path, just what happened here? Like so, but let me show you log time. So this is varnish, Apache, varnish, Apache, ah. So that means that, you know, like, I haven't, what's the right word, not normalize, but I didn't process the log file in the same way. So what's request page in varnish, it actually comes in the log message as a full URL while from Apache log, I have only the path. And it's up to me to do the processing if I want to store it in the same field I didn't here. But, ha ha, knowing that everything which it Apache must have gone through varnish, I can say and log type is varnish. The last thing, I will create a graph out of this. I think I have a typo here. So last visualize, yes, infra code and Sible, and Sible, and Sible. I haven't set up any script which would do this. And so let's visualize that. So I will create a little pie chart from a safe search. The term is request, do you guys remember? And the field was, I think, request page. Server string, request page, yep. And Sible, 71% salt, 15. Infrastructure as a code, 11. So I think, and Sible has one, thank you very much. So wrapping up, I show you how to run the Elk stack locally running three Docker commands. Please do try, come to me, I will help you to help if you need to help. I will also write a little blog post follow up on this. We processed an Apache log file, store it. I hope it was easy. We examined it, we visualized it, and we looked at a little central logging solution which I put together at Hock, but I wanted to show you that I can get data from four different locations, three different server types, and still have all of there. There are some links. There is a very good book on LogStash from James Turnbull. There's the URL of the blog post I'm going to write. Every time you run Docker, run LogStash, it actually runs the official image. So these are the project pages of the LogStash, ElkStash, and Kibana Docker images. You don't need to know it when you run Docker, run Kibana, it does download it for you, but it's just good to know what it's doing. Any question, please come and ask me. Maybe we have time, and please, before you leave, do rate my, if you liked my presentation, please do rate my talk, I will be very grateful. Thank you very much. Guys, I have some stickers being at the meetup. I have Beats, LogStash, Kibana. If you like this stuff, I just brought it. That's not my cup of tea, but please, please have it if you like. Any question, I'm here.