 Tom here from Lawrence Systems, log management. It is, well, not the most fun part of SysAdmin, trying to solve where to put all the logs, how to get them all from many locations into one location so you can consolidate them, parse them, go through them, and create some actionable intelligence from it. And those tasks, well, they can be daunting with all the logs that all the many disparate systems we have generate. And the solution I came across that I really like is an open source tool called Graylog. Now they do have an enterprise version and we'll talk about the differences, but I'll be using the open source version in this demo today and show you how to kind of get started with it, which is relatively easy. They've even made OVA files so you can just download a VM, they've got a Docker image, and there are instructions that I followed to set this up on a standalone Debian server. Well, they were pretty easy too. We're gonna talk about how to ingest some logs, how to configure a couple different servers and kind of the process flow and how Graylog works. This is in no way endorsed or sponsored by Graylog. I'm doing this completely of my own accord because well, I wanna share a really interesting tool that I use to solve my log manager problem with all of you. Before we dive into those details, let's first, if you'd like to learn more about me and my company, head over to laurancesystem.com. If you'd like to hire a short project, there's a hires button right at the top. If you'd like to help keep this channel sponsor free and thank you to everyone who already has, there is a join button here for YouTube and a Patreon page. Your support is greatly appreciated. If you're looking for deals or discounts on products and services we offer on this channel, check out the affiliate links down below. They're in the description of all of our videos, including a link to our shirt store. We have a wide variety of shirts that we sell and new designs come out, well, randomly. So check back frequently. And finally, our forums, forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics you've seen on this channel. Now back to our content. All right, the first thing I wanna cover is open source versus enterprise. Yes, they do have an enterprise version. It comes with support and licensing. And we're gonna be doing all of this in the open source version, which does include quite a few things. And particularly ones that I was just asked about the other day was alerts and triggers. Do they work with the free version of Greylog? Yes, they do. I think that's an important distinction and that means you can not only ingest the logs, you can create some data on these and create a trigger point by which you want to send notices. Goes a little auscult, but we're gonna cover this video, this is really just getting started, but I wanted to mention that's available. Now, if you're doing only five gigs or less a day in logs, they do give you free the enterprise version as well, which has a few more features, which includes the offline archiving, which is pretty cool. And if you're working with a team of people, it's got user audit logging so you can figure out who looked at what log and sometimes for compliance, it's not just about logging, it's about logging where people went in your logging server. So they do have some advanced enterprise-level features that come with their enterprise version and I don't know how much it costs. I have no idea because like I said, we're gonna be doing this with the open source version. They also have an entire library of data sheets, case studies, webinars, tech talks, et cetera. And it will give you ideas, for example, of how the dashboard looks. If you wanna build out really cool dashboards, they have a bunch of tutorials. I wanted to do this video because it'll give an overview of how to get started and maybe something, even though I've went through a lot of this, I didn't really understand until I started really using Greylog was how to build the data sets and how to get the data in to Greylog. So this is, like I said, more of a getting started video, but they do have quite a bit of documentation. Now let's close these here and talk about actually installing Greylog. It's rather simple to install. Goes out of scope of this video. I'm not gonna walk you through the installation and I just don't feel it's necessary because you can just go here and grab a virtual machine file. They made it that easy. So if you just wanna play around with this, no commitment to a bunch of time loading a Linux VM, you can just download something like VirtualBox, VMware VirtualBox, import the OVA file and you're up and running. They also have Ubuntu, Debian, CentOS, SUSE, ChefPuppet, Docker, Amazon Web Services, any fully manual setup. So if you are a fan of building it or pulling a Docker image and putting this together, pretty straightforward there for a quick start. I did this one right here, which was follow the Debian installation. Now, my production environment versus the test environment I'm gonna be showing today, which is right here for the test environment. I did not bother putting a proxy in front of it. That is an extra step you can do to put something else to handle certificates in front of it. It says by default it just runs on port 9000. I chose to just leave it at this because this is a demo that's gonna get destroyed, but yes, that's why it says not secure for those wondering. There is another step in here about putting in something like Nginx or whatever reverse proxy of your choice in front of it if you'd like to. Probably because you want security on there. Now, let's actually talk about the process flow because this is the part that confused me a little bit and we're gonna go over to this little flow chart that I made. This was the missing piece at least in my head and maybe this will help you understand how they ingest data and gray log. They have a couple of different visuals in there about how things are parsed, but I wanted to break it down this way to kind of explain things because this is what was a little bit troublesome for me when I first got started. And we're gonna be doing all this with Syslog, but it doesn't matter what your external log data is, there's quite a few different formats that it supports. We're gonna be using Syslog for all of the demos today, but we'll cover some of the other ones that are supported in there. We take the Syslog data, we have to define an input in the system. So we take that defined input, we have the option to add an extractor. Now, extractor seemed kind of unusual, but it's a parser essentially and it's a parser that can use Grock, it can use Reg X and it takes the data then parses it into fields and basically organizes it into the Elastic database, but you don't need to have an extractor, you only need the extractor if you want things put into nice organized fields. So yes, while they're great to have, and by the way, you can have mixed content. You can have an extractor to extract some of the data, but some of the data can remain unparsed. So it kind of goes back and forth and the extractors get attached essentially to the input. And we're gonna walk through all this, I just wanted to kind of define a workflow. So we have an extractor, for example, on my PF Sense that extracts the filter logs of PF Sense into parsed data. Then we have no extractor on the Unify one that we have in here, and it just has a bunch of unformatted data. Now, the unformatted data versus formatted, yes, you can search any of the data, whether it's formatted or not, and you can write your own extractors and I will leave you the extractors that I have posted on my GitHub for my PF Sense. There is a trickiness and there's a few of them out there, the ones I will leave you with, you can just import from my GitHub and they work perfectly fine. We'll talk about how to put those in. Now, once you have that parsed and unparsed data, then it needs to go over to an index or it can go to a stream, that's why this is kind of an or. If you don't create a stream and it sounds like the stream should be part of just the way you stream data in, but it's not, a stream takes and you build rules around a stream and those rules land it in a different index. If there's no stream, everything gets to the default index, just everything lands there. That can be a little bit tricky with gray log because you've dumped everything in one single index. Each index has its own retention settings and parameters. So you can parse it still, even if everything's in one index, but the index, let's say you have a 30 day rotation, everything has to follow that index 30 day rotation. But if you have different retention for different servers, you really want them each in their own index, not just for the filtering, but so you can have retention that is relative based on how the stream parses out the data and lands it in the index. But I just wanted to cover this part real quick to kind of give an idea in an overview. It may seem confusing, but hopefully this will alleviate some of the confusion later on as we create these. And we're going to walk through the process of creating another input in addition to the two inputs that I have. So I can show you how all these inputs work. Now let's look at the inputs that are existing. We have one for PF Sense. PF Sense, you go to the logs, go to settings and my gray log server is at 192.168.3.200. So 3.200 is the IP address. And then we have the syslog right here, port number. So it's 1514 is the one I chose. Let's go back over to the gray log server. We're going to system, we're going to inputs. Here's the PF Sense input, more actions, edit input, we called it PF Sense, bound it to this IP address, which is zero zero zero, which means bound it all IP addresses. We defined 1514 as the port. Now 514 is a default syslog port. I put a one in front of it to make it simple. You can use any port number you want, but remember to deal with firewall issues, such as do you have ports under 1024 allowed open by the gray log service? Depending on the permissions you have running, that could be something that hangs you up. So I'm choosing a higher than 1024 port number. And I know I have port 1514 open. This will kind of depend on the configuration of the server. Actually this particular server, I've got the firewall turned off because just makes my life easier for doing these demonstrations. Now, port 1514, allow overriding data. Allow to override data, current date, if not parse. Yes, store full message. Yes, give me the whole message and store it. Now, pretty simple as far as that goes for creating these. And when we go over here to manage extractors, this is where the parsing comes in with the PF sense. So if we actually wanna look at the parser, we can hit extract and it's a JSON file. Now, I actually have this same one that I'll leave a link to in my forum so you can just copy and paste this in because you can literally copy the extractors or import them, you just paste it in here. So any extractors that get built, they can be dropped in. I only have a couple in here right now. They're a little tricky to write because I'm not the best at Regex, but if you are, you can actually manually write these and there's a marketplace they have where you can find a lot of these in here. It can also be done with more than just Regex. They can be done with Grock and a few other things in terms of how you wanna parse the data. But this just takes the data that comes in and if it matches the PF sense filter log, that's all this is filtering, not all data that comes in PF sense, it parses it out into the fields. And this is what that looks like in action is just a string match. So filter log and it parses that out into this. And then right here, as a matter of fact, this one's UDP, so I'll actually change this. That's just a sample. You can load any log you want, but this is just for demonstration. We kinda get you an idea, UDP. And you can see this matches and then it assigns all these right here, which are all the different fields. It's just basically parsing as a CSV, assigning the field names with the delimiting character and putting all the data in there. All right, let's go back over to the inputs again. So that's how the PF sense comes in. Now, the Unify one, no extractors. We're just grabbing raw data from the Unify. And like I said, we can still parse it. Let's go over here to edit the Unify input, port 1515, we go back over here to the Cloud key and port 1515. So this is what allows the data to flow. I've actually got debug logging turned on just because I needed more data. This is a demo lab I set up in Unify just for this particular video because we're gonna actually show how the correlation data works and how we can find things within it. Now I mentioned the streams for the next step. So the data comes in here and goes into the streams. But before you can set a stream up, you need to have the indexes. So here is the Indice and Index sets. Here is the default one. And I was doing some testing so there's actually a lot of data in the default one because if you just create something and turn it on, it'll just land in a default index until you parse it out with the stream. I have one called Unify and one called PF Sense. So there's two indexes here and two indexes here because it's doing a rotation. Now we're gonna actually create a new set in a second here and I'll show you how the rotation is handled but you can actually edit one of these, go down here by default. I have this set up to, I don't think that's a default. We'll look at the default when you set it up. Right now this is set up to one day of rotations and after 30 days, delete anything older than 30 days. You can actually change that by going here instead of saying time, we can say by size for example. We can say don't let this index get more than one gig. What happens? It rotates the index and after so many index rotations of one gig, it purges the next one there. So you can set it by size, you can set it by date. You can set it by message count. Only hold this many messages and then afterwards delete the index. This right here though where you have the option to archive the index, this is one of the enterprise features so archivification is disabled. Once again I said we're doing this all with the open source version of it. That's why that happens there. We're not gonna make any changes to this. PF sense one is the same set up. I bought both of these set up to a rotation of once per day and a number of indexes to 30 and after you're done, delete anything older than 30 days because that's all I really need to keep. Now we go over to the streams and this is where the filtering happens. So there's one called PF sense demo and Unify. The messages that come in from these, we'll go look at the actions. So we'll click on more actions, edit the stream. I called it Unify, Unify demo and this is the index set that this one lands on. Now the relationship is where the confusion may come in is how does that calling this one Unify and demo make it actually land on that index? That's where the rules come in. So you hit manage rules and there's a lot of options you can do for these rules. We're gonna do the most basic and that most basic is we're gonna open up a another window here called inputs and we'll go to the Unify and we'll hit show received messages. It's just one of the fastest ways to do this. Zoom in a little bit and we see right here, it says GL2 source input and we have this long number right here which we're just gonna copy. Let me go over here, zoom out so it's not quite so hard to read and we'll look at what this rule does. GL2 source input and by the way, it autocompletes as you start typing. So GL2 source input match exactly to this. So what this rule says is when the messages come in if they match this GL2 source input we will take them and send them to that particular index. This is what I was talking about right here where we have the stream data. If you don't have stream data lands in default if the stream matches a rule that we created such as GL2 source index equals those indexes that equal are going to be what streams it over there. But you could also create rules so we could add stream rule and because this parses out so many different things that we have in there like ICMP data, IP specific data so filter log data that I have, TCP flags for example you could create a rule, create a new index to say parse these and send them over if they match this. So you can create different indexes with very specific data. This is just a really nice feature and this is some of the flexibility I really like with Greylog. They give you very granular control. It seems like a lot but once you kind of get the flow of it you can start parsing certain things and maybe you want to keep certain data longer than others or just put all the data in one particular index for convenience and logging sake. So these are really nice features that they can do that. We're only going to create the one rule. Now we go back over here to the streams and we did the same thing here for the unified demo. So PF Sense demo, manage rules. There's that GL2 source must match exactly. If we go over here to system inputs and we look at the PF Sense you can just click again on the show receive messages. It's the fastest way to see what the GL2 source is. There's other ways you could find it but I just found click show messages it puts it up on there. By the way, let's go back over here to search where we actually see all the log data. And we can say search all messages within the last 30 minutes. Here's just raw all the data coming in by default it searches for everything. And that search query can be put just in here and it will instantly filter for that anyway. So if you did want to just filter for one particular thing the search query language is common across wherever you want it to work and et cetera from there. But now let's talk about creating a new one. So here's what it looks like for the existing ones and let's create a new one. I wanna actually take my XCP and G server and ingest all the logs from that. So we're gonna go here to gray log. First I wanna create a place for it to land. Let's create an index set and we'll call it XCP and G demo. Oops. All right, here's our XCP and G demo server analyzer standard index message count. We're gonna do this one by size. So I don't want these to get over one gig and I don't need that many. There's a lot of logs created by some of the virtualization servers. They maybe I don't need an archive of them. That's kind of an up to you thing. But once again, this is why we will want to put them in certain buckets because well storage is not unlimited. All right, we've created the place to land it. So we've created the index. Now we gotta create an input. So if we go over here to inputs we're gonna need to select our inputs. Now I said we're gonna be doing things in syslog but if you wanna do a lot of different options AWS flow, clog trail, kinesis, AWS logs, an older version beats, beats deprecated. There's actually a handful of different options and you go here to find more inputs. They have in a marketplace people who have wrote other functions and other features that you can grab off of GitHub. It's actually kind of cool. There's a lot of different things and there's a lot of support for things like let's say Palo, Palo Alto Networks, threat URL filtering, download from GitHub and there's just a link to the parser you can paste in there. So they give you quite a few things on the gray log marketplace so you can kind of find a few different things but that sometimes can create a problem because anyone can post in there. So if we search for PF sense there's a few different content packs for PF sense and I'll leave you the one that I'm using because they're not all the same and some of them are a little bit older that may have a little bit of issues working but if someone finds a better one that's a great discussion in my forums and if you link to something that's better than the ones I share, please let me know cause I'm still learning all of this even though I've been using it for a little while to parse data. So we've selected the input and we're gonna choose syslog UDP, launch new input, XCP, NG, demo, what's the port? We'll go 1566. I just like that number, store a full message. Really straightforward. We gave it a name, we set a port up unless these are something you need to tweak for the most part the default should work other than in special circumstances. Now we're ready to start receiving data. Now before I receive the data cause it'll just land in the default bucket I wanted to land in that new index instead of the default index but let's create a stream so it goes there. We're gonna do that by show receive messages open it in a new window. There's that GL2 source input again. Gonna go ahead and copy it. Streams, great stream. All this XCP and demo again and default index set. No, we wanna put it in XCP, G and demo and remove from all message stream. Yes, I don't wanna go in in both places but there may be times you do. This is one of those things that's why I said the diversity of the streams. You can create a stream that parses out some data out of another index drops it into another index but you may wanna leave it still there. So you have all of it in one place but then you have that extra data somewhere else just one of those options of why that lets you do that. So we're gonna hit save again. We're gonna go manage rules and we're going to add a stream role GL2 source input, paste that value that we have right here. So I left it open in another tab in case I forget it that's the value put that in description. If you want it save now when the data comes in it's gonna match that rule because it comes from that GL2 source which is the input source we defined and when we go here to inputs we'll see the data flowing through here. There's no extractors again we didn't do any type of extracting we're just gonna let it go and flow with unformatted raw data. So now we're gonna go here to XCP and G and put the server in which is right here on this page which is the host page specifically. Remote syslog server, 1566. So 192.168.3200.1566 and now we can start ingesting logs from here and let's start making some logs by turning on this. This is the Windows 10 VM that I have in here and we can actually probably see right away some data moving. Yep, 1.3 kilobits. Let's go over here to search. There's all the data and let's filter it for just this right here. Clearly I did something wrong cause there's no data in here. Let's close this real quick and say let's search for this particular UUID which was the UUID we copied from here. Okay, the data is going to the default index set so I clearly just messed up the stream. Let's go back to the stream. Oh, I didn't start the stream. Whoops, there we go. Start stream. Go back over here to search. Go back over here to XCPNG demo. There's a little play button right here to say update automatically and okay, now all the data's going once we started the stream. Some of that data is now left over the default index. But here we are and if we go back here and copy this again, let's filter for only data related to this particular VMs. We'll go ahead and stop it and we can see it now parsing that data and it's only giving me the data for this. It's pretty easy even though the data itself is unformatted so it's all just dropped right here into the message format. It's not parsed out into individual fields but you get the idea that even with that I can still index this and sort it and start understanding the data that's flowing through. Now let's go over and talk about the Unify side. So here's the data for that but we also put in the Unify system and the you have since demo system here. Oh, like that. At this, it plays so it automatically updates. Now some of the data such as the filter log data I'd mentioned we have that parsed. If we look here at the parse data for this it's all nice and organized not just dropped into a message. So we can say protocol TCP and we can then filter so we can maybe build statistics on it or add to a table we can actually say show top values. There are UDP and TCP so we have 1200 over this time period. Actually let's change the time period a little bit further. Start it updating again and you can see how we can start slicing the data up. There we go and we have that many over the last day and because we told it to keep updating it's still changing these counts as things go on. Now my phone's actually connected to the firewall behind this so we can probably open up stuff on my phone and let's do an update. If there's some app updates and this will probably make it go that much faster with all the connections that are going. Still not too much but like I said this is a lab environment so that's what's on there. Now speaking of my phone let's figure out what IP address my phone is by going over here firewall actually a DCP server under services and right there is Tom's phone 192.168.40.138. Let's go back over to the gray log interface I got two of them open. Kind of start this over and we're going to find my phone's IP address and everything it's doing but right here we filtered for the IP address and once again here's the filter logs. Here's places it's contacting. I bet that's Google if I had to guess or where it probably looked up because I'm downloading some updates and source IP. So let's actually filter for add to query. So let's instead of just filtering free form we can go through and say source IP just for this particular piece of information in here and you can see it's going through the filter log and pulling up all that information. Now let's actually go specifically to the Unify and we'll query this again actually get rid of this, turn this on to update and I'm going to take my phone and drop it from the Wi-Fi and put it back on. We turned off Wi-Fi on my phone and we actually see in the log here it's going to tell me that it came off and then we put it back on. So now we can go through and say all right what's connecting to here and right there I can capture the MAC address and the IP assigned to it. So right there's the MAC address for my phone. So we'll go ahead and hit copy and we're just going to paste it up here. Show me the MAC address. Now let's add in the PF sense again then update the query. From here now we've got logs from both PF sense and from Unify going in one spot. And this gets kind of cool because this is that correlation data from two sources especially and I bring these two up because I know it's a popular question that comes up is when I'm having a DC problem looking at the logs from both devices the logs that are going on inside of Unify and the logs that are going on in PF sense simultaneously and walking through each device and parsing it out. This is where you can kind of get the information and consolidate it to try and troubleshoot some types of networking problem. Because if you didn't see the logs because we have a discover request right here from DHCP we actually start with right here the handshake. So custom wireless event I think we go a little further should be a wireless event that is the actual authorization. So let's go down right here says code W E zero do, do, do, do, do. All right, found it right here. Zoom in a little bit. Here is the first piece of this where it started the authentication for the WPA key. You can then start with the surrounding messages for up to 10 seconds around this open up a new window and start organizing the data, zoom it out so it's a little more on the screen and start organizing the data around it to try to sort out what happened. This is some of the ways I've really grown the love gray log as it allows me to do this relatively quickly and find certain pieces of data I'm looking for whether it be wifi data whether it be data in a syslog server from something else such as Apache from a web server putting all the logs in one place and be able to follow like from someone logging in on a VPN and figuring out what other servers they may have touched internally is great from a security standpoint and also from a troubleshooting standpoint. So hopefully this kind of gave you an idea how to get started with gray log. I will mention there is a whole bunch of dashboards that can be done. I've done a little bit of testing with it such as a building source dashboards and all kinds of analytics. I'm not great at it. Maybe in a future video I'll cover that but they do cover some more of this in their own documentation of how to set these up and how to build things like messages per source based on source of the filter logs, et cetera. Now also beyond the scope of this and something beyond the scope of me at the moment this will do threat intelligence looks up on your firewall. That's something that at some point I might get a little bit more into is actually using this for a deeper level of threat intelligence. Now the last thing I want to touch on is hardware what we have it running on and how it runs. This is a XC PNG server with an Intel Xeon E5 26070. Nothing outstanding in terms of processor does an older R720 server this particular machine is running on because it has a lot of storage to be able to store all the logs. Right now we have about 120 gigs worth of logs in our server and over here to the console for stats. I was playing around digging through stuff but generally speaking if we look over the last week it just does not use a lot of processor. It's not all that intensive on the processor. Now let's actually talk about what it looks like to pull some data. Right here we have H-top pulled up and you can see the load average is relatively low and here's the interface which we're looking at roughly four to five million logs coming in a day over the last 30 days. Let's find my IP address inside here. 168.3.9 We're gonna search back 30 days worth of logs and everywhere that it has a log where my IP address did something. You can see it's updating results. You can see the processor and the load going up a little bit while it does the query. And 30 days of results took a few seconds to do and you can see what days I was here or not here. I was really doing something on January 20th to generate quite a few logs that particular day but it's not a problem. It's even faster for doing things like when you say let's just say what did Tom do in the last two hours? That's pretty much instantaneous one day. Pretty much instantaneous 30 days. Well, we've did it, it's now indexed so it was able to pull it up relatively fast. We haven't really had any problems running it in production. It seems to work quite well. I haven't had any crashing issues or anything like that at all with it and it's not on the highest end of hardware that we have in our office. I did that on purpose. Well one because I had a lot of storage on it, our 720 server and two. I wanted to see how it handled or does it need something really high end to run and it turns out no, it really doesn't seem to. I've, even with the amount of logs that we ingest on it and as much data and it's going through the rotation because we mostly for most of the logs are purged out after only 30 days. That's all we need to keep for certain logs. We have longer for others, et cetera. But yeah, the system's been pretty stable. It survived a couple updates and hasn't given me any trouble. That's why I wanted to review it because the question comes up all the time about what do you use to ingest all your logging server? And I know there's a lot of people with home labs that are looking for, you know, kind of an easy setup log server and it gets you in started in that logging. So this is actually something I really recommend for people if you want to start out using your home lab, it's relatively easy to set up. You can just grab the OVA file and you can start just finding all the data that you have and all the different things you set up and just start piping it all there. Let me know in the forums because I'll leave a link to the JSON file that I have for my PF sense. But if you find some other really good use cases or extractors and other things that I didn't cover in this video that you would like me to do a follow-up video on, let me know. I will leave links of course to their documentation. It's pretty easy to go through that. They have a lot more tutorials on some of the, once you have it set up. That's why I wanted to make this getting started video to kind of try it out, learn with it, test it out and kind of wrap my head around how to get the data into it. But once you have the data into it, playing around as you can see is pretty easy to kind of sort out how you want to parse the data and just figuring out what dashboards you want to build and figure out what statistics you want to generate and of course, figuring out the extractors. I'm not great at REGX. I'm still going through all that. So if someone has a good suggestion on that or wants to help write some, I would probably hire someone to do some of that. It depends when you're watching this video whether or not I've got that accomplished or not. But leave a message. I'll leave a link to the forum post. That'll be the place to post that information. Thanks. And thank you for making it to the end of the video. If you liked this video, please give it a thumbs up. If you'd like to see more content from the channel, hit the subscribe button and hit the bell icon if you'd like YouTube to notify you when new videos come out. If you'd like to hire us, head over to laurancesystems.com fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on. If you want to carry on the discussion, head over to forums.laurancesystems.com where we can carry on the discussion about this video, other videos or other tech topics in general, even suggestions for new videos that are accepted right there on our forums, which are free. Also, if you'd like to help the channel in other ways, head over to our affiliate page. We have a lot of great tech offers for you. And once again, thanks for watching and see you next time.