 We are live Welcome to the homelab show episode number 14 gray log and we're talking about gray log because logging it's kind of a problematic area in open source Most of the products in this category are not open source In which leads us always to just running a sys log because we're like hey I can set up our sys log and send everything to one server So you do that but that doesn't necessarily you put the log somewhere It doesn't really solve the problem of doing something with them or creating action items or making them easy to find So we figured we'd pick this as a topic because this is free and open source for you to get going in your system It is a product that has a commercial backing So yes, there is a paid version and paid support that can go for people that want to use this in the enterprise as well So it's got dual purpose, but it's it's gonna be fun topic and Jay's been looking at I've been using it Jay's been looking at it as a product. We figured this is a good time to address it before we do that with a little housekeeping this Video literally is brought to you by Linode Well this podcast and if you're downloaded this podcast saw you pull it off on the Linode servers that run the homelab Dot show website and Jay's been a longtime Linode user and What do you think Jay? We'll keep going with Linode. I think so. I love the interface I've been using it for I don't even maybe it's going on three years now I always say to but at some point at the stop saying two years as I switched everything over it might even be three or four I lose track of time I've been using it ever since that pengu con that we went to you were there too When they were showing or they had a little stand and they they gave out the card so you can get a free credit And I used it and I'm like oh, I like it. So literally everything on learn Linux TV is on Linode This podcast is on there too as Tom mentioned and I love their marketplace. They have a lot of one-click apps They have all kinds of features node balancers You can upload your own images now and another thing I've always liked about them and even before they had an images feature properly It was very easily is very excuse me very easy to use DD and just take a local backup of your cloud server Which is awesome And now you can upload that right up there and spin up a server with a custom image So they have a lot of great features and I like it quite a bit Yeah, Ben Ben saw platform a longtime user of Linode myself to for lots of little projects That I've done on the channel over the years and you know, it's a good solid system So we do have an offer code for those you that would like to get started with Linode It'll be down in the links below also You can run gray log in Linode with that's one of things me and Jay were talking about is Where you'll in gray log will be part of this conversation But it's open source there's not any licenses attached to it for the free version therefore you can spin up an instance local and one in Linode and we'll talk about why that might be a good idea because you really want your log server close to wherever Your servers are and wherever your workload is because you do have to get all that data over to where it needs to be Yep, and if you're lucky enough to have a static IP, they have a firewall feature in Linode now So even if you do roll it up roll it out over there You can basically make it only accessible by you in your local land Even though it's not in your land which gives you the flexibility to control. What's local and what's not? Yeah, the reason we're also swing it back to logging and one of the problems the logs is the Promise that every salesperson pitches the single pane of glass the single source of truth where you can learn about what's going on in your stack This is the promise every logging company makes and it is it can be done It's there are products out there that do it But it's also a challenge because you have to ingest all the different log formats and put them all in one place Now one of the real reasons this is important is because let's say I have a web server then I have a series of VPN users that may log in then I have a Firewall and with those different data sources I want to be able to track when the user logged in Correlate that data against what they may have done on the web server What changes they may have made correlate that again with what they ssh'd into and maybe look again at you know Timeframes and how frequently they log in and then build triggers off of that This is also where firewall logs become really important when you have a system breach Where you need to reverse engineer how they got there and unfortunately, this is one of those Missing components that many companies that are breached have is they don't have a good logging system They just have to run around from computer to computer looking at the logs of each one to try to really determine what happened This is where something like gray log can come in because it's not just be able It's not just able to pull in Linux logs by the way We can go way further than Linux logs. It'll ingest actually a massive list of formats This is one of things that we'll start with real quick is J's familiar with AWS So among the inputs as it referred to in gray log inputs are different ways to ingest the logs is AWS cloud shell AWS flow logs AWS kinesis AWS general logs beats I'm not as familiar with CEF am QP Kafka logs TCP logs UDP logs HTTP logs Guelph style logs, which isn't gelt. That's an export for Apache is it right? I'm not familiar. Yeah, it's I was reading a little bit about it like Pulling JSON pass from HTTP NetFlow UDP data There's a series of Palo L auto so back to the enterprise stuff like Palo Alto's network TCP pan OS version logs version 8 and version 9 Raw plain text logs via TCP and of course syslog, you know the usual The one that at least most things will export and yes, there is a way to Also ingest windows logging from here that we won't get too much into that topic But if you just Google real quick how to do it they have work instruction on how to get that going Yeah, sounds great to me. I don't do a really good job with this So basically, you know, I think people roll their eyes when they find out how I do it and this is why I wanted to look at gray log, I'm literally Manually going from one server to another so SSH into server a and then I'll go to PF sense and go to the diagnostic logs And the browser have a terminal window on one side I'm going to random servers and trying to correlate myself That's a lot of work to do manually so I don't want to do that manually anymore And one of the interesting things I've done is you can pipe in true nas and your hypervisor in my case XP and G and then you can get two pieces of correlation data if you're having a problem with a virtual machine and This will allow you to look at what happened with the storage server and correlate it with a time slice of what was going on in a Virtual machine like hey suddenly true nas is under a really high load What caused that let's swing over to this. Oh, this virtual machine also was under high load at the same time Oh, by the way, it has a storage target of that true nas so these are really helpful tools when you want to start troubleshooting and I should look that beforehand but the gray log Gray log extended log format is gulf, but it's actually really popular and some things that are like fluent D Also has the ability to speak that so there's a ton of large enterprise things that can ingest logs in this Which is kind of cool because learning this as a tool you're gonna find it's in use in the enterprise world as well This is one of the things we like at the homelab show is to bring you products that have that dual purpose They're not just oh, I'm using it, but I'll never see this in the real world No, you're very likely to find this at a large enterprise And we know some larger companies that I've talked to that are using this product and are pretty happy with it I've actually talked to the team over at gray log. They're really cool They got a cool community for them, but I guess we have to first start before you can get to any of these cool features is installing gray log Now until recently and and recently being the last version it came out about a week ago And that's when I found out apparently they're not doing the virtual machine updates They were just talking about that right before we went live. Yeah. Yeah, I didn't realize that they had stopped doing that so So basically yes as I understand it from what Tom mentioned they they offered Virtual machine images for various was it various? hyperfisers or just a generic they were just supporting virtual box, but they gave it to you an OVA format This allowed you to import a lot of different things support the open virtualization architecture So you can import them into a lot of different platforms It's You know, it's pretty easy though to install matter of fact they they have installation for a bunch to Debbie and sent to us They also have chef puppet and Ansible different playbooks and scripts set up They have a docker so you can build it in Docker. You can deploy this in Amazon web services. So it's actually part of AWS. We probably should have looked before the show It may even be available inside of the node as a as a system If not, I'm sure we can even talk with a note and see if they can do that It's actually not I just okay But you know usually anytime someone mentions that it has a way of happening because they have the stack scripts So I saw someone I think it was Gardner from the one of my friends from YouTube that created the Jitsie Version that they have up there that automatically installs that so I mean anyone can do it and it'll probably happen now that we mentioned it just takes one person Hey, I can do that. Yeah And the fact that they've already put some shot chef public chef puppet Ansible and docker together One of those tools will be used to help build an auto deployment and Linode so I'm sure Linode's listening and we know some people so Now Installation overall pretty easy one of the problems in the logging world is if you've ever built your own Elkin elastic stack and put it together and don't give you wrong Those are cool products and there's a component of those that gets enter and agree log I always feel like I set up a house of cards that I'm afraid to update because it'll break that has been a Consistent problem of because I know there's always the what about these other open source was I don't think they're bad I think they're a little bit more challenging to maintain Great law gives you kind of an ecosystem where I don't mind because they do offer I built mine in Debian and you can load the Debian repositories on there And I've had excellent luck with each of the updates doing exactly what I wanted it to do it updated to the later version without destroying it and Also had to do is you know sometimes rebuild or make database changes based on changes that came down the stream But that's the point. I'm not scared. I'm not in a panic attack to click apt-get update That's one of the problems I that I've seen a few people complain about is oh the new version of this is out But I know if I update it things might break They are using some Java and some elastic in there But they're doing it in a way that they maintain so you don't have to worry about it There are some prerequisite warnings when you first set this up that if you run your own elastic stack instead of using the way They build it you can run into problems and you're kind of on your own But they're pretty clear about that from the upgrade right from the front and their instructions are solid like you can go through absolutely step-by-step copy-paste out of their Installation and it works. I've built this a couple times There is some important points and Jay asked me if there were some gotchas in the config and there's at least one of them that is Definitely a silly problem, but it's it'll it'll get you because it'll create a lot of confusion So once you set great log up and you start Setting up the inputs one of things is cool is the inputs tell you the rate at which to receiving logs And this is part of the troubleshooting So first is you know, did you have the firewall port open to allow logs and you build a series of inputs? So if your gray log instead of being like a one spot with one port open to ingest all the logs You can be very granular and you create an input for each server you want to ingest doing it This way gives you a lot more control and we'll get to that later about why that's a good thing in a way They do the parsing but The nice thing is just telling you how many kilobits of data is coming in is very helpful So you'll get it set up you set up your first input and you watch these kilobits of data come in But then you go to the main dashboard and there's no data Now this is the gotcha in the config that you'll find a lot of posts on the forums about of it's taking data in the Indexes are getting bigger in the data sets, but there's no data being displayed The default dashboard displays to showing you data from the last five minutes Why that's important is because the time zone is set in the config if your log server is in Eastern standard time like mine is and you chose not to change it to Eastern standard time It is very important that the log times line up or it's not putting them in the same time zone It may be putting them in the future or in the past depending on your time zone But that creates a lot of confusion because you'll actually see all these logs coming in and the system index is getting bigger But you're looking for maybe 10 minutes and you pull down you'll show me what happened in the last hour Well, if it's 12 hours out of sync with you You have to go for a full 24 hours and then you'll start seeing the logs and it just creates confusion because first You don't see the logs and you go. Oh, they're there, but they're all the wrong time time zones matter a lot with logging systems So it is important that you make sure that the time zone of the server is in your time zone So that's it's a small gotcha But boy is it easily overlooked when you're setting up a demo server. You're like, I'll set the time zone later Don't do that No, no, so so are you referring to the time zone settings on the Linux server itself? Or is there like a separate it both Linux server and in the config files when you walk through the config It's very well labeled There is like a master server config for gray log itself where you can set a ton of different parameters once you're doing this though The that's where you got little things like time zone that you just need to set it I can't I think it says to set the time zone But it's probably only easily skipped over things and I would it had me stumped for longer than I almost want to admit Why my log server wasn't working properly Now multiple servers set up if you plan to have multiple servers taking care of different roles in your cluster You need to modify a few things covered in the multi-node setup guy goes on the scope of this and probably not needed for your Home lab, but I've seen a question fly by in the chat What if you know gray logs and available for an update you can build? multi-node large-scale server systems and cluster them together gray node scales to Large company size for enterprise log ingestion So it's important to note if you want to build a scalable system They even have an instruction for doing that probably out of scope of your home lab and what's needed But nonetheless, yes, you can do that. Yes, there are ways for you know getting that working Now few of the components it runs on I mentioned Java. It also uses MongoDB, but don't panic It uses MongoDB for some of the settings. It's not the main database where everything is kept And the elastic search gray log can be used with the last it search 7.x follow the instructions that they have inside there to get that set up, but the elastic search system is Is specifically Reigned in as as you will it doesn't quite use their query language natively they have their own but it's pretty intuitive when you get to the logging in the queries on there So that is a prerequisite that you have on there. And of course then you have the gray log engine itself resource intensiveness That can be a little bit of something to think about it depends on how much of the logs you're ingesting will Generally determine how much resources you have to this so I want to give an example of what I'm running here at the office and currently we have about 53 gigs of data Stored in there, which is is parsing all my firewall logs from pf sense which makes up the majority of it and I get roughly let me change it to One hour so I can get you so you in real time how many logs are being ingested on there I get about 4,000 data points per minute and that's mostly coming from pf sense Actually 90% of that comes from pf sense where I have it logging all the firewall logs and even with a system like this It's using about four and a half or five gigs worth of memory to run So if you start ingesting a lot of logs, you're gonna have to scale the system accordingly So if you look at it from that and that standpoint like you know, ingesting firewall logs Yes, you're gonna need to have a beef your server if you want to run this to monitor your web servers Depending on how busy they are and depending on how many hits those particular servers get you'll end up with You know a much much smaller scale. I also have my unified system is piped into here as well So unified produces very very small I have 30 days of retention with all my unified logs Which is all my switches and everything else only accounts for about 600 megs of data over 30 days versus the 45 gigs That pf sense produces every 30 days. So scaling on the server You're gonna have to kind of take a look at what you're ingesting and how you're parsing it and you can set per server limits on how much data you take in maybe for things like a True NAS server and your virtualization server seven days of logs Maybe all you need because you're usually dealing with problems kind of more real time Then going back, but of course if you work in the corporate enterprise world Absolutely, you're gonna have to kind of figure out What those retention policies are sometimes policies like where my friend works at a large service security company six months? They have petabyte servers that store logs because they have six month policy retention across a hundred and sixty thousand devices Yeah, he says that he says they have just a These is they have just big data arrays that are just storage for logs It's just like when the solar winds it's gonna happen I asked him if he was able to parse that he goes We were able to answer the question for the last six months where we in the solar winds Incident involved in any way because they can reverse look up an IP address for six months that any server had talked to I'm like, wow, they they spend a lot of money on that it's this is why it's kind of an important aspect when you get to the corporate scale and I'm Trust me the majority of companies you talk to will not have that answer They're like, we're able to keep seven days worth of logs You know, what's weird is that some companies don't even know what their retention requirements are Like like let that sink in for a minute like you like if I was working with a client and I and I you know They want to set up a server like this they would probably ask You know or I would ask them how long to keep the logs for It's surprising how few people in the company know their own requirements And yet they have to go find the person that wrote the policy and knows where the policy is so If you are going to roll it out to a company You should at least know the you know what the requirements are there But the value for homelab people we don't have legal requirements as far as I've ever been made aware of So we can have one day two days three days three months whatever we feel like Yeah, and if you can afford more great and but right away you'll you'll run into the problem Even that's somewhat I do is balancing out, you know, I keep 30 days I've thought about extending a little further, but I'm going to have to up the resources I have dedicated specifically the storage To that server now one thing this is where there's a little bit of a gotcha is active data versus archive data that is a paid feature of Gray log and one of those things that is kind of an upsell for the product if you like to archive data off of there and Using it in archive is nice because essentially goes Static it's not part of the active database, but can be queried as needed and brought back in That is one of the paid features that they offer If you want to buy the commercial version I'll just mention that out there as people may say Well, can't you just keep putting it somewhere like compress it and stack it over here? That is one of the so-to-speak upsell features of it A little little bit of a balancing act figuring out how much you actually need But that is an option for people that use the enterprise version Now let's dive into log ingestion as I mentioned there's lots of different formats out there So you'll have to make a determination of which one works for you But we'll just say sysmon because that's your standards or syslog Your standard syslogs are what you're going to find most everything has an output for the unify system The true nas system while your linux and bsd based systems generally have no problems sending that data out You also can pipe specific types of logs like things out of apache on there From there, they have the filtering system that brings them in And that's where it gets pretty neat because you build these and I have them shared on my github And so there's a lot of them in the forum. They have a marketplace of free extractors extractor is the word they use So as the data comes in The extractors take unstructured data because data structures are although similar Maybe not be the same across each device that you import So you have an extractor that will line all the fields up Now if you ingest unstructured data in randomness No problem. It actually has a way to parse that even in their search system But it's way better when you assign field names to it that way you can search by field name Whether that be ip address or The way it labels drives or any of the other type of data you're in there The extractor takes the data and uses regex You can be done with other ways too, but regex is the most popular way to do it And you just define all the field like in this position because it's a lot of times just comma delimited depending on how it's brought in or space delimited You say based on this marker this separator put it into this category This is really handy, especially with firewall logs because it tells me the filter rule That that firewall log was being processed. It tells me the inbound ip address the outbound ip address The nat rule if there was one it'll tell me what the outside port was and what the inside port was So for example in firewall rules once the extractor parses all these I can start filtering in the dashboard for certain ip addresses find me this public ip Then pivot and say show me everybody internally The internal ip is that ever talk to this external ip address This is why the structured data Becomes so important on there when you're doing these extractors and the tractors are they're done I I almost used the word plain english, but that depends whether or not you speak regex So they're they are human parsable humans created them They can be a bit tricky But what I did myself and I covered this because I have a full tutorial on gray log as well Is taking these extractors and I just downloaded somebody else's and modified what didn't work to add the extra field For what did work for what I needed So it's not it's daunting at first depending on where your skill set is with regex I'm I wish I was better at regex. I I need to just like sit down grab a book and Make a weekend of it and beat it back into myself. I think jay's probably a lot better at than me I wouldn't make that assumption if I were you I need to do exactly the same thing Actually, yeah, if not we we will um, we'll have to call our friend whose license plate is said awk I don't know if you know that Yeah, that's yeah, that's Phil's license plate. Uh, what our friend I think thinks in regex, man Yeah, I never knew he had that license plate. Yes. I love that license plate. So Nothing and before someone says hey, you're giving away details about him It's just pictures of it online already. So he's I mean, you don't get that license plate not brag about it. So Oh All right, next thing is streams This is kind of Where I I have in my video a breakdown. I wish in matter of fact, I've actually talked to gray log It says can you guys build a chart of this and I built one for my video Basically, it's a flow chart to talk about how the data comes in So first you create an input you ingest the logs you run it through the extractor to create a formatted data Then you create the streams Now the stream is one more intermediary step that has alerts and rules that Break down the data now the alerts and rules are important because There's a lot of data coming in and you don't necessarily want to trigger on anything So you want to make sure when you're building these like, okay, where does this data go? Where should it land and it can land in more than one spot? So you create a series of indexes for this to land on So each one of these different indexes or sections of the database essentially one for pfSense one for unify One for each one of your servers. You can put them all in one It's just just me way easier because your retention policies are set on a per index You then want each stream to stream to that index with perhaps different rule sets Um in that rule set like I said can be removing some of the extraneous data that you know is not important Because sometimes with servers when you're sending out syslog data It just sends it all and that may not be the best thing for you to send all the data This is where you can get a little more fine grain and you can also say look, I only need seven days of this data I don't even care about it after seven days. It's not real relevant Or maybe even parse out things that are not important because there's all kinds of noise or just notice level data I mean ideally you go to the log server or whatever sending the logs and turn notice data down But I live in a real world where that isn't always possible some servers Unify can be one of them. It creates a pretty decent amount of logs that are not always really helpful Um, so you just kind of say either you keep them or you don't but you can that's what these streams are for Is to kind of narrow that part down But I have found it really helpful because what happens Behind the scenes when your wi-fi is working is a good example of this as you roam between Devices and as things move around the network. There actually is a lot of handshake logs There's the whole process of passing off Testing the rssi or whatever parameters you have in your wi-fi and saying this is the vice go over here Did this vice go over there? And actually I found the logs is to be very useful in that because we we ran into a weird Handshaking problem where devices wouldn't roam and it was helpful to figure out What devices would give the error and what that error was And not have that lost in time Especially because when you can see when it happens frequently over a few days It can be very helpful But it also is a lot of logging data To be able to stress that and it helped us determine I think I can't remember when they'll mention it on the show or before the show when we had them on Remember how we saw about mac addresses changing mid-flight? I think so. Yeah. Yeah. As a matter of fact, I do remember that Yeah, this is one of the things we were able to look for was we could see the ip address not changing With the mac address changing and the data set and we're like that's not supposed to happen It didn't get reassigned, but that led to understanding why the handshake wasn't working These are some of those little extraneous things that you start getting out of there Now let's talk about the alert system. I'm not the best at setting this up They do have good documentation on it, but alerts are those trigger points and This can lead further into something I haven't done yet, but I'm going to be working with their team That they've done a lot of tighter integration with You can build series of alerts those alerts can be based on things like all right this many logs a log related to a specific system Or just a quantity of logs. So you can set either, you know You parse it for a certain word a certain set of words or a triggered event because there's too many logs Suddenly someone's trying to log in with open vpn. Someone tried more than three times with a password Okay, now we have a trigger an actual event and then you can have it set to alert you Where this goes a step further and where gray logs extending capability and if you go on their site They talk about being essentially like a sim tool. So a security and events monitoring The ability they give you is to pull in other data And let's say you have a list of known bad ip addresses or active threats And for those of you who ever want to dive into what active threat looks like The free version of the free online account you can get with alien vault is a great way to look up active threats You can start pulling data like that in there And what the alien vault does it will create correlation data with free feeds It's a these ip addresses are known command and control servers for some particular threat Of course, you would love to know if a server in your office is talking to one of those command and control servers Because that means something bad is about to happen. These are what we refer to in the security world as indicators of compromise Well, these are also things that there is the ability to and granted I'm speaking from something I have not built yet inside of gray log But this is something I'm really looking at doing if gray log going a step further Pulling in threat feeds and lists of ip addresses and then alerting you if it finds that one of your systems has decided To start talking to those ip addresses This is why I find it so important to have things like your firewall logs being ingested in there Because that gives you that piece of data that you're looking for to go. All right, you know This is going in there and all of a sudden I don't know why but it's he started talking to this command and control server and What you want historically is How long has it been talking so you can try to determine when it happened because when it reaches one of those threat intelligent feeds Like the one I mentioned with alien vault when that's when alien vault found out about it That doesn't mean that when it started to exist you want to know when that When something started talking to it that gives you a better idea what happened on that day And this is where going back historically is extremely helpful when you have these being able to put these in And this goes back to the first part of the conversation about importing structured data So if you know the ip address you can type in in the field for search in a dashboard ip equals this Then show me over this amount of time and now you're drilling down to well the alien vault feed triggered on tuesday But it was last tuesday when this device started talking to it and you can start you know Running around like a mad person and locking everything down figuring out what they had resources and access to hopefully before You become a cyber security incident because that's that's It's funny how a lot of this is very reactionary in the security world It's when we find these command and control servers that we have to start reversing it But unfortunately many of them are found as things get detonated and As as the command and control server starts sending out all their Things that's when often we learn about them. So hopefully you can catch it before all of that But it does have that full alert system in there built in for doing that Um, like you said, there's actually a lot of trigger alerts where you can look for system load information and things like that Anything you can parse out of there and then build alerts on Can be very helpful and this is especially if you were looking for weird attacks on your web servers If you're ingesting all your web server logs This is that same thing create triggers for when someone keeps trying really hard on the admin page or Some function on there You can put that data in there and now you have a way to go and reverse What's going on one of the things I I see a lot of in my logs is WordPress attacks against my non WordPress web servers Which is just kind of fun to watch but you can alert for certain types of attacks you may see against a WordPress system especially like The shell escapes. I see a lot of those backslash backslash backslash root backslash edgy backslash pata Etsy shadow and things like that. You'll see them and you're like, why are they escaping it? And it also becomes if you start google searching it great ways to learn about how people attack web servers A really good point Yeah, what do you think jay about running this in in linode? You could probably gain some insight if you started piping all the data you have on your linode servers Yeah, I think I'm going to I'm setting it up locally and probably another one in linode I'm going to try to split them and then kind of see where it goes I'm actually running it or installing it right now. As a matter of fact So I think there's some I don't want to say concerns or just I should say questions because I don't know if this is an issue with graylog And maybe you'll know the answer Have you seen situations where Logging itself becomes a network problem because of all the data being thrown across the wire From all the servers all going to this one. So you you have a lot of traffic there if you don't really I think it might have been elks tech I'm trying to remember where if you don't kind of set some limits then it could actually cause the hyo issues is That's something to take into consideration. It is worth considering As you scale up the data The server has to process and index it. That's a database function That means it's got some processor usage going on and a lot of hard drive access going on And this is where let's say I'm watching a virtualization server because I am and let's say that virtualization server is also running graylog And I'm suddenly producing a lot of logs because my virtualization server is at a stall for something else Now graylog just got a massive amount of logs, which is going to turn up the hard drive And create that so there is that ample amplification As far as bandwidth goes logs are generally because they're plain text not huge You're not sending actual data. You're just saying the plain text Uh, it's but it's something to note The other thing to note is that you should probably have your logs as much as possible secured and locked down on a separate management network Uh, obviously passing some of the logs do pass in plain text. They don't always pass securely Make sure of a lot of devices. They don't have a secure because uh syslog is just It's an old protocol sent on Open ports essentially so just kind of be aware that The systems will have problems Um of slight note when you're setting up graylog and I probably should have mentioned this beginning, but it's still When I say low port number for syslog, you should ingest on a higher port number There's a little warning that pops up to remind you. Uh, but graylog does not run as root They did a when you if you've gone through the instructions jave may have noticed it actually sets up, uh I believe a graylog user for uh management. This helps, you know Mitigate if there was a flaw in graylog privilege escalation as much as they can But that being important that also means you want to open up ports above 10 24 for ingestion So even though your syslog server in most of them are low, I think it's 514 I think a syslog off the top of my head Um when you're setting up ports to open on your graylog server You're generally choosing higher port numbers because it's in the user space because graylog's running as its own user on the system Um, they did structure the back end of it that way, but these are also things. Um As he someone mentioned in the comments, you can't pass graylog logs via tls Graylog absolutely accepts encrypted log formats. It's whether or not what you're ingesting from Has that option to do it? That's it's it's less a problem of graylog and more a problem on the other end of What you're sending and unfortunately if you're you know, especially printers and things like that Maybe you want a bunch of printer logs in there and this could be important If you're dealing with you know, an enterprise environment I'm sorry. Those printers were lucky if we can get them support any type of secure protocol We're just happy if the printers don't have telnet enabled in 2021 We're we're we got a we got a really low bar here Uh, but printer logs could be something else that you want to look at and ingest them all and Consolidate all your printers into one place and that's going to be a use case, which I really I would be amazed if printers sent things over tls Yeah, I would do printers are like the bane of the it persons existence like every Company I've ever worked for during every stage of my career from entry level All the way up to administrator and beyond printers and print servers have always been a nightmare to deal with Yeah, uh, and someone pointed out Someone did point out a valid point of you could always do uh traffic shaping and set these at a lower traffic priority on the network If if the traffic part was a uh concern, you know generally networks are With exceptions, of course if you're especially if this is working remotely Locally You're gonna have plenty of bandwidth But there the restrictions really start to come in is if I wanted to ingest logs from the web Now first if i'm gonna ingest them from the web back to my office here locally in that circumstance I'm really not gonna want to do it via You know something that's done in clear text for one Uh, you may want to do that best ideas over vpn But of course now you're talking about real restrictions of how much bandwidth do you have and can allocate to it? But once again Is it I look at if you're building something like a modern web server You're gonna tune your apache logs or engine x logs or ha proxy system To only really send you the data that's relevant and those are modern systems that you can modify how the logs work So you can fine tune it properly to get the right logging information the actionable one not all the noise I mean, maybe you need to know all the requests or maybe you don't need to know every line item request of everything Someone hit you want to know the connection they made was a successful certain transactions You maybe want to know if the database reaches certain load levels and but you don't need to know every query If you start sending every query down the pipe, you're gonna create your own problems Yep, totally agree. So what's your opinion on storage? Somebody asked about that in the live chat. So We have a couple different places. We can put gray log. We can install it Obviously on Linux server or we can you know locally or on linode But when it comes to the storage, I mean already i'm thinking lvm For sure if you're gonna use uh block storage, you should always use lvm. You can expand live But beyond that, uh, what are your opinions based on what you've seen when it comes to where um, it's best to store and also Whether nfs is a good idea or a bad idea? Um, you know because you're building on a standard linux platform They do have documenting gray log where it saves its data. So you And this is just good hygiene of when you're building the virtual machine build the virtual machine and then creating mount where it mounts a data store Maybe that data store either via nfs smb However, you want to mount it within that machine and then mount it over or even ice because there's plenty of options here That way you're separating the operating system drive essentially or where the operating system is stored and where the data is stored because Especially if you you thinking you maybe you're going to ingest a lot of logs something like a You know a share over at you know in nfs will work if you properly set that up What was the other one used? Is it sshfs? I do use sshfs, but there's going to be some overhead there. I'm a huge fan of um auto fs Auto Yeah, well, that's what plex and all it is is just an overlay on top of whatever you're using you could use it with sshfs samba nfs whatever and You know with nfs you deal with locking issues sometimes and other miscellaneous challenges of nfs I mean, it's an old technology. Yeah, hasn't really aged all that well, but it works fine both auto fs the It's not mounted until something tries to access it So in the case of plex I have the movies on true nas and they're shared via nfs the Vm is like a 16 gigabyte vm. It's teensy So if I was to ls the directory where the video files are supposed to be mounted Or plex goes to check or scan the movie directory Auto fs intervenes and says, oh, yeah, you're trying to check this directory Well, let me go ahead and mount that for you And it's so quick at doing that that plex for example won't even ever know that it was never or ever not mounted in the first place So when you go to reboot it and everything it times out after a few minutes if nothing accesses it It just doesn't eliminate locking issues or other nfs challenges, but it makes it a heck of a lot easier to deal with So already i'm thinking container auto fs nfs A container could be docker or a container in proxmox lexie container for example Those are some of the thoughts that I have off the top of my head with how to roll it out Yeah, and even when you spin things up, um, it's about thinking about where the data part is storage That's the big part It's like the the operating itself in gray log is not a huge product, but the results the Indexes that are being stored. That's the part and when you break down and go through instructions You'll see where it's putting those and you should consider putting that on its own mount that is somewhere with expandable storage This is one of those problems when people say well, man, I need to expand my vm I'm not sure how to do it That's where you've added some complexity If you just take the mount where the data store because I honestly the vm from when I went from version 4 to version 4.1 Gray log did not get substantially bigger but The retention policies are absolutely driving The size of that data and the data sets are separate from the Functional application running so just it's just a real good design to separate those two out And having it you know on something like a chunas system where hey if you ever upgrade your chunas system Just point it right back at the NF a share make sure you copy all the files from the old system to the new system And magic we have more availability for that storage and because it's a database application This is where we can go way off topic if we wanted and dive into storage design on something like chunas and building the most optimized data set for a database which would be in setting lower block sizes usually because of the way the right commits work and Storage optimization by the way if you just search my channel for you know how to set up zfs I actually dove into this topic and I highly recommend Checking out level one techs. Wendell who was on the show a weeks ago He's dove into this topic as well for optimizing zfs When you're dealing with database applications with lots of transactions This is a popular use case for why you don't want to run this a inside your p.m You know at scale the homelab people I don't think you're gonna have a problem But if you are thinking about how to scale this up, it's a good skill to learn To have that run separate it also makes it easier to back up is backing up a virtual machine That's 50 plus gigs is harder than backing up a you know couple gigs of virtual machine and Deciding how you want to back up the data now you could back up the data Of course once you have it on something like true nas or a zfs system with zfs replication. Once again making your life easier One one thing i'm thinking about and i'm i'm i'm going to try hard not to make this a storage discussion because that's a bottomless pit of a rabbit hole, but I mean I think we could probably simplify it if you agree to You need something reasonably fast or if you have multiple solutions and one is faster than the other I would assume you want to choose a faster one So for example my true nas server is all spinning rust And even if I was to implement 10 gig which you know spoiler alert tom and I will be working on that But before I get to that point my local proxmox server has um an m2 ssd is this local storage And it's a really good one too. So I would assume that a local um, you know storage device On the proxmox server directly would probably um outperform Um that you know 10 gig to a true nas unless it's splitting the rights across a bunch of disks So you know keep in mind of you know, whatever you have that's faster Yeah, so In this is just an overall application design. Um, because I've had people ask me about xcp and g2 Why does it only store files as big i'm like if you're storing files bigger than that You should be putting them on a storage server not keep creating more vm's at some point. Um, so The last little piece I want to cover though with gray log is yes It has a dashboard that allows you the pretty stuff and uh, the it'll let you create some Charts it'll create pie charts. It'll create that. I didn't want to die too much into it because this is a podcast So i'm not going to get all the visual on you But trust me check out their website. They have the ability to do that I'm not the biggest chart lover But I will tell you if you ever have to sell something to a management They love charts And so yes, there is the ability to put this together. They have some instructions on it. Um, you can create, you know Uh distribution charts so you can understand things and look at things over time with graphs I think they do have their place. Um, I people get really excited about them But at least that ability is there and because this is all open source You're able to access this so if you wanted to set a few people commenting It's not something I've tried at all But when I did the gray log video a lot of people said They loaded things in addition to gray log On top of gray log so they could access that data with something that produced even cooler. Look in charts I said, oh cool I mean all the data is just stored in databases on an open source system So you're able to pull that data back out and massage it in some other way and make it as pretty as you want It's a part of the beauty of having an open source platform for all of this I agree. I'm not really a graph person either because I don't want to be that kind of I team manager, but um, there is one thing I really do like about graphs and that's seeing trends, which can be helpful Yes If you're looking at your disc and it's not, you know, maybe it's 40 full But every week it's gaining an additional one or two percent You could try to predict at what point you're going to have to take action before it's like alerting that it's 80 90 And then your cpu if you um, you know your server is getting more popular or maybe you're hammering it more You'll see the cpu kind of, you know, slowly over time. It's it's baseline usage is going to get up there Um gradually over time So it just kind of gives you an idea like when you might have to add another core if it's a vm or up the storage So um, that way it's not like oh, I got to deal with this right now because it decided to alert just a few minutes ago um Something I probably should mention speaking of alerts the inversion of an alert is true and you're able to do this What I mean by that is a system Um, just because the system isn't sending logs doesn't mean there's a problem. Matter of fact, that can be the indicator of problem That system just quit sending me data. Oh, I'm sure it's fine Yeah, that's that's a good point. I mean if it's not sending something It's like sometimes you have to watch the watcher and oh, yeah, everything's fine because I'm not getting alerts Yeah, you're also not getting data either Um, and that's not a something surprisingly that a lot of people they don't really um They don't really pay attention to the baseline of what is being used all the time And I I had a client one time reach out like my server is always 50 or 60 utilized for the cpu. I'm like awesome Then that means it's doing a job and it has a purpose. Yeah, what do you mean? It need to get that down. No, it's doing work. Let's let it work Let's let it do its job just like with the the logging. It's going to have a baseline too So um, just if it drops off like just like load average drops to zero on a server That's probably not a good idea logs actually drop to zero. That's even worse Yeah, and one of the problems you can run to especially if you're talking about trend charts if you're watching a hard drive fill up Once a system in especially linux systems once they reach that We're full point Um, sometimes it'll stop services and stop sending out data because they can't write the logs that it's trying to send So that can also be an indicator that like yeah, just stopped logging because it had no room Which then failed the serve the syslog server Which means it quit sending data out because it didn't have anywhere to write locally either and chaos ensues What i'm hearing here is people need to check out your gray log video and also your zabx video So with the zabx we're gonna alert about disk the disk being full So you don't have to discover that on you know just because it stopped working and then of course um You know, I don't think it was that long ago that you did the gray log video, right? No, it was just a couple months ago. The gray log one's pretty recent. I covered gray log four 4.1 is a minor obviously just minor release. So it's not dramatically different in terms of the install base They mostly just added it's kind of cool. They it's a bunch of small enhancements I've actually survived several version upgrades since then they had a couple big updates to the four The feature updates really came in the 4.1, but it's updated perfectly fine But there's not anything Major that's changed that would change that video around eventually I will do a 4.1 video because they did They updated the interface to make it I think a little bit more intuitive to use One of the nice things is it's got a really nice autofill feature. So when you're typing It's not like you have to understand everything. They have a little side List that comes up and it reminds me of you know, some of your Like a like a IDE environment where you start typing it can autocomplete certain fields because it goes Oh, yeah, this field is related to you're asking for IP address You're asking for an event ID and start parsing it and one of the things it does immediately Visually now it gives you really cool bars That's one of the reasons I just clicked over to my dashboard to tell you how many logs I'm ingesting per minute. If I as soon as you just stretch it to all logs It'll tell you how many logs came in per minute for that log or per second Those little breakdowns are all in little sliders that I like that little intuitiveness to start Understanding it so it's instantly even just while you're searching giving you some of that trending information that you're looking for Awesome Well, we will leave links to all this their documentation like I said, which is thorough they have all kinds of Resources and community forums and all kinds of extractors for a wide range of devices not just the ones we mentioned but everything from Cisco equipment to You name the enterprise piece of equipment. There's a log ingestion and extractor tools they Quite quite a bit there and of course all your usual things like Apache servers and stuff like that So whether it be a hardware physical piece of equipment or some of the software servers They have it. They do have an instruction on how to get logs out of windows windows logging is its own animal and Microsoft's event system is just Not good. Even if you have the information, I wish that's if there was an improvement I would be excited about from them would be that but yes, it can ingest some of the less than wonderful Logs where things fail successfully in windows I will say as a completely unrelated aside What a lot of people don't realize is that windows logs are amazing What and the reason why I feel like that is because if you're buying a used computer From someone on facebook marketplace or whatever that Yeah, you just go right into the logs and view the you know The warnings and the the critical logs Under system logs and if there's memory issues the disk is about to fail or something You'll find out and then you'll know they shouldn't buy that computer Yeah, there's that too. There's It's it's a it's a mixed bag for what windows does and doesn't log and yeah, it's it's probably a different topic I don't know if it's one we'll dive into on the homelab show in the future now, but I'm I'm We'll clear that but thank you everyone for joining us And for those of you watching a live show. Yes, this will be published as a podcast for you soon Um, this episode will be probably available within the next 24 hours So if you want to listen to it again, uh, check out all the show nuts. Check out our sponsor linoad and thank you Thank you very much