 Okay, I think hello everyone. Let's welcome our next speaker Philip Cran with the talk scale your auditing event. Thank you Hi everyone, um, I do have a couple of slides But since we only have 25 minutes. These are the slides. You've seen them. They're online We will forget the slides. We'll do stuff live. Otherwise. We'll just wasting too much time. So Who is using audit D? Okay, that's very few so just to give you a quick idea of what audit D is it's something that lives in the user space Can capture kernel events basically you can have configuration rules of things that come from user space And that you want to capture it could be like system processors. It can be user actions It can be file events those you can capture and then work with audit D And you have something that looks like this. You have all report It just gives you an overview of what has been happening or what is the state of your system So you can see this has been running just very shortly on my system just a few seconds basically But you can still see logins failed logins Authentications users so this just gives you an overview of what is happening in your system But obviously this has raw events in the background as well. So that would be all search And then if you add dash raw it would just give you a lot of raw events And these are basically the raw events that are happening in your system Every line starts with a type you have lots of different types. So for example a user started something then you have in a Bracket that is a unix timestamp and after the column there is a unique identifier if One event is causing multiple things that you want to capture you can have duplicate timestamps and IDs And then you have a lot of meta information depending on what is happening in your system. So for example here You can see something was ended in success And I opened a session with a specific user in the process ID and we can capture all of these and that is Audit D basically the one thing that you can already see here a bit is that this line looks very different than this and While this is nice to capture a lot of events. It is a pain in the ass to Capture that in a central fashion. You can try to parse it But it's kind of semi nice So just to give you a quick idea Also, why am I talking about that? I work for elastic the company behind elastic search logstash Kibana beats and we have a cloud service as well and We kind of had this need at some point that we wanted to audit security events for our own cloud service And then we looked at what how can we capture security events in general? And the first thing we did is we wanted to capture this file and then basically parse it So that's something No looks like If you've never seen Kibana, this is Kibana. We'll head over to a pre-built dashboard. We have so-called File beat modules. They're basically configured and they know like I want to Let's head over to the file beat modules You can see here that we have various file beat modules that can collect specific log files The one I'm interested in here is we have one for auditing events and basically we know on my operating system Ubuntu 1804 That log file is under that specific path and by default the pattern of that file has a specific layout And then we can just parse that and collect all the data We can do that for engine X Apache my SQL prosgres Kafka lots of others And we can do that for auditing events So what we get out of these auditing events and since I've only been running that like yesterday for a short while You can see here which users we're doing which kinds of actions. We could filter down on those You can see we don't have any top-executed commands But you can see here we were running this and we had a lot of events in a short amount of time you can also see This was probably me doing something yesterday And I had a Belgian IP address probably we could zoom into that to see where I was actually coming from And then you can see the actual commands and what we have captured on those however Parsing all of this was still a pain and we didn't really want to do that So we were thinking like how can we make this easier and basically what we then did is we took What audit these doing and we package that in another beat and we call that the audit beat So it's just a beat to collect security events and this is using the same configuration like maybe you could say arcane configuration for Audit D just to give you a quick idea of what we have here. So like every other beat. It's living in each city Whatever beat name you have so we have audit beat here. We have audit beat YAML So we have the YAML file here You see we have the audit D module that we are running here. These are some configuration parameters that you might know that How failure modes should be handled how many backlog items do you want to capture if you want to rate limit this if This is capturing or generating too many events and then here I've written myself some notes because I always forget how these filter rules actually work They define like how this is working and for example here I'm saying like watch We're watching a file e to see past WD With the actions read write Execute or change any attribute and we tack this event with past WD axis. This is one rule for example or here We have a slightly more complicated definition where I say always on exit Follow the file e to see pan.com When somebody reads the file, but only this users in the group 1001 and tag that K is always tagged that with developers Pam read That specific file. So these are just some configuration files And basically what we've done is we we've kept the same configuration format We've just wrapped it into audit beat and audit beat rather than writing it out to disk And then you try to parse it back Sense it as a structured format directly into elastic search and then you can just visualize that and see what is going on So to show you Let's head over to this one here We have some pre-built dashboards for those as well. Let's head to the audit the overview Maybe the last 24 hours are a bit too much Let's say we just figure on the last or focus on the last 15 minutes You can see we had for example 22 open file events or connections to or that somebody executed the process and you can see over time how they developed Where those were coming from for example, that somebody tried to log in and Further down you can see the actual events and you we capture a lot of metadata Something that might be nice for you to see is for example where you can see we have a host information So we know on what kind of host this is running and what operating system If you're running in a cloud pro on some cloud You can see this is running on AWS and we enrich that with all the information So you know it's just one specific instance that has a problem for example security wise or you could filter down to say like Oh, I know only Ubuntu has a specific vulnerability in one specific version and then I'm only interested in those hosts So you could filter down to those if you're running in Docker or Kubernetes you could enrich that with the same metadata here So then you could filter down to a specific pod or namespace or a specific base image if you saw that know that there is a security issue To look into that So this is what we have running here So let's see if we can actually find something. So I will Quickly head out here Leave this user. Let's say we want to log in with an elastic user Well What happens if we forget the password that would not be good But that would be kind of a nice opportunity to actually figure out if anybody is trying to SSH into my box Since this is a security track, did anybody try to SSH into my box? Otherwise we'll see in a moment So let's see. I always try to throw that in so let's see What has been happening SSH and well, maybe not just the last 15 minutes But let's say the last 24 hours and then we can see how many people try to log in this time We were lucky. It's not even that many login attempts And you could see like public key password. You can see where they are coming from which users So these are me probably admin is not me. We could filter down to the last 15 minutes for example Just to see okay Okay, something didn't work here. You can see okay, this is too small this time The IP is coming from somewhere else. Anybody has any idea where this might keep you coming from and why? Okay, somebody That's surprising somebody from Norway or so is trying to log in The other one is from Vienna. That's where I'm from I'm using my phone because people are doing funny stuff with the Wi-Fi And if you're roaming with your phone, you will always get an IP address from home And then it looks as if with a GUIP look up as if that was coming from home But this is just a coincidence. This this was me trying to log in Then down here you could see the actual fail login attempts and if you zoom out and to make the time frame large enough You would normally have either Russia or China trying to brute force into your instances or Probably you now Root me yeah So let's see where are you coming from you can see five from here ten from here And we seem to be doing better than China and Russia today or India is also trying to catch up But you just get the idea of where stuff is coming from but this is just the off log And it's still educational to see what people are trying to accomplish here. And so Let's actually log in with our user and do something useful So for example, I've shown you the rule when somebody is trying to access ETC past WD. So let's say my user bless you Bless you again We're just running a cat on ETC past WD. We're showing the file because this is something we are trying to capture so Let's see if we actually found that so I'm heading over to the raw events. These are not the ones I want and You see in the last 24 hours We didn't have just 31 events, but this will be we had 600,000 events, but We had we set this nice little tag and with this tag We can filter down and say like I think it's called ETC past WD access You could filter down on that one and instead of 600,000 events. You just have 131 because this is run very frequently So this is still not helping you too much You can still see which user is calling it and what command and was it successful or not successful But this is still a bit wide So the first rule that we said was maybe not super helpful The other one that we set for the Pam conf Was a bit more limited because it was just bound to that specific user in just a read operation So let's see if we switch down that filter to that one if we find something more meaningful and we have Developers Pam read so only the developer user is trying to read that file If you drill into that you can see this was just me now if we open that one you can actually see We were running being capped on that file you can see Which users which operating system you can see the exact process information user IDs, etc Obviously this makes sense if you have like some sensitive files or Something that you want to keep track of very well that you configure your rules accordingly for that like it's a See Pam conf is not making much sense here, but it's just to show you the general approach of what you could do Something else you could do we for example is if I if I run to restart engine X like we have We have engine X running here. So I say service engine X restart I picked my elastic admin user because I know the password for that one We would be capturing those as well. So for example, you would have multiple places where you could find those But for example, we have a dashboard for process executions in audit Beat we have the process executions Here you can see for example, this is the stuff that is being run Let's just filter down on the elastic user whatever that user is up to Then down here in that list you can see here. This was actually Restarting engine X this was one of the actions that the user has been up to and that way you could just fit like if somebody broke Into your system with a specific user you could just see what have they been running and what have they been up to to follow along Whatever stuff they tried out You could also do some other fun stuff. So let's log out from that user and let's say we are an admin user and the admin user is Kind of curious and the admin user tries to look in What do we have in our home directory and then he sees like oh we have an elastic user Well, let's see what the elastic user is up to And the elastic user has a file secret txt and obviously our admin cannot resist looking at the file So let's take a look That's not where we want to go elastic user Secret will this work Yes, no, maybe no we need pseudo We can run this with pseudo if I type correctly To secure and you can see my secret So this was collected you can by the way we have set up a rule to collect those events So we said like if somebody was route permissions looks into the home directory of a non-route user we want to lock that event and That is something we have in the raw events as well. So if I Add my filter rule here. I think I called it privileged or elevated print. No, sorry not elevated briefs It is called Elevated briefs. Oh, yeah, sorry power abuse. Yes, this is the one I'm looking for But elevated briefs would capture this as well since we ran it with pseudo Elevated briefs is just in my example anything that you run with pseudo. We are capturing here as well So we might probably find the elevated briefs here if we open that one So you can see we have a syscall and you can still see this But this was only found now because it was run with pseudo, but we're still capturing any pseudo event But yes power abuse is the proper one If I remembered my own tags And with the power abuse If we scroll down to those you see okay, this was tagged with power abuse And you can see this was as well the command that we have been running So you find out what somebody is up to something else that you might be doing is that let's say Anybody wants to talk to me, but I think the Wi-Fi here is filtering that out but generally If I connect by telnet to my instance on that port and I write hello It will arrive here and anybody who wants to can just chat But I think the Wi-Fi might be filtering this out. So you might need to reuse your phone How would we find out that somebody has opened a port and is doing stuff now? For that we have slightly extended our configuration format because I said initially the audit deformer There's a bit crude and in the very latest release of the stack that came out this week We added another way to configure things like that which might be slightly nicer To look at Here if I scroll down a bit No, not so far This is the one I want here for example We have a so-called system module that is looking for Processes and sockets every second and is capturing those and then you could just say which Processes have been run by a user or which sockets have been opened which also might be interesting if somebody opens weird sockets on your system And to show you what that might look like is Let's throw all of this away and let's assume Somebody opened something on port 1,025. I'm just randomly searching for that. You can see We found about a bunch of events and you can see okay Somebody opened an outbound socket. You could just open that event and then see okay This is the destination IP the process has now exited, but this was The specific thing that happened on your instance. You could also see which source port was hit You could see that we would also have a process so somewhere you would find The netcat command that we have been running Depending on which event we're looking at right now I'm just capturing this in various ways to show you the multiple ways that you could see that for example here You can see okay netcat was stopped But we opened that port and this was the command that we had been running and this was also what opened the port So you could just figure out like why are suddenly new ports listening and which user was that what has been compromised or where is stuff happening and breaking One final thing that we have at the bottom here is we can check the file integrity in the file integrity Basically, we can just monitor a folder or a file Hash it and then every time the file changes we will compare the hash and see let somebody change the file Which can also be interesting if suddenly your website is starting to serve some weird content You might want to figure out when was it changed and maybe which user changed it or what were the actual actions They did on the file system. So I'm watching my Web directory and to actually see something we'll need to change that. So let's say no Valid oh Barby HTML index HTML Let's say Nope, this is the wrong line Andu undo undo Let's say we're adding something here. So what this looks like is Right now It's just a welcome page. I'm not super fancy, but let's say somebody wants to add any more emoji This should work Prep and now when you reload your page Did it? Okay, doesn't look that professional anymore, but we do have another dashboard to actually figure out what happened here So let's say audit beat Did beat and we have file integrity module Let's say we just wanted to change this in the last 15 minutes and then you can see here we Moved files and we created files. This is because we eyes are having this weird Replacement pattern if you use nano or ed or something like that You would just see a changed file and if you scroll down you can see like which file is changed And you can see actually the actual events where the file was Created and replaced etc. So you can see all the changes on the file system for that as well And I think that's pretty much it you did very well on the slides If you want to try that out yourself, I'm not giving you the root user, but you can use there the other user It's the elastic user in the password is secret and you cannot create or change anything on the dashboards But you can look at all the existing dashboards and just filter down on them if you want to play around with that I let that run for another day or so then I'll just throw it away Please don't start sending spam or anything else just Do some proper research if you want to get all the configurations I've have those on GitHub. We can just look at the configurations and set it up It's just some terraform and Ansible to set up my instance automatically and I will just destroy it tomorrow again with that Yeah, there are some similar solutions Look at those if you're interested or look at audit beat. That's what we are trying to do and put in our stack and with that We have two minutes for questions any questions Okay, why the one microphone is on your way on the way to you is anybody using audit beat already just off look your recipe Okay, not that many not yet. Hi. Thanks you For the slides and great presentation But are the next steps so we have collected a lot of audit events and other automations or some further stuff that you Reach automatically. So it's it's a kind of manual stuff until So we have to get get in the Kibana and let's try to find some events Yes, um, so there are There are two things. Um, however, they are how my salary is being paid So those are commercial features or you get them on our cloud service included We have something called alerting where you just specify rules and it will send you slack sms page at youtube Whatever the other thing that is very helpful in that regard probably is we have something called machine learning But machine learning is a super overloaded term. It's a normally detection of time series So it basically learns what is normal that could be like Logins or failed logins per user or started processes or which ports are open And it just learns that over time it knows for example during the week You have like a regular curve and on the weekend It's much lower And then if you have an outlier, for example, if a Saturday looks like a weekday It would just alert you because it knows what the pattern is that it is expecting So for example, if you have normally very few login attempts over the weekend because you're kind of cooperation And nobody's working on the weekend It could just alert you and tell you hey somebody is trying to do as many logins as during the week Which might otherwise with strict alerting rules be very hard to find So we have these commercial add-ons But admittedly they're commercial because that is how we finance the rest of the stack But to get the events That is all Free and you can just go crazy with that All the stuff I've shown you is a patchy tool licensed Thank you Cool, I think we have one more minute Anybody else wants to I was wondering so you have very nice Rules you you can give sorry. Yeah Now if you end up with a quite an extensive rule set How heavy would that be for your system? I mean it would not be the first time that all the thing or mounting would like take down the system and obviously Obviously if you have too many rules it could take down your system