 All right, these are the hardcore people right you guys are here at five o'clock to come here this this is really cool Is it six? No, it's 620 This is what happens So anyways, we have okay. I'm gonna I'm gonna butcher his name, but I Have to look it up again I'm still gonna do it wrong Rick Van June That's wrong. I know he's told me that's the American version of it. So he thinks He and Leandro Velasco are gonna talk about and see I got that one, right? Now come on guys that was easier, right? You're gonna talk about endpoint monitoring with free and open source software. This is gonna be a great talk I'm looking forward to this So hi guys Thanks so welcome to our talk on endpoint monitoring using free and open source software as Discussed before my name is Rick van Duyne Which is way different, but for American purposes. I'm Van June now I work as a pen tester. I'm that dude and the security researcher. We work at a company called therebytes And we do everything from pen testing up until policy Monitoring viral management just IT security in the broadest sense of the word I give this presentation together with my colleague Leandro Velasco, which is very easily pronounceable and for English speakers He's a according to our HR department. He's a threat Intel analyst and a security researcher and does not have a profile picture So why do we need endpoint monitoring? I would like to give a short introduction on the on the use of it And some reasons why you need to be thinking about your endpoint monitoring before somebody else does So antivirus is currently being bypassed left and right is getting increasingly more easy to infect system without antivirus being annoying to us There's lots of EDR mumbo jumbo passing around so service managers product managers and sales dudes are looking at Gardner So you need to think about it before they do a Network monitoring is getting increasingly more difficult. It's great. Let's encrypt is very popular Meaning that most of the traffic nowadays encrypted that also means that if you're just doing Network monitoring you'll see less and less and less or you need to rely on a lot of threat Intel, which is Easily bypassable as discussed before in the writer talk. He's seen that the pyramid of pain Like tracking just domains and IPs might be not so effective so We also think you're currently boring a lot of analysts of death with contextless alerts So there's lots of blinky boxes doing saying there's an alert There's something going wrong and there's no real context there Sometimes they put like one packet and so this is the packet the alert fit the fire down and that's it So that's not that much to go on. It's really hard to then determine. Okay. Is this something and what's the context? So I think that's the motivation for us to be doing this And we were thinking we're sitting in a room and we were thinking, okay, is there a product there? Do we build something? the thing is like we're Engineers so we like to build things so we thought okay, we're building something where we need to have some data We need to figure out something and then Leandro started complaining because we needed requirements so then requirements we Yeah, I just said we need this but that doesn't count according to to the academic approach Leo has So we had some requirements. So one of the first things was okay. We need a way to gather system activity data so we needed a way to Gather data from the system and see something some of the system activity user activity commands being run So we looked at PowerShell script blog logging which allows you to if your version of PowerShell is new enough Allows you to actually log all PowerShell commands that are being executed Which is really great because you will see lots and lots. It's also has a downside. You will see lots and lots And next to the whole PowerShell thing we decided to go with sysmon, which will allow you to log things like registry key creation WMI event filters being created DLLs being loaded you can actually View almost anything happening on a system, which is pretty amazing. However In order to do that you actually want to have a good config in order to know what your log is Because you could literally log anything But that would like if you're using like a traditional scene that you would probably kill it with a couple of hosts Next to that not all data is as relevant as the next so there's two major projects currently in in development there's the Swift on security config, which is pretty great to start with it's based on like a lot of events or new Research or threats being disclosed and then they update the config to log the new things and It logs things in a very broad sense. So you get some more data, which is really good because you can see a lot for those of you who are stuck with a solution that Yeah, you need to pay per event or you need to pay per megabyte stored You might want to look at the project. Olaf Hartung is doing. He's a Dutch dude And he's been working on a modular sysmon config that allows you to Only log what you already think is like evil So it it scales down the the amount of events you have it will also maybe limit you in looking back in the in the past Obviously, but for those of us that have to pay per megabyte parsed in our system It's a very interesting project, but as I said before we chose to go with Swift. Thank you Thank you, Swift for this great configuration We modified it a little bit We're logging some other things. We've added some things and we'll be submitting a pull request to the project And hope to improve it. That's one of our goals So as I've said before we have we gather system data, but gathering data is not enough. So we need to Have the ability to actually collect that and parse and store that centrally One of the things many people think about when talking about open source would be elastic So we're just using elastic in this case We're using winlog beat to gather all the data We're using lost as to ingest it put it into elastic kibana to look at it We had some great feedback by our friend Eric. Eric's not here right now, but thank you Eric To use the winners event for order We haven't actually played with that, but for those of you working in a corporate environment It might be nice. So using the winners event for order will allow you to use a GPO to push all your data to one central system Install winlog beat there and it will help you ingest it into your elastic stack Which is one less agent to run on all of your hosts So that might be nice for our project just installing winlog beat everywhere wasn't really an issue So I will now give the mic to my friend Okay. Thank you very much So up until now we had like a nice pipeline We have we are gathering information from the systems using sysmon and skip log in we are shipping the data to a central location In that location we we are parsing We are like reaching the data and we can display it using kibana But so we can do threat hunting because we have nice context We we have the means to query we have the means to visualize using nice dashboards But we don't want to do that manually all the time We want to provide some means to automate a certain scenarios. Let's say like after an early spine a nice scenario like a Thread or the particular malware. So, okay, this is nice This is a query to find it, but I don't want to do it every time the same query I want to have like a way to have it at an alarm Yeah, that's it. So there are ways to implement that and Elastic provide one, but we chose a different one. We chose to use elast alert because it's free It's really powerful and it's quite flexible and simple Basically allow it allow us to do like alerts based on pattern matching So when certain query is found or triggers or give us a match in elastic search It will create an alarm, but also it allow us to do more advanced Alarms based on frequency for example or when distant events happen in the same Host or using the same domain or using the same user for example So it's allow us to do pretty much anything. Well that we need and as I mentioned before Elastic and provides a mean to do that. It's called what it was called watcher now It's called Xbox, but you need to pay for that. So this is a nice alternative for that system Then we Have way to trigger alarms But we want to write the rules in a way that we can share with the rest of the community because it's great But if you want to every time we are building nice rules We identify a lot of use cases or different interesting scenarios and we want to share with our colleagues or whatever We don't want to transform us into another Like file or we don't want to reprocess our data or do it from scratch So we need to find a way that is standard and shareable. So for that we use in Sigma Sigma is a really cool project made by Florian Roth. Thank you, man Basically, it's you write files in YAML and it's really easy. It's human readable It has a simple schema that you need to follow And it allows us to transform from that schema into different systems such as ArcSight and I would show you a picture later Also, one of the biggest benefit of this Repository is that a lot of people already contribute a lot of rules based on the mitre attack That you may have here. It's a really nice framework It provides a lot of insight into techniques and procedures that attackers are using so if you build a rule based on that You're you're going for like a not just a simple detection You're going for a detection of a technique is something more advanced and a lot of people ready Build some rules and those are published in the Sigma repository So as I was mentioning before we have Sigma from Sigma we can compile the rules into different platform We are using elastic elastic alert, but you can do that to Kibana cura ArcSight Splank and there are a few more Okay, so now we have alarms. We have the rules We have a way to share the rules with the rest of the community and it's a way to transform those rules into whatever you're using But we want to test that how can we test out? How can we test that after we modify a rule? We don't break it and it still work So we need to have a way to do automatic unit test So for our using red team automation in this case We're using RTA from the endgame company. It's a really nice one Fred Kennedy also provides a framework such as this one But we chose rating automation because it's a bit more simplistic. It's based on Python. So you have like Small scripts that they are trying to emulate certain attacks and you can build them as complex as you want or as simple as you want So for our framework, it works really well and Another reason is that rating automation is so hard right now, right? Okay, here we have the complete system as we were talking about the different component But before moving forward, I want to give you like a refresh because we have been talking a lot So her written automation with rating automation. We emulate and we create event in the system We see someone we parse this event or we actually monitor we generate events that will be gathered by will of it Sheep to love touch love touch will pass them and reach them and storing elastic elastic search Then we have the sigma rules here that will be compiled into elast alert rules And elast alert will be querying constantly elastic search looking for certain scenarios to trigger, right? Those alarm will be displayed by Kibana and Kibana also allow us to do thread hunting basically manually But enough of talk. Let's do some demos because we don't want to bore you So the way the we yeah, we plan the the demo is we have three different case studies We we're going to analyze different threads The first one is a lot of stature basically this a lot of stature is like a Station for a motel that is based on word this word will is a has a macro that will spawn CMD CMD will spawn a power shell this power shell will download the torsion from internet and execute it directly from memory so The idea is to first infect us then analyze the Sample or analyze the thread in our system and show that the alarm works so for the first one we have Okay, let's hope it works Okay, so because I already accept the editing of macro. You don't see the banner here But trust me it did work Yeah Okay, it's not working. Hey, you're not nice working. Sorry. Just kidding. Just kidding. All right All right so basically because of Seismon we can see the entire process creation or process tree. So on this side for you will be like The left-hand side we have the parent and the right-hand side. We have the The child so here we can see how We were executes a CMD also I want you to to look at this CMD and See what's going on here? You can see there is a lot of obfuscation, right? so What happened here is that somebody used invoke those obfuscation from Daniel O'Hannon. This is a really cool Okay, cool for the red team perspective, right? And it's a script that allow you to you give like a Remote creation or like you like a command line and obfuscated like this and We talked with Daniel and he confirmed that this is 100% invoke those of occasion It's quite cool. Is this is the first time we this is seen in the wild because it's not that far since he published this his research But how can we detect this right because it's hard So for this because it's the first case study. We're gonna use a simple Approach and that is why we were is actually calling CMD. I mean, that's not normal. I mean why he's doing that, right? so let me show you How we create the sigma rule so you get into Our mindset so this is sigma. It's a Yama file. You can see it's easy. It's him a readable Here has some documentation The most important thing here is we have we need to give some fields in this case is event ID one This means process creation. So we are instructing and the next ones. I like look at this event ID Then we provide the parent images in this case We are given like the office suite because it's not only what that we are interested But we are also interested in like Excel PowerPoint, whatever because none of those should spawn any of these Here we have some suspicious binaries. I mean why we were should be calling a C or 32, right? Thank you much Okay, so after we compile this and there's not much fun to show you the compile version because it's hard to read We load that into a last alert and we hope that it works and yeah, it did work so what is interesting here is that It didn't trigger and when this event happened But also we have another rule that triggers when CMD calls PowerShell. That's not a really strong Indication but together with the wind with the wind were calling CMD and CMD calling PowerShell That becomes something more interesting. So that you hear like if an analyst see these two like something fishing is going on, right? Okay, let me move you to the next case study Okay, now we are gonna talk about unicorn stature this unicorn stature is a tool to generate stature's based on PowerShell and The interesting thing of this one is that it takes different type of payload such as powers for a crowd strike Empire Metapeter and embed that into a series of PowerShell commands heavily associated in coding base 64 and the payload never touched the disk it goes directly into memory and Unicorn start that directly from it. So What is important is that we can expect that in a certain moment PowerShell will be executed virtual analog and then copy because that's the way to place things in memory, right? Okay, let me get infected So we can analyze So something interesting about the unicorn is that it provides or it allows different delivery methodologies So when it's HTA, but you can also embed this into macro DDE or different ones. It's a ongoing work It's really amazing So here you can see that it is working some PowerShell happening and that's it Okay, so again here we have our system and we can see the process creation So here we can see mhda calling Okay calling PowerShell and again, we see a lot of obfuscation and encoded commands So from this alone is hard to determine what's going on We might have some indication that something fishes or not because it's not normal to have like so many PowerShells Calling each other, but also please take a look at this This is strange basically what's happening here is that PowerShell is trained to avoid calling PowerShell using the argument encoded because a lot of tools are looking for for that in particular So the way it's doing it is by setting value. That's why we have SV and get value Somewhere here here and it's constructing own memory the PowerShell encoded command So this is one way to detect such a thread will be like hey Look at this. This is quite unique. I mean it's not that common to see this in a PowerShell execution But the problem with that approach is that it's quite simplistic to bypass You just need to add a comma or another caret or whatever and your rule doesn't work anymore So this rule is simple. It works really well for that specific version, but it's easy to bypass So we need to think of something better. How can we I mean what is the underlying? Tactical what what is the the attacker trying to do really so as I mentioned before it's trying to place some Content into memory and execute it directly from there The problem is that Sismon doesn't allow us to to see this information likely as Rick mentioned before We have a PowerShell script lock login and we can see way more with that so This is what we can see. This is basically what PowerShell is trying to do one of those encoded command end up executing this One of the interesting thing here is that is loading manually the kernel 32 DLL and it's calling functions directly From this DLL such as virtual lock create thread and then set So if we create a rule that is looking for these keywords, we have a detection, right? What is important to to make it clear for you is that? PowerShell is calling that as a command. It's not PowerShell as a process calling this function It's it literally the commands in the common line So let me show you the rule here So in this case instead of using event one that's for Sismon. We're using 4104 that is for Remote execution and we have a bunch of functions of keywords. We extract this keyword from Mitre, different Empire, different Suites that do like a post-exploitation such as PowerSploit and the idea that they all share this idea of Injecting things in memory and executing So when we see that a PowerShell Ancient is there is a PowerShell script that is secure or calling these functions directly We might have a trigger in this case We create this we compile it and here you can see that it is working So we call it the memory section command let's again We have a unique on a specific version and if we take a look at these two like this is really fishing Okay, let me show you the last case study This is a bit I so quite interesting and we want to have like a spin-off we modify part of this demo today Thanks to Matt Grego Yeah so the idea that we were analyzing a minor it's called goes minor and the interesting thing of this goes minor is that it achieves persistency and Using WMI objects. So basically what it does it creates a WMI Filter that will be looking for a particular event in the system and When that event happened it will trigger at the way my consumer this consumer has pre-programmed an action such as Calls in the old star PowerShell or even as you did this JavaScript And when it calls a different command it will do it via this process So also if you execute that remote and command using the way my see this process will be in charge of Executing such a command. So the other we will monitor this and we will monitor this event. So For that we create a batch file So what is what's happening at the moment is first we are setting we are creating these objects We create them the filter that will be looking for the task manager That's why we triggered this event because we want to show you that It's working and the consumer is instructed to call notepad and as you can see it's actually working here because we started task manager and Then How can we detect that right so we have a bunch of Dashboard here. Yeah, so what we what we did first was to modify This is this one configuration file because the one provided by Swiss on security doesn't allow us to do WMI monitoring so we modify that and now we can see the creation of objects So first one we see this one. This is the most interesting one because you're like, okay Why this consumer is trying to call CMD and then call PowerShell with an encode command Here we can see the filter that I was talking about so basically is waiting for the task manager creation So this is one approach, but in alone. It's not that strong because it could be Something that is happening normally in the system. This is a little bit strange But that's a different approach Also, we want to monitor this process because this process will be calling something suspicious in this case This process is calling CMD that will call PowerShell that will call notepad So what we did was to create those two alarms and let me show you one of those So here we are monitoring what this process is calling and when this process is calling one of these binaries we trigger an alarm In this case we monitor when In event 20 that is the creation of WMI objects has as a target So this destination of this is the consumer calling something one of these So as you can expect it works because we test this several times so Here will be our persistent and we am I calling suspicious official program Okay, so I will give the voice back to Rick So as shown before we we have the technology to actually do this However, just having a system that does this for you does not mean that you don't have to do anything I think the most important part of having the ability to see all this is having a process and a team behind it That's actually capable of translating Threats to actual work. There's like the WMI event consumer. That's a trick described the miter It's fun, but you need to just translate that to an actual detection rule so you can do something with it So we had a couple of discussions internally and we had many many different flow charts And then we decided to not have a big one here To not try and not to know you so how would this actually work in in in the yeah, if you implement this I would say you you wait for some infosec news an incident happening at your company a new Information being released a report on some APT group and the first thing you do is you try to understand What's actually happening the moment you do you can start searching back in your own network Have I ever seen this before because that might also be interesting to know and if you don't you can go and look for a way On how to emulate it so make your own way of emulating the threat or maybe get a sample executed See what's happening so that could help Create at least a hunt and if you get a really great hunt you can let that evolve And eventually create like a nice detection rule And you can and in the end have something implemented which will allow you to automatically detect a specific threat So it's not like something you put in immediately. It's something that has to has to evolve So for those of you who don't like flow charts, we made another one, but don't with pictures So the thing how it will actually work So let's say you found something you found your your hunts perfect and you made a Nice sigma. I hope sigma rule because we can all benefit from that The first thing you do is you generate rules for your platform, which in our case is like a last alert because we splunk or I Don't know arc side or something that will put any that will be keep searching on our elastic stack you can then Automate or do a unit test of all your rules using retina automation and any Well, that will obviously generate some logging the log will put we put into the elastic stack a last alert will hopefully Find whatever you're trying to do And put that into the alarms index. So you will be able to review that alarm So how would we actually how would this actually work? So let's say you wake up one morning and Matt Graber thinks of another thing And you have to handle it. So this was while we were doing this research This was a tweet by Matt Graber describing an application whitelisting bypass That would allow you to execute commands Using some signed windows binary. So the great thing of this this research was actually that only 20% of the blog focused on Exploitation and about 80% focused on defense so that already does a lot of work for you So that's lucky. It's also lucky for the demo So it describes a way to do to actually but execute and bypass the application whitelisting and it also came with a nice Pock so a nice proof concept in this case. I think it executed something like cock or at least it does in our demo So what we did is we actually created a retina automation script for this which will allow us to Well execute squibbley foo because that's the name Which what it does drops a XSL file and then tries to execute that using WinNRM.vbs The thing is there's like four five four ways to call it and there's two different Files you can actually use to exploit this. So there's now eight calcs being popped up for those of you Are very interested. I can show you later how eight calcs look Sorry So first thing there's two files There's the WSM TXT and WSM PTY on the XSL Which need to be created in order to to have the file. It's a XML ish format which has Which you can embed some JavaScript in our VBScript which will allow you to execute however that on his own wouldn't be enough so You need to actually See something because just alerting on the existence of like a XSL file would be a bit silly So the thing is there's actually pretty particular way you need to call this So the WinNRM.vbs needs to be in the command line because we can't modify that file and and as you can see You can see C script.exe slash winword.exe So the WinNRM script verifies if C script.exe is somewhere in its Command line, but it doesn't have to be the binary that calls it So if you would just look at the binary name You would actually get bypassed and the same goes for all these formats. There's like four different ways to call it So the RTA file helped us actually figure that out So what would be nice is have the ability to actually correlate these two events so we have two loose alarms and Well, we could use a human to do that, but then you get very bored analysts and that's not so nice So what we did is we created something for this because elastic search does not allow for correlation Elastic alert also does not allow for correlation. So we invented something called ghetto correlation Which is correlation for people who don't have fancy systems What we actually did is we use we look at our own alerts index And we use the frequency Function within a last alert to see okay, can we get two squiggly foo related? Alarms on the same system within x period of time and then you could be it would be able to actually correlate Which is great because as mentioned earlier by Leandro There are some events that are suspicious, but not enough to actually alert on But you could actually add like a suspicious tag and have like okay There's multiple suspicious events on the same system within an hour or one minute doesn't really matter so you could actually like gain something from from all those like Quirky things that are not necessarily bad, but you might want to do something about So ghetto correlation might be for you well, and the funny thing was this morning we woke up and Matt Graber and I think Lee Holmes Published their slides on subverting sysmon which actually kind of defeats the purpose of the whole thing So we were looking at that and it's actually pretty cool because they were actually pretty good at subverting it So we didn't make a full demo of this and it might fail But the thing is it turns out that sysmon. Oh, sorry dude, so this was this tweet I woke up to this thing. It's not very nice And then We looked at the slides there are multiple ways of subverting sysmon There's also a lot of ways to subvert like PowerShell script log logging But the fun thing is you can actually look for this version of attempts, which also are Irregular users not gonna try and subvert sysmon or PowerShell script log logging. That's weird So what we did is we looked at it So what happens is the the way sysmon logs the event consumers or any event filters and all the things Is actually by looking at a very one specific namespace So what the guys did was actually create a script that adds another namespace And that hosts the active script consumer. So then The script consumer will still like filter and execute things But sysmon doesn't detect anything anymore. So we looked at that in order to figure out. Oh wait Yeah, there's a github too. So for those of you Want to play with that? Where are you? It's not in there So so as we said, we just built this like an hour before the talk so So we're reusing their Pock And as you can see there's like if task manager opens Wmi file drop.txt gets created. It's just we needed to execute something. We actually stole their Pock So no credit there But the thing is we started looking at that so what is How could we even detect this is there any possibility of the thing is because we checked it doesn't log the creation of the Wmi event consumer or any of those so What what is there? What's there is limited but might be interesting As you can see there is a Windows event ID 63 message event message That says that a new provider Was registered in a new namespace so We looked at that and we this machine has been running for like a couple of months now and If you start looking at that and you do a Windows update you get a hail storm of alarms So it won't work if you update systems But if you get one lonely one and we could do something with frequency there again You might be able to detect the fact that something's happening on that system It at least it's interesting to look at the moment. You see other weird events but The reason we put it in there was not not just to say like our system kind of works on that But more the fact that you will be able to like literally sit in the back of a room Somebody tweets something and within an hour have something Something to look at I'm not saying it's like the most robust solution, but because you're you can move pretty quickly. It's pretty nice So enough positive Let's talk about limitations so Yeah, it's all great. It's amazing. However It's there are some caveats or there are at least some things that we haven't fully Explored and so there's some things that might be interesting to improve on And I think the main thing would be to look at scalability and robustness of the entire system We know the elastic stack and handle lots and lots of data. So that shouldn't be a problem However, for our project, we haven't tried so it might fail Obfuscation is an issue. We showed the demo with demo Ted's invoked justification by Daniel Bohannon is pretty cool Currently the system doesn't really handle that so what you could do is add Ingestion time try to enrich your events by detecting Obfuscation and say like oh, this looks obfuscated. So I have a flag or something or maybe even try to de-obfuscate I know there is a power shell script that for the invoke invoke invoke obfuscation That's meant for power shell. So there's an invoke de obfuscation for that for the dosfuscation However, there isn't one but it might be very interesting to look at that To have something that add ingestion time will fix that for you Looking at all the commands and things are nice. However, what if somebody just drops a binary that executes a command and it's gone That might be tricky and then you might lose sight of what's happening actually So there might still be nice to add ingestion time also enrich your events with threat intelligence information Such as just looking at simple hashes. I mean if it's there, why not do it? Kibana has a Vega framework which allows you to actually create your own visualizations Which would allow you to have fancy things for your analysts like maybe have a process tree or a call stack all those things So you could actually visualize that a bit better and not have to look at like a table with all the things in there I mean fine for me, but some people like pretty pictures There's been a lot invested into sysmon Conflict tampering and power shell scribble loggering evasion or disabling of it. I think that's actually Great. It's very good to see that people are actually investing time into that So it's good to I think that means you're on the right track, right? All attackers and red teams are now looking into ways to subvert that so I think We still need to look at the subversion attempts in order to determine that it's happening And I mean the moment they happened you lose visual visibility, but the fact that somebody is trying to Disable power shell script log logging might be a strong indication of something weird going on anyway And the last thing would be bring your own land. There are People now like dropping their own run DLL 32 but named bloodblahblah.exe Which kind of defeats most of those detections because they're all based on names and that doesn't work anymore So that might be an issue So in conclusion, I think this whole endpoint monitoring with free and open source software really works Wow would have guessed I would say that But it does allow for experimentation without a budget. It's very easy to start and there's no licensing cost It's flexible. It's scalable we think And it will give you and your analysts a pretty fun time trying to figure out all the things that are happening So you can go from an incident to a hunt to an alarm pretty quickly as shown with with a cismonster version thing You can give your analysts more context than the blinky box with its bad sign And it can also Get your analyst very enthusiastic about new red teaming attacks because it's not just a new attack You have to wait for an update so your system really detects something but you would have the ability to actually do something So figure out what it does figure out how it works and then try and stop it And that makes we're happy and skill that list So that's what that's our presentation If any of you will want to do anything with this we uploaded all our box the configs the sigma things to github Just trust the QR code There's no goat see there I promise you You can also read our paper because this is all based on a paper called file a stretch analysis and detection Just review it if you have any questions or any comments like you fucked up here or you did this wrong or I Have done it better. Please let us know we want to learn So does anybody have any questions? Do it in the back So we actually tried and we had like 40 megs a day But it might be worse if you're using like a server or terminal server a domain controller That might be way more Any other questions? That's Olaf Blog I guess is it recent blog series. I don't know might be the one by Olaf He focused a lot on tuning it down So if you have a system that that like you have to pay per event or you just don't have the ability to store too much Look at his config. It's really nice And it like narrows things down and he made like an entire blog series on it on how to do that And how to work with it and he made a nice Splunk app to hunt and stuff It's really good. Any other any other questions? Nope, okay. Thank you guys. Oh