 Hey everyone, CountZero here. Welcome to Blue Team Village at DEF CON. This is Project Obsidian. Today we're going to talk about incident response. Specifically, final reporting made exciting. Really, this is a way to discuss why this dreaded yet critical part of incident handling is so important to the overall process of incident response. So with that, let's talk a little bit about the incident response lifecycle. You've probably seen this if you've got any of the other talks or pre-recorded videos, content for Project Obsidian so far, but just kind of letting you all know where we are in the incident response lifecycle. Final reporting falls into the post-incident activities. So that's where we are now. This is where we're going to wrap up the information that we've gathered throughout the incident and the activities that have been performed as well as making sure that we successfully communicate the needs and any after-action activities that need to occur. So what actually goes into a final report? So these are some of the items that we'll cover in this discussion. These are some of the key components to be sure you include not only for your executive levels or your legal team who needs to understand what occurred and what the business impact may have been, but also to cover some of the technical details as well so that your peer groups and detection engineering with forensics, anything of that nature can be used to improve the security of your organization. So we'll talk through a little bit of each of these items and how they relate to what goes into our final report. So the executive summary, detection analysis, we'll talk a little bit about containment actions, eradication recovery, and then finally lessons learned. So when it comes to the executive summary portion, this is the hook. This is what you use to get the attention and this is often what gets seen most by your executive leadership teams, the folks who are controlling the purse strings. So that's how you get some help if you need help in order to deploy new products or get some staffing or make some changes to the security posture of the environment. So a key thing here is to understand for each one of the sections a final reporting what is the audience, what are the components that go into that, and how do you really successfully communicate those needs. So for the executive summary portion of a final report, your audience is the executive leadership team primarily. So you really want to make sure that you write to that level and help them to understand what happened in straightforward and simple terms so that they can communicate that successfully and make informed decisions. So for the components that go into this, you want to ensure that your audience understands what you're trying to relate and then relate that back to your either business operations or functions depending on what the organization is. That will tie your incident and your investigation into business language or into legal language that is consumable by those folks who are making those decisions at the highest level of the organization. Another key component to the executive summary is a high level timeline. The gory details, you can save those for later in the report to communicate with your peer team so that you can build detections and other actions that may need to happen at that level. But for this initial timeline, you really just want to go over the high points so the leadership team understands what happened and when. This gives them the understanding of how to communicate potentially even with the media depending on what is in play or with external legal teams as well. So this is a critical piece to highlight that time frame and the actions that took place during that time frame to show when something occurred, what actions were taken to mitigate or contain that risk, and then finally what was done to close the entire loop. From there, you'll move into some recommendations. It's good to define these recommendations in both strategic and tactical levels. So at the strategic level, you're really going to communicate with them about large changes or potentially expensive changes depending on the recommendation that comes to mind when the incident is investigated. For the tactical tasks, looking at some simple things that may be able to impact the security posture, maybe it's as simple as if we just changed the event log sizes for our security log, we would have had more data to see what happened during the course of this incident or some simple actions that could take place on the particular things that are being recorded in your event logs. So keep in mind those things and try to define those in a level of priority that is understood by the executive leadership team. And as you can see here, some of the things that you want to put into particularly maybe an appendix or even a separate document is like the full timeline as well as the indicators of compromise or IOCs. So moving into the detection analysis phase, whenever you talk about this in your final report, think about your audience as the technical leaders. So maybe your IT systems administrator leaders or network administration leaders and also your peers in your security teams. So the components that come into this, detections. So what was detected and how did that come about? Was this a user that reported a phishing email? Was this an external report that came about from a vulnerability that's new in the wild? Or was it an alarm from your SIM? So understanding how the detection occurred or if no detection occurred and you got it reported in some other fashion, that all feeds into that final reporting and betterment of your program. So you can take the information and say, we didn't get an alarm for this, but could we have and then you are able to build on that. Finally, the analysis portion. So documenting the actions that are taken. And again, you'll hear us talk about this, especially in the instant response and handling part of the project obsidian task that we've taken on is looking at how important documentation is for all of these different facets of incident response and digital forensics. So in the detection analysis portion of this specifically, you want to document any of the actions that were taken. How did you dig in? Did you look at event logs? Did you look at firewall logs? Did you pull triage from the system and work with the forensics team in order to find out what occurred in user browsing history? So there are a lot of different actions that can be taken during the course of an incident, especially with the investigation portions. So taking that into account and truly documenting that well is a critical piece, and this is where you communicate exactly what happened. Another important thing to list during this as we talked a little bit in the detections portion, opportunities. So what could have been done better? And what would have helped with your detections next time? As far as what could have been done better, that could be process. That could be better documentation on using some of the tools that you have available. Another thing to keep in mind is, you know, with detections. So maybe your team is the operations team and you're responsible for the initial triage of the alert, but you've got a detection engineering team that you can rely on or some security engineering folks that can help you build better detections in your SEM or in some other reporting or alarming tool that you've got. So keep that in mind, document those things. And it's really important to write those down as you're going through the process. So keep a note of that while you're doing your investigation. And that really helps whenever you come back into this final reporting phase. You've got those notes taken. You've been documenting what you've gone through and what your thought process was at the time, so you don't lose that chain. So we'll move on into the containment phase. So we've moved into the containment phase for final reporting. Again, you're still going to be talking mostly to your technical leaders and your peers. So this is the area that you want to specifically discuss. How did you stop the bleeding? So how did you take that action? Did you lock accounts? Did you isolate some hosts or some network segments? Being able to define the exact actions that you took and when is an important part of understanding the containment. So this again feeds directly into those timelines and that critical communication that goes all the way up to the executive leadership team. And as always, talk about any opportunities here. What would have helped in the future to stop the bleeding faster? How could you have contained this more quickly? And whether or not that is a limited pool of people resources or if there are some missing tooling in your environment or missing logging in your environment, these are all of those opportunities that you take to communicate to the folks that hold the purse strings who control the money to help them understand we could have done better if we had X, Y or Z and Y. So putting that into that language, putting that into the business terms, putting that into impact is a critical piece of this and understanding how these containment actions take place is a good way to translate that. And again, this gets your technical leadership on par with you and helps them understand what you need to do your job. And then they can help to push that message up to the executive leadership team. So moving into eradication. So after we've contained the incident, of course, we're going to move into eradication. How do we get the adversary out of the environment? Again, this audience is going to be technical in nature. Documenting your actions continues to remain the critical piece here. So documenting how the adversary was removed from the environment. So did you need to reset passwords? Did you need to rebuild the system? And how were those actions taken? Looking into that a little more deeply for these opportunities in the eradication phase and how you would communicate needs there or opportunities there. Again, do you have a solid process in place to rebuild systems? Do you test your backup so that you know you could go back to a pre-adversary action backup and restore that and everything works fine? These are the critical pieces that you can learn. And you follow through with the closure of this incident when this final reporting phase to understand how those actions need to be adjusted if they do. Or if they worked very well, that's also important to highlight. So highlighting your wins and highlighting your opportunities is a critical piece of this reporting function. And as always, we'll continue to talk about thinking through and thinking through the future. What are your limitations? What are your successes? And communicating that back up the chain constantly so it can highlight what you've done well and what you need help with. As well as helping the leadership communicate that again up to that executive level. So finally, we'll move into the recovery phase. And in recovery, this is really where we're getting back to business. How do we get the organization back up and running? And how did we test that those things are actually functioning? So this is where you can communicate all of the things that went well and the things that were of issue again with this recovery phase. So this is the important part to help the business understand. Here's what we did. Here's how we got you back up and running. And here's how we're now serving our customers or our constituents, you know, whomever it is that this particular organization serves. What you'll see is a lot of these phases. We have similar types of details with the actions taken and the opportunities. And the reason that we suggest breaking them down into the incident response life cycle pieces is really to help you understand at each one of these levels how this is occurring and to make it easy to communicate across the board with the different functions that may occur in the environment. So you may be a small team and everyone does all of these things together in the small security operations team or you may be a very large team and you've got these different components that can take place. So maybe you've got a malware engineering team that can help you dig deep into that or a service or if you've got a forensics team that really focuses on the deep dive. So keeping that in mind and understanding how to communicate through this final reporting process is a critical piece of that. So post-acid ad activities, so this is where we are in our incident response life cycle. So this phase really goes through not only your executives but your technical leaders and your peers as well. So this is where we're talking about final reporting today, looking at all of the above, wrapping that all up into an easy to consume document, potentially even a presentation, and being able to pass that along and develop those strategies and tactical recommendations that may need to occur. With that we'll talk a little bit about some reporting pro tips. So use a template and one of the things that we've got developed is some basic templates for a lot of these different functions. So you may have already seen a template on scoping or note-taking previous prior to this presentation. And the reason that we suggest templates is it ensures consistency and completeness. So why does that matter? So consistency is important to your peer groups and your leadership team. They start to understand here's what to expect. I know where to look for the important parts, that I need to focus on if it's that summary and that timeline for your executive leadership team. They know, hey, here's where I go and I see what happened when and here's a quick timeline so I can easily communicate this. And completeness, this way you don't miss something. So if you've got all of these fields or sections in your final report template that you know need to be completed, you can take a look at that and it gives you that opportunity to think, have I done this? Did I think through this part of this final report and put that into action? Another important piece is just the facts. In incident response, we really must concentrate on understanding what we know and what we don't know and trying to move from that hypothesis of what happened into what we can actually prove that happened. There are going to be times whenever you don't have all of the data available to prove that. So stick with what you can prove and then highlight the things that you can't prove and why. That's where we come in with our opportunities to improve what's happening and what's happening and what processes we use in the environment. As you're going through your investigation, whether you're reversing malware or if you are on the forensics team or if you're on the operations team on the front lines, write out your findings in report format. Make good, complete sentences. Drop your thoughts into those notes for the steps that you're taking during the course of the incident. This comes in very, very handy whenever you get to the final reporting phase because then you're mostly copy-paste and do a little bit of word smithing to make it read better or read more clearly and you're good. It really helps you to solve any of those cases where if you've been working on a particular thing for a while, you've finally figured out the right syntax to find what you were trying to hunt for and you don't take good notes about it. You don't really show your work and then your peer group can't replicate the actions that you took and so then it becomes harder to define that and if you've forgotten about it because you've been working for a long time and have slept since then and then that kind of went away in your brain, so keeping those notes, writing them out in report format so that you can easily communicate those to others will help sink that into your brain as well as translate that data over to something that someone else can use. This can become a critical piece if you're talking about building a new detection so if you have found a way to find the evil and you can communicate that over to the detection engineering team or if you are the detection engineering team and you need to do that later after you finish this incident response action, you can translate that and remember what happened. Finally, normalize your time and map to a framework. So normalizing your time, UTC, UTC, UTC, UTC. It's the only time zone. It's the only time zone. We don't need to worry about any other time zones. The reason this is so critical is whenever you're trying to put together a timeline so everyone understands what's happening, you've got to understand where your data sources are coming from for the investigation and what happened during those periods. Well, if you're trying to chain events together and you've got something that is in Pacific Time and you've got firewall logs that are in UTC time and you've got server logs that are in Eastern Time in the U.S., then how do you map all those things together and if you're trying to normalize that into a SIM, maybe there's challenges there too and you've got issues with communicating what happened when and trying to follow the chain of evidence. So really think about trying to get everything converted into UTC and then it's okay to display in different time formats, but when you're doing the incident analysis and you're doing forensics, try to stick with UTC and try to get everything converted to UTC if you can. Mapping to a framework helps you communicate the bits and pieces a little easier as well. So if you're focused on NIST or if you're focused on Mitre Attack Framework, stick to that, learn that and use that. It'll help with the clarity of communication amongst all of your different teams. So those are some critical bits and some pro tips for reporting. So finally, let's take a look at just a small snippet of a final report and what we'll do is show off the basic template that will be provided as well and help you understand some of the look and feel for that and then hopefully you can take this and use it for your own needs. So now we can see this demo template and so what we've provided is just a basic word document to highlight some of the sections that might be critical to your investigation and putting that information together. This happens to be in a Microsoft Word format, a DocX format, but this also works in things like LibreOffice as well. So if we scroll on down, we'll take a look at a basic incident summary and this is just a little bit of detail for Kill Chain 3, which is mostly being discussed during Saturday here at DEF CON. So we'll look at what we've done with this very broad summary so far. Again, this isn't fully complete. That's on purpose so that you can have fun with the data that's going to be released as well, but it kind of starts to give you an understanding of how to well communicate some of this information for your executive leadership team. So we've got some specifics on a date. We've got some specifics that say there was an alert that was triggered and maybe the security investigation was delayed and why. So this can be used to communicate back to your team to say, hey, we had some trouble getting to this alert because we had some other alerts that were stacked up ahead of time. Do you have some issues with people? Do you have some issues with resourcing or is it related to cleaning up some of your learning? So again, do you need some people to be able to process those alerts more quickly? Again, we throw the details out quickly at the top of what was happened. You know, system was compromised. Here's the time that it happened. We've confirmed data exfiltration. Here's the time that we believe that occurred. And also a few more technical aspects of what happened during the course of the investigation and the course of the incident and then some of the actions that were taken to contain that. So here we see, we've blocked some network access. We've done some other things to kind of stop that bleeding. And then giving them some tidbits on, you know, why did this happen? What are some things that we could have done to better protect this? In this case, we see, well, it seems to be this was a password spray on some Internet exposed systems. And the fact that we didn't have multi-factor authentication rolled out to our end users, there was no second layer of security kind of protecting those accounts. So the adversary was able to get in. And then we're going to kind of wrap up with, you know, what are the actions that need to be taken? And in this instance, we're saying, you know what, we can't trust our Active Directory any longer due to what happened here. So we're going to have to, you know, rebuild this and make sure to throw out some, you know, estimates for, this looks like it's going to take us, you know, a couple of weeks. So we're saying 14 days. And what does that mean to our, you know, users? So maybe Office email is going to be impacted for a while. So, you know, in our instance, fortunately, we've got the company website maybe was hosted off site. So that's not impacted right now because it wasn't connected into our Active Directory. So we can, you know, post some messages up there to let customers know, hey, we're going to be delayed for a little while and getting back to you on some of these actions or some of these, you know, business functions that we need to do. So this just kind of gives a quick highlight of, you know, using this template and putting some things together. Some of the other items that you can see here. So we've got a call out to the timeline. So if you look at the incident tracking, you know, spreadsheet that's going to be thrown out along with some of those other templates that we're providing as part of this project, you can type that and tie that into here. So you can put your, you know, detailed timeline, you can put like the summary timeline and throw these contents into this final report to make that easy to discuss. Another option or another item that we want to cover is the adversary actions. And here, we're mapping to, you know, the MITRE attack framework and we're calling out some of those TTPs. Again, this makes that cross communication amongst the teams and also amongst, you know, peers out in industry a lot easier and a lot simpler. So they understand the actions that were taken and what happened. So that just gives a rough idea of some of the template actions that we've got in play. And as always, you know, use some good headers, use some good incident report contents. So that way your executive teams and your technical leadership teams can click through this. And you can update this as a field and word. Or if you've got, you know, something like Libre Office as well, this comes in as an index. And again, the update process is very similar. So with that, we'll be wrapping up this discussion shortly. So we'll move back into the presentation. So wrapping up with your recommendations, again, talking about strategic versus tactical and making sure that you highlight those quick wins. So strategic in this case, if we're talking about, you know, multi factor authentication, that may not be a simple to execute thing in your environment. There's a lot that goes into that as far as, you know, cost and implementation and, you know, what systems may or may not work with MFA. Looking at anything that is policy change related. Again, that's a bigger picture item that the executive leadership team needs to take into account. You know, there may be legal implications. There may be staffing implications that come into policy changes on how things might occur during the setup of your security program. And finally staffing. Staffing often is something that is tied directly to budget because it has to be paid for somehow. And so that usually ends up falling into that staffing or that strategic portion of your recommendations. Just a quick hint on this, you know, tactical recommendation. So talking about enhanced logging. If you've got a Windows environment, that could be as simple as starting to test deploying Sysmon on systems. Again, it's something that's, you know, free and is potentially low cost to implement and manage, depending on how your system is set up. Or it could be as simple as updating your GPOs for your Windows hosts to enhance the logging that's already built into the operating system. Again, highlighting these quick wins in this communication for the final report is key so that your technical leaders and your executive leaders understand that you're thinking forward about how to better the environment and communicating that well for them to understand and take action. And with that, thank you very much for your time. We truly appreciate it and we hope to see you in the Blue Team Village Discord. And you can jump in the Discord and ask lots and lots of questions regarding project obsidian. And again, the data sets that we've used to build this training content and these templates are going to be made available to you to use at your discretion. And hopefully you'll learn some things and be able to implement and better your security posture. Thank you.