 Welcome. My name is Mark Lovelace. I am a security researcher at GitLab. I'm very pleased to be presenting here at Adversary Village as a part of the overall DEF CON experience here in 2021. Today the topic is advanced persistent threat, otherwise known as APT. I know a few of you out there are probably hoping to learn a few things that you can use for red teaming and penetration testing. So I'll point out some important differences between those and APT, but I know a lot of you are just interested in the topic. I'll discuss how one goes about tracking APT actors over time. I'll cover some elements of attribution, although as you will see in this presentation that's not necessarily as important to, at least so frontline defenders. I'm going to cover some technical details from an actual attack that I dealt with against my home systems from a very well-known APT group and I'm hoping that all of this information will kind of bring the entire subject into focus. It should be noted that some of the data associated with IP address assignments and domains discussed later in this presentation are no longer current. Also, all specific data I am discussing has been referenced in public by other parties or came directly off of my home systems, so I'm not revealing any state secrets. But first I wanted to give you some relevant information on my background as well as some of the history of the term APT and hopefully all of that will give you a little perspective for the rest of this presentation. A couple of decades ago I formed a hacking group known as NMRC. We did the usual goofy hacker things, but we did have a serious side when occasionally help out to a group that needed some technical support. Now we were doing this for free. Most of these organizations we'd help were involved with human rights. Some were activists who were working against a repressive government and other like-minded activities. We didn't make a lot of this work public for obvious reasons. We didn't want to tip the hands of these governments, nor did we want them to know who we were doing this work for. Alright, let me give you a quick example of something that we did that probably got us on the radar. I was working at BindView, which is later acquired by Symantec. We were developing a web scanner that would look for known security flaws, things like bad CGI scripts and whatnot. We needed to stress test it, so we decided we would try to scan a whole bunch of websites at the same time, and since China had a fairly poor reputation when it came to human rights, we aimed this thing at the entirety of China's allocated IP address space. Now, technically it was only looking for the existence of a file on a website and not code execution. We thought no harm, no foul. But when we did scan certain websites, it would occasionally elicit an interesting response, usually in the form of a DDOS attack. I mean, we weren't trying to really hide where we were scanning from. I was doing this from my house. After we collected scan results, a couple of us had copies of all this data, and we provided it to those that we knew were fighting the good fight, so to speak. Mainly repressed Chinese nationals who were angry at their government. Now, in retrospect, this was pretty poor off-sec on my part. I had static IP addresses. It wouldn't be too hard to figure out that this scan led to this type of an attack, et cetera, et cetera. In particular, when I started doing things like talking in public about how I didn't like what the Chinese government was doing as far as human rights goes. So it wasn't too hard to be able to track me down and get my feelings on the subject. Now, another area that probably impacted this just as greatly was my employer. I used to work at MITRE Corporation, which is basically a U.S. government think tank, where despite my hacker background, I somehow managed to get a security clearance, which has now expired. While I was at MITRE, I worked on a number of projects, including frontline battling of APT attackers against that organization. Now, because of these hacker activities, my employers, a few other issues and scenarios I won't belabor the point with, I was on the radar. I've apparently been added to a list of assets that some of these APT groups might want to compromise. I might have access to something interesting as far as they're concerned, where I may be in contact with someone that they're trying to target. Between direct contact from friends that work in government agencies and a rather interesting exit debriefing when I left MITRE, I realized that while it's not life-threatening, I was on a list of potential targets. I'm not saying I'm a special snowflake by any means. I'm just saying that there are a number of us that have worked in this arena before, and all of a sudden, we've become kind of a target, so to speak. So, in general, my perspective in this is a little more than simple curiosity about the subject. So, APT, let's talk about the history of the actual term itself, APT. In the 90s, the U.S. military used the term Advanced Persistent Threat, or APT for short, to refer to small self-autonomous enemy commando units. In 2006, the term was applied to cyber-based groups where it stuck, probably because of the similar independent nature of these groups. Instead of the usual distract, disrupt, delay mandate of the commando units, the cyber-based units were all about information acquisition. At the time, these cyber-based groups' main goal was intellectual property for competitive or military advantage. Often targeted were things like defense plans, including details of various weapons, troop operations, what projects are being funded, certainly any other form of intelligence. Additionally, things like network diagrams, software inventory information, email addresses of personnel to further existing and future cyber-based operations were sought. Basically, they would try to gather information that revealed military or economic weakness, or gave them a military or economic advantage, and gave them a way back into the place that they were trying to get into. So the ATP battle plan was roughly as follows. There would be a fairly detailed reconnaissance of a target that would include not just what type of systems the target might have, but also identify some of the people and what their jobs were to help with things like spearfishing. They'd develop an infrastructure for an attack, such as email accounts to send phishing emails from or register domains and set up websites or compromise existing ones, code up and get ready to deploy command and control systems, and so on. Based upon the reconnaissance attack at that point, using a method that should work on the target, once compromised, attempt to move laterally to as many systems as possible, backdoor as much as possible to allow for remote command and control, as well as potential future entry points in case of discovery, and being kicked out, they could get back in. And if this was a return visit to a site, perhaps they might be going after specific assets fairly quickly once they get in there. They would then perform the main objective, which it could be to exfiltrate data or something else, such as maybe backdouring source code, etc. And then of course, they're constantly adjusting things over that entire time as needed to maintain that all important foothold. Okay, for some reason, attribution has been a hot topic, particularly in the past when this entire APT thing began to be known in public circles. The main argument, the main arguments I should say, I heard were the following. First, an attacker is an attacker who cares who it is, just keep them out, right? Well, that's true. We should be doing that in any case, but that's not the entire point, and we'll get to that here in a little bit. Number two, consulting companies were using the term APT and they're marketing to sell goods and services, and so a lot of people just deemed this to be overblown marketing, just some type of weird marketing ploy. True, that was happening, but no, that really wasn't necessarily the problem. The third argument was that the term APT, Advanced Persistent Threat, advanced being the keyword here, was rather offensive to a number of people. The main argument was that someone read a report from, let's say, FireEye and see that an attack group had used a year-old bug to gain access to a target, and some people would say things like, oh, that's not advanced. If it were me, I'd be using zero-day, not some lame six-month-old bug. Now, this last one really pissed me off. To me, as a hacker who used to actually break into systems in my youth, I fully understood why you would not want to waste or essentially burn a zero-day on a target unless you really, really wanted into that target. A number of people saying it wasn't advanced, they did red teaming or pen testing as a part of their job, and they had a different perspective, completely different perspective. True attacking, truly committing felonies involves cheating, for lack of a better term. When you're pen testing, you're usually looking for multiple ways in and documenting all of them, and there are some assets that might be off limits or out of scope, and some techniques might be completely off the table, such as denial of service. Red teaming often involves testing subsections of the infrastructure. It might start with the idea that, assuming we got root on XYZ or some other scenario, just trying to test detection systems themselves, lots of different things that go into it. In other words, there are limitations that are imposed on those groups. Now, with APT, there are no limits. You can launch a DDOS attack against one part of the infrastructure or unleash a piece of nasty malware, just something to keep the admins busy, and then use your crafted attack to slip by defenses. Nothing is out of scope. Nothing is off limits. I'm giving you an example of this. I was aware of a very large software company where dozens of ISPs in the area of its headquarters were targeted, and the attackers were all looking for VPN connections from home systems and this large software company headquarters. Those home systems were targeted, and all of these VPNs for this particular organization allowed for split tunneling, which in turn allowed for the attackers to gain a foothold in this large software company. Now, APT attackers are in general better at this kind of thing than usual run-of-the-mill attackers. I mean, this is their full-time job. They're good at it, and so this gives you kind of a rough idea of what I'm talking about. Attribution. Attribution is uniquely identifying a particular APT group. In classified or political circles, knowing that nation-state behind the attack is important, otherwise that part actually doesn't really matter. TTPs, the tactics, techniques, and procedures help with this identification process. Now, by gathering TTPs and other indicators, patterns will begin to emerge. The more you gather, the more defined the patterns are. A set of these highly defined and related patterns is called a campaign, and a campaign is basically an APT group. Now, after gathering data from dozens of attacks, extremely refined patterns emerge, and one can ask if certain patterns are similar over time, which could mean a repeat attacker, and this is the true reason you do attribution. Based upon the historical data, what is their next move? If, for example, a particular group attacks on Tuesdays every four to six weeks with a phishing email from a yahoo.com address, you can attisapate that kind of attack and even stop it. Data from launching that executable from the last attack on your honey net tells you how they move and exploit access. So if they do get in, you know their next move. That's a big, big part of this. Will they crazy Ivan? If you haven't seen the movie, The Hunt for Red October, I highly recommend it. Now, in this film, the enemy Soviet submarine would sometimes stop and turn to look for someone tailing them. Typically, they'd be looking for the Americans, and this was referred to as crazy Ivan by these Americans chasing these people in their submarines. So some APT groups did something fairly similar where they'd swap out backdoors every week or so. Some even left a few of the old backdoors on a couple of computers so that if that particular backdoor was discovered and then all of those machines were wiped, the victim would think they'd cleaned up the attack and were okay, whereas the APT act was actually still in there because they were using multiple backdoors. Now, tools used to collect this data can include honey pots, honey nets, even honey clients. It also can include data from when you're going through and looking at the remnants of a successful attack on their part. This data gathered is stored at MITRE. It was done using a software package called CRITS, which is now an open source project. This started at MITRE who was dealing with this APT data collection and is still out there and you can still download it and use it. Data sharing, if you wanted to be able to share data with others, this can actually help with others in your industry, for example. It can be crucial, particularly if your buddy down the street gets attacked on Thursday by one particular APT group, then you're going to get it on the following Monday. You hear from them that it's Thursday and they got attacked, Monday you know what to expect. So that becomes important. So that kind of gives you a little bit of an idea about attribution. Let's talk about an actual campaign. This attack was from a group known as APT-18 or Dynamite Panda. I knew them by the name WEKB. WEKB is still active to this day. This particular attack took place a few years ago. I don't know if you remember the company hacking team and the breach they suffered in 2015. As a part of that breach, they had an Adobe Flash Zero Day, which was leaked. So it was going to be patched fairly quick. Now when WEKB saw this breach, within days, they had launched a campaign with a fully weaponized evil flash file. While it is possible that they managed to get the evil file working flawlessly without causing a crash and deploying the payload and all this accurately, it is more likely they already had this particular Flash Zero Day and decided now is the time to use it because all of a sudden it had a limited shelf life. They quickly developed a spearfish and I personally received this fish as did other select individuals and various organizations. Here are the raw headers of the phishing email I received. Okay, so the TTPs are in blue and the important data associated with that is in red. This shows just some of the dozens, if not hundreds of things that are collected when doing data gathering. Now here is the body of the email. There are obvious mistakes in the email, probably because WEKB was in a hurry to get this out the door. But the IP address of 137.175.4.132 really caught my eye. I recognized it, immediately thought of WEKB. I did go and look at the page that it was pointing to and downloaded the movie.swf file. The page and the swf were there for maybe three or four days, maybe less. I don't know who took them down, probably WEKB. Movie.swf had a payload that simply downloaded and installed a remote access trojan, also known as a rat. So I manually downloaded it and took a look at it. The rat was known as ghost and there are ghost variants. These were often used by WEKB to use a ghost variant. A bit of analysis on the IP address revealed more information that I remembered as WEKB as did the rat, including the command and control IP address. Now ghost is an HTTP browser style rat in that it does periodic queries and it looks for new responses which helps direct the next command that ghost will run on the target. In this particular case, it's using DNS text records for command and control. So ghost traffic looked like DNS queries and for a typical victim location, this type of traffic would not be blocked and would very easily make it outbound. As you can see, there is the potential for dozens and dozens of different attributes one can use to track individual attack groups. Depending upon the campaign, I have seen TTPs in the hundreds for a single campaign. When you compare these for looking for patterns and let's say that you get 80% that are common between a newly documented attack and one that's happened previously, you can almost be certain it was the same attacker. Especially when you consider that maybe the second place contender for common TTPs might only have say 10%. Anyway, enough about our attack attempt. Now let's talk about some fun facts about APT in general. Alright, some of our fun facts. The initial intrusion is going to be the easiest way in for the attacker. If sending in an email asking the person to run an attachment called backdoor.exe does the job, they'll do that. More sophisticated targets might require some more advanced techniques up to and including zero day. Credentials, tokens, keys of all kinds are going to be harvested. End user devices, servers, containers are all infiltrated and the more sophisticated attackers will rotate their backdoors. These are the APT basics that seem to happen at every intrusion. The main objectives still happen but the basics they do seem to be fairly consistent. Advanced attackers will compromise a target software and their services for downstream attacks. The 2001 attack on RSA, attributed to Wekby by the way, was an early version of this. SolarWinds is probably a good example of a more recent one of a kind of a downstream style attack. Now these supply chain attacks might be cheaper for the attacker as there is a lot more bang for the buck so to speak. Although the risk of detection is higher, the larger the spread of the attack, the more potential services that are touched. This could result in a breach being detected and that was the case for example with SolarWinds. Now as far as the nation state themselves, there are a few big players in this that includes China, Russia, Israel, France and the USA. There is also kind of a next tier group that includes countries like India, Pakistan, Iran and North Korea. Now as far as who is best of the best, that would definitely be, I would put that is USA, Russia and Israel, hands down. They're going to be the top ones. Security people do need to understand these types of threats. They're not common but they exemplify the sharpest attacker that you will ever face. If you sell goods or services to customers that are the type of victims that APT actors are going to go after, you should consider yourself a target. The RSA attack in 2011 is probably the best example I can think of of this. Not all of your customers would potentially face an APT threat but some might so just keep that concept in mind. At the same time, the tracking of TTPs tends to show general adaptiveness. So, secure your barriers, go up in one place, attackers will try others. Give me a quick example of this. Spear fishing worked a lot for APT attackers and then at some point it wasn't working quite as much so they would try to become friends with these people on, say, LinkedIn. Pretty much like, hey, you're my friend. Can you check out my resume and tell me what I need to do? I got a job interview coming up. Tell me what looks good. Sure, let me take a look at your resume. Okay, here you go, victim. Here's a nice zero day in a PDF file that has now owned your system and your organization. So that's like an example of how they might adapt to and take on other attack scenarios. So just remember that the whole information acquisition mandate and all of the associated tactics that are used by APT actors are being adopted by non APT attackers as well. These non APT attackers say such as ransomware attackers, they see what's working and they they'll go ahead and try to use it. So defending against a potential direct APT attack does have some advantages. All right, let's talk about some mitigations. The basic security tech we have such as using two factor authentication, automating our patching on endpoint systems immediately, all that helps. If I were to go further in general terms, I'd say not using Windows and being an all remote cloud based company also helps because that's not as common. And it creates a much harder target than what the APT actors are going to be used to. Using things like Google's GCP, Amazon's AWS, this helps because they've been hit themselves pretty hard in the past and they do a lot of stuff that happens behind the scenes to help secure things. So moving to that cloud environment if you haven't already is definitely a big help just for that alone. Get aggressive. Use your red team and employ outside pentest firms and with extremely broad mandates. Remember, APT attackers cheat and ask these people to cheat on your systems if they're going to give them a fair shake. Dicks everything they find. Don't just check off the box of we did the annual test and then wait until next year and do it again. Test frequently with phishing campaigns. I think this is still the number one way that APT actors manage to gain a foothold is through stealing of credentials in some form or fashion. So make sure that you do phishing campaigns. Finally, do not punish users when you get breached. Reward them when they report things even if it's a false positive. They are not the security experts. We are. So help them want to help you and that will obviously help in so many different ways. Okay, that's the end of the presentation at this point. I want to thank you very much for your attention. There is contact information where you can reach me and just reach out to me if you have any questions and I'll answer them as best I can and thank you very much.