 To welcome to the annual DEF CON convention, this meeting was held at exciting Las Vegas, Nevada from July 9th to the 11th, 1999. This is video tape number 14. Some of you, I'm sure, are here to find out what not to do in order to not get caught. Some of you are here to find out what to do to catch the ones who want to find out what to do to not get caught. You may take from this what you wish. We're going to talk about this evening and then I want to thank you for all coming. I realize the bands have started. We're going to talk about cyber forensic analysis and there are all types of forensic analysis. I make some distinctions. Forensic computer analysis deals with specifically the removal of information from hard drives or floppy drives, floppy disks for that matter. In such a way that the information can be used as evidence in a court. I'm as caught up in hype as the next guy so I had to figure out a way to use the term cyber. I use the term cyber to refer to things that have to do with networks. Cyber forensic analysis then is the forensic analysis of information on a network that we can use then as evidence in a court of law. Then we have one other kind of forensic analysis and that's software forensic analysis which we won't be discussing tonight because it's black magic. And that is the forensic analysis of code to determine who wrote it. Alright, change. Thank you. First of all let's define what we're talking about here. We're talking about crimes against a computer that's crimes where the computer is the victim. And the FBI tells us that they define that as computer crime. They say that everything else is in fact traditional crime and it just happens to be done with a computer. So we refer to that as computer related crime. And that includes crimes where the computer contains the evidence and crimes where the computer is used to commit the crime. Now let me put everything we're going to talk about in a little bit of perspective. Probably 90% of the investigations that I'm involved in are corporate investigations. They haven't got a thing to do with some hacker or cracker or a thing that goes bump in the night breaking into my clients from the internet. Probably 80% to 90% of the investigations I do have to do with inside people who are doing something naughty within the organization. And so there are a number of things that I can do in the course of an investigation that are much more difficult to do if we have to do them over an internet network like the public internet. However, the techniques are not dissimilar. With that in mind, let's simplify things greatly. Let's say that there are only four kinds of attacks. There are denial of service attacks, social engineering attacks, technical attacks, which is where I put the whole gamut of exploits. Whole gamut of attacks that exploit bugs in a system. And then we have sniffing. These are the kinds of attacks that an investigator needs to be prepared to deal with. Yes, sir. Sniffing is passive. But one of the things that an intruder wants to do, probably the most important thing that an intruder can do, is to gather information that will let that intruder enter the system without triggering an alarm. Therefore, the intruder wants to see if he or she can get passwords and accounts. And that becomes part of a total attack and therefore we're concerned with passive sniffing. Furthermore, one of my favorite attacks when I'm doing vulnerability testing for a client is to compromise a machine on the ISP, put a sniffer there and watch. But nobody here has ever done anything like that. So yes, sniffing is passive. But sniffing is an important part of any attack. So, if we're investigators, put ourselves in the position of the investigator. And if we're investigators, we need to deal with the attack as the attacker dishes it out to us. Well, the first thing any good attacker is going to do is gain information about the target. And they do stuff like internet searches, who is, NS, lookup, dig. All sorts of those little things, which everybody does, doesn't break any laws. Isn't there more unethical or legal of fattening? But they do it. And we can't tell as investigators. The only way we're ever going to know that the attacker did these things is if we're able to eventually do a forensic analysis of the attacker's drive, the attacker's disk. And then we may see evidence there. And then again, we may not. The next thing the attacker may try to do is do some preliminary probing. Now, they want to get passwords. They want to get information. But they don't want to trigger any alarms at this point because it's too early in the game to get caught. And so they'll do things like another one of my favorites is a pop password attack. Just run a little attack against the popper, do some password guessing. It doesn't usually log. Maybe a DNS zone transfer just to sort of figure out what's out there mazes me how many ISPs leave their name servers unprotected. Maybe do some simple mail probes, other simple probes, and these probes may or may not log. And on a big system, they're going to log in such a way, probably, that they're going to be very hard to find. So forensically, as investigators, we're going to need some tools that are going to be able to parse these big logs. Anybody ever seen a firewall log that's been running for about three days on a big company? Yeah, you're going to need a tool to parse those logs. And then, of course, some attackers will get frisky and start looking for backdoors and do something like a ping scan. That ought to show up in a log somewhere if it doesn't, your logging's not set up right. And then we're going to try some technical attacks. We may try a little social engineering. All of these things are things that the investigator needs to look for. And let me tell you, folks, there is no such thing, and I've been doing this for a very long time. I have never, never seen a smoking gun in a forensic dump of a disk. Never found the big clue. It's dozens upon dozens of little clues that you put together, and they're buried inside of these big logs. Next thing that's going to happen if the intruder is gridded what he or she does and does in fact penetrate the system is they're going to try to clean up. They're going to try to delete their tools and work files. They're going to try to modify logs. Here are some UNIX examples of logs that they might try to modify. And we're concerned then about what's left on the computer. In a few minutes, I'm going to tell you a little bit about criminal profiling and how we use it to investigate a security incident. Now if you do like I do and you deal in a corporate environment, the people who are in that corporate environment have an expectation of privacy. And if there are no policies to the contrary, you're not going to be able to investigate this incident from the inside. Because we've got a law called the Electronic Communication Privacy Act that says that without a policy the only thing we can look at is source and destination address. We can't look at content. Well, that may make it very difficult to investigate an incident. So we need policies. So those of you who are here from the corporate side, go back and tell your people we need policies. Policies define the actions you can take and they have to be clear and simple to understand. And the employee needs to acknowledge them. Usually in writing. Now there's been a trend lately towards putting little snippets of privacy policies in login banners. And that seems to work. However, here's a caveat for you. And the caveat is that we're dealing in murky waters legally here. There isn't a lot of case law. It's not well understood. It depends upon what part of the country you're in. And the courts may or may not interpret those login banners as sufficient. So from the corporate side, see your lawyers. If you're a hacker on the inside of a company and you're going to get even with your company and break in before you do, see your lawyers. Yes, sir. Yes. The question is, are there not provisions in the law that allow for troubleshooting and for monitoring? You may monitor for the normal course of business. It's called the business exception. And in the normal course of business, you may monitor for maintenance for those types of things. But what you use as evidence is strictly limited based on the laws. Now, if you have a policy, and again, corporate people, if your policy says that the employee has no expectation of privacy and they consent to monitoring, then you're okay. You're okay. Now, here's the Electronic Communications Privacy Act. This is the law that enables you as a corporate investigator to investigate. And for those of you who may put yourselves in a position to be investigated, this law is one that you really need to know about. First of all, it says that the owner of a system can intercept communications between an intruder and the owner's system. That means that if you come in from the outside, the victim can monitor you. The next one deals with inside. You can intercept with prior consent of the user, and that's where the policy that's been acknowledged by the employee comes in. Once the employer acknowledges that policy, that constitutes prior consent. You can always intercept the portion of communications necessary to determine origin and destination, and you can intercept where necessary to protect your rights or property. That's murky. So ask your lawyers. I had somebody when I gave this talk over at Black Hat, I had somebody say, well, that covers everything. And my answer to that is maybe yes, maybe no. It depends on how the court, in the particular area where you're located, interprets it. So ask your lawyer. Okay, now we've got that tool. Let's figure out what it is we want to do when we do an investigation. First of all, we want to ensure that all applicable logs and evidence are preserved. And I can't emphasize this too strongly. That is among the first things you do, and you preserve the evidence, not by walking up to the victim computer and typing down, copy such and such a log to another disk. Don't touch that computer. So, corporate investigators, if you walk up to the crime scene, the victim computer, and the screensaver's on, don't wiggle the mouse. Don't touch that computer. I'm going to tell you what to do with it here in a while. And those of you who may find yourself in some small amount of trouble for minor transgression, watch to see if you happen to be fortunate enough to be there during the seizure. Watch to see how that computer sees. If the cop walks up and starts typing on it, you win. He loses. Because the cardinal rule of forensic analysis is you must not, you being the investigator, must not contribute to the evidence. And a good defense attorney will know you every time if you do. Trust me, I know. This is the voice of experience talking. Okay, the next goal of our investigation is to understand how the intruder entered the system. That's the critical goal. The next one is to obtain the information we need to justify a trap and trace of the phone line if the intruder is using the phone, or to subpoena information from the ISP if they come in over the Internet. Of course, if this is an entirely internal attack, we don't have to worry about those things. Trap and trace is something that you will need to get law enforcement involved in to do. You can't just go put a trap on a phone line. You can call a phone company and ask them to do it, and they'll do it. But you'll never see the information. You'll never see the data unless you have a warrant. Yes, sir. You know, a lot of PBXs now have called for or called ID, and the answer to that is yes. No. Unless, unless you have a policy. Okay, an acknowledged policy, and that's prior consent. Yes, sir. I don't know. It's the same as this? Yeah, I don't know. I've never been involved in an investigation where we were doing that, so I can't answer the question. But if it's the same as this, I'll buy that. That makes sense to me. Okay? All right, the next thing we want to know is why the intruder has chosen the computer. This is our first little bit of criminal profiling, and I'm going to come to that more in a minute. It helps us to decide what kind of a person may be doing this, their skill levels, and it helps us narrow down our suspects. Now we need to have as much evidence of the intrusion as possible, and we're going to talk about collecting and preserving evidence here in a minute as well. We want to obtain information that will help us narrow the list of suspects, document the damage caused by the intruder. This is important because the amount of damage determines a lot of things, among them the threshold as to whether law enforcement will investigate. Another one is what the penalties are if there's a criminal prosecution. And we need to gather enough information to decide if law enforcement should be involved. Our immediate objective in any network-based intrusion is to preserve the evidence. It is extremely fragile. The evidence lives in a variety of places on the Internet. Most of those places, if not all of them, will be outside of our control. Many of those places do not log. If they do log, the logs may turn over very rapidly, and the information will be gone. So remember when you have an Internet-based attack, you need to do your back traces rapidly as you can. And if we're talking about something like harassing email or cyber-stalking or something of that nature, the data that you need to trace back that email header may disappear very rapidly. We contact system administrators on intermediate sites and request log preservation. Now, if you're dealing with private sites, generally the system administrator will give you what you want if he's got it. If you're dealing with ISPs, it's a whole different story. ISPs are terrified of being sued. And so in order to protect the rights of their subscribers and their own pocketbooks, ISPs will not give you the information you need without a subpoena. And with a subpoena, they usually cooperate very well. So what we do is we call them up and we say, look, here's the information. Please preserve the logs that contain what we want. Now, a word of caution. Again, when I gave this over at Black Hat, somebody raised their hands and said, well, you know, we've had a lot of trouble with getting logs preserved, getting response from the ISPs. We send them email and they either don't respond or we get a form letter. Two things, okay? Number one, don't send them email. You will eventually, but you're not going to do that until you've got a human being on the other end instead of a bot. Next thing you do, you always call. And I think there isn't anybody here who doesn't know how to find out who the contact is for an ISP. So you always call. Some of you have found out without particularly wanting to, I'm sure. So you always call. Second of all, you ensure that you give them enough information. You can't call them up and say, I've been hacked. Save the log. Because you'll be able to see over the phone. Yes, back here. They have every right to say that. You can't force it. You can't force it. There's no law that forces an ISP to keep logs. And in fact, there are ISPs that purposefully do not keep logs because they believe it's how they protect the privacy of their subscribers. So if they tell you that, I've tried crying. I put my wife on the phone once. Anything you can do to get them to do it because they don't have to. Yes, sir. How long does the process take with law enforcement, ISPs and so forth? First of all, I'm going to answer your question, but let me digress just a little bit so that we're all talking the same language here. Warrants and subpoenas are different. A subpoena is something that an attorney can draft. It has to be pursued to a filed lawsuit, but an attorney can draft it. A warrant is issued by a judge. Law enforcement generally does warrants. Several cases generally do subpoenas, although law enforcement can do subpoenas also. If we have to do a subpoena, it can go very quickly, and I'm going to show you how to do that here in just a couple of minutes. If we have to do a warrant, law enforcement is involved. It can go on forever. My advice is, if you're a corporate investigator, first of all, make sure you know what you're doing. That's real important because you can really screw up an investigation royally by not doing it right. But if you're a corporate investigator, you know what you're doing. You've got a team that knows what they're doing. You've got consultants that know what they're doing. Conduct the investigation and then spoon feed it to law enforcement. You'll have a better chance of getting it investigated. So it can take a while. It can take a while. The reason, by the way, you may have trouble getting it investigated is because there's a lot more incidents than there is law enforcement to do it. And so they set thresholds. And the other thing is that it isn't really the cops that decide what to investigate in this particular case. It's the prosecutors. If the prosecutors don't believe they can prosecute the case, they won't take it. And I've had that happen many times. Yes. Yeah. And you can do that, but there is a caveat. And this is all a matter of judgment here. The caveat is that once you do that, you've essentially given the case over to law enforcement. Now, some organizations have policies about that and some don't. Myself, the type of work that I do, I prefer, frankly, to get law enforcement involved because usually we're dealing with something, we're not dealing with simple hacks. We're dealing with fraud. We're dealing with some non-computer crime. In other words, a traditional crime where a computer was involved. And those kinds of crimes, especially in a big organization, need to be turned over to law enforcement for investigation. On the other hand, law enforcement does have thresholds and some companies have policies about not bringing law enforcement in because they want to keep the noise level down. So frankly, what the lady says is absolutely right and it may be your only hope of preserving those logs. On the other hand, the ISP may say, hey, we don't got no stinkin' logs. And then it's between law enforcement and the ISP. But they better really have no stinkin' logs if they say that, too. Excuse me? Okay, so next thing you need to do is contain the damage. And then you want to collect local logs, any logs you can get your hands on. And finally, we're going to image the disks on the victim computers. And we're going to talk here in a few minutes about disk imaging, so let's pass that for a moment. Now, crime scene analysis is a branch of criminal profiling that uses standard investigative techniques to analyze crime scenes. This is what normal investigators are comfortable with. They're used to looking at crime scenes, they're used to analyzing the evidence. And it's good in computer incidents because we can look at what kind of activities the intruder performed on the computer. And we can learn a lot about the intruder from the kinds of activities that they performed. So as we develop a profile of an intruder, we'll start with crime scene analysis. How is access obtained? What kind of skills were required to obtain access? It's quite one thing to take a canned program off the internet and run it against a system and cause the system to go down. Simple denial of service attack, maybe a simple sin flood or a mail bomb or something. A monkey can do that. It's quite another to go into a system as one that I saw, a mainframe where they zapped a single word of a program in memory that caused that system to go down. And they bypassed all the security in the mainframe to do it. Those are two distinctly different skill sets required to conduct those two attacks. And we can learn a great deal by analyzing the crime scene as to what kind of skills were required. And we can know a range of suspects considerably. Now how did the intruder behave on the system? Did they do damage? Were they able to clean up? If they're a skilled intruder, did they clean up their tools? Can we find nothing? Or do we go into on a Unix machine slash temp slash dot tools? And in there we find a copy of our canned shadow. Oh yes, this is a skilled intruder of the first order. Did they steal anything? Was it worth stealing? What was the motive? Why did they break into the system? Are they a disgruntled employee? Or are they like a group that seems to have gotten into the DEFCON website? Oh, probably none of you have had a chance to look at the DEFCON website today. You'll find it somewhat different. It's been changed. We should be able to tell a great deal about that intruder. And the personality types. Also we want to consider opportunity. In the case of the mainframe attack, the attacker had to have been on site. The only access to the mainframe at this particular level of security had to be local. So it had to be done at a time when the intruder was physically in the computer center. So this tells us a great deal. It helps us build a picture of the event. So we're going to build now an incident hypothesis. We're going to try to figure out what happened. And we start with witness accounts. And what I do is when there's an event, the first thing I do is I talk to as many people who are present at the event as I can. System administrators. Users who are on the system and observed some event occur. How did they know that they'd been attacked? How did they know what happened? How did they know when it happened? All of you I'm sure have heard stories about eyewitness accounts of automobile accidents where there were 15 people saw the accident and not one of them got the color of the car right. This happens here too. Absolutely happens here too. So we need to begin to get a picture. So I start with witness accounts. Then we consider how the intruder could have gained access. And we eliminate the obvious. I mean, if we know for a fact that the intruder dialed into a maintenance modem on a unit's host. And we can establish that unequivocally. There's a pretty good probability they didn't break through the firewall and come in from the internet. We can pretty well eliminate that as a possibility. It helps us when we do a back trace. Then we again consider skill level and inside knowledge. And we want to create mirrors of the affected computers. I'm nibbling around the edges of this collection of evidence thing here. And I'm going to get to it here in a minute. But a mirror is a perfect physical copy of the disk that we're investigating. Because we never, never, never work on the actual evidence. Never. Because we don't want to contribute to it. So we've developed the profile of the intruder. Considered a path to end of the victim computer. And now we're going to try to recreate the incident in the lab. Using our mirrors are perfect copies if we are fortunate enough to have them. Typically we will have a mirror of the victim computer. And consider alternative explanations. Let me tell you. It's re-embarrassing to go up before your boss with a report this thick of an investigation. And say, we have absolute conclusive evidence that Joe did this. He had motive, he had means. And somebody in the back of the room says, but wait a minute. Joe was canoeing in the Rocky Mountains when this occurred. And it was an inside job. Well, it could have been Joe. So test your alternatives. Now we're ready to start looking at some back tracing. The elements of the back tracing are the end points, obviously. Any intermediate systems. And if we're dealing with harassing email or something of that nature, we're going to need email headers. If we have a sniffer online, we're going to want to look at packet headers. And obviously we want to look at logs. Our objective in a back trace over the internet is to get to a dial-in point of presence. Now maybe this is coming from another network. And if that's the case, that's what we'll do instead of the dial-in point of presence. But the reason we want to get to a dial-in point of presence is because that's where the logs are. It's like, well, he's setting, why do you rob banks? Well, that's where the money is. So we want to get to a dial-in point of presence because that's where the logs are. The only messages that we can't back trace when we're talking about email, harassing email, those are who's a true anonymizer and those where there are no logs. And let's define a true anonymizer. A true anonymizer is something similar to a mixed master. Something where encryption is used to obscure the source and even such things as the time of the connection are changed. And even the owner of the anonymizer couldn't back trace this if he wanted to. Those are not traceable. Yes, sir? I'm going to show you in a minute. The question is, how do I get around email that comes from an unobvious place such as somebody masquerading as him or using hotmail or something of that nature where it isn't directly traceable to an individual? In that particular case, your ISP will tell you, oh, yes, we can do that. It comes from Joe Fabeats. And Joe Fabeats lives at 1234 West Avenue, New York, which is, by the way, a parking lot. Because we know for a fact that nobody would log into hotmail and put a false name and address in there, wouldn't it? But that's who did it. No, we have other ways that we have to do that and they don't always work, but I'm going to show you one here in a minute. Yes, sir? It depends upon the community. If I'm dealing with a community of sophisticated computer users similar to the ones we find around here, I see a lot of anonymizers. And I have to tell you a story. I generally don't go heavy on war stories on these shorter presentations. We had a situation where we were tracing harassing email messages where the victim had had her life threatened. And the person who did this was doing it through a service-like hotmail. I mean, we had 13 messages and we had taken those 13 messages. We'd gone through the subpoena process. We'd gotten all the information from the ISP. We knew who was doing this and then they started using an anonymizer. Why? They were caught. Oh, well. So these are not traceable. Yes, sir. Anonymizers, they can be, but they can be effective. But I find that when I use them about half the time, the anonymizer delts my message and it never pops out the other end. Yes, sir? So it's about making a decision on a target machine between a senior or a father-in-law distribution session. Wow, that's a tough one because it's hard to take that machine down. One of the things I'm going to tell you in a minute is that the way you do it is you pull the plug. And there's a good reason for doing it. It's kind of dangerous with a firewall. And if your firewall is built on a computer, then we can take our chances and do that. But what if it's something like a Cisco PIX? You're out of luck. Yes, sir? Yeah, the question is basically if I may paraphrase. What happens if it's one of those machines you can't take down, like a server or something like that? This is a judgment call. We have had clients tell us that we cannot take their machine down. They have to keep it running. It's a file server or something like that. And we've had to make the best image of it. We can knowing that the evidence has been contributed to. It's not the best. But business drives security, not the other way around. We're going to do another quick one here and then we're going to move on. Yes? That's a problem too. The question is what do you do if a portion of the disk that you're imaging has intellectual property and they don't want you to take it off-site? First of all, let's make the assumption for a minute that I'm working for you as opposed to being your adversary. There won't be a problem. Obviously, we'll work it out. If on the other hand I am your adversary and I've subpoenaed that information, then what you need to do, and you need to do it real fast, is you need to go to court and you need to get an order to stop me so that you can have the judge can issue an order that will protect the information on that from being revealed. That's intellectual property. This happens fairly frequently. I've had it happen several times. We've done some popular cases where that has happened. You may lose the case because of that. That's right. The judge may step in and quash the whole thing. That happens. Yes. Yes, one more. And then we've got to move it. You guys will be here too long and you'll miss the music. If it involves national security, you can bet that the situation is going to be such that that information is going to be protected. You can just absolutely guarantee it. Other than that, it's whatever you can work out with the owner of the information. You know, they may get the judge to issue a protective order. I've had situations where I've imaged a disc that had medical records on it and those medical records were deemed to be private to the individuals. And so a representative from the organization that maintained those medical records was permitted under supervision to delete them before we imaged the disc. It's tricky. It's very tricky. Okay, we've got some enabling relationships when we go after a back trace. And we'll start down at the attack victim down at the bottom. I don't know that I have enough cord here. I guess I do. This is the first place that we're going to look for logs in the victim. The next place we'll look is intermediate hosts. Now, here we have a router. Those don't have discs. Some of them log to an external device. Some of them don't. They have cash in them, though. And they cash their routing tables and they cash the traffic that goes through them. That information can be extracted. Then we want to get up to the ISP where the point of presence exists. There are going to be logs on the ISP's dial-in router, like an Ascend or Cisco dial-in. There's also going to be logs at the telephone company. Now, you've got to subpoena those logs. You're not going to get them just by asking for them, but they're there. So that's where we want to end up. And that's the key to the back trace. Now, obtaining subpoena. The way we do it is we notify the organization, usually an ISP, that we're going to subpoena and request they preserve the evidence. And we ask them who to send the subpoena to. Most of the larger ISPs have people that you send subpoenas to. So we always get or usually get cooperation there. Then we get our lawyer to file a John or Jane Doe lawsuit, which basically says, somebody hacked me. I don't know who it is, but I'm going to sue the bastards. And you've got to let me go and look for information so that I can find out who they are, because if you don't, I'll never be able to find out who they are. Or even if not, the court will let you do this. This is fairly controversial, but in most states they let you do it. Then subpoena the logs you need and get everything on the first pass. Don't expect to go back for more. And you may have to depose the people who collected the logs to find out how they collected them, because we have issues of things like chain of evidence and chain of custody. Okay, if we're going to use logs for evidence, they can be modifiable, which of course isn't possible. But we want to raise the bar as high as possible. If they've been spilled off to a protective log host, if they've been saved to optical media, if we have corroborating copies and backups, then we can probably use the logs as evidence. However, there will be questions of authenticity and there will be questions on accuracy of the logs. You must be prepared as the expert to answer those questions. And when the opposing attorney says, could the logs have been modified? What's the answer? Yes, absolutely. The logs need to be complete too. And complete in my mind is we have to cover all super user access, log in and log out times, attempt to use controlled services that's an M.T. thing, attempt to access critical resources, and I like to cover email details as well. That makes big logs. And there are organizations who do not want to log email details because they don't want them to be subject to discovery. And then there are those organizations who have been subject to some minor mischief where the logs just disappeared. Appropriate retention, assuming our lab is usually six months or more because we can go months and not even know that there's been an attack. Okay, tracing email headers. I told you I was going to show you how to do this. Everybody knows how to trace an email header when it's easy. Okay, and everybody in here I think probably knows that the farther down towards the Dear John part of it is the source and the farther up the stuff that nobody seems to pay any attention to is the destination. In this particular case, you'll notice that on number one there, the grade-in part, it was received from Read at Local Host on Web03.iname.net. Now what this is, for those of you who have never used I&A, this is one of these outfits where you can get one of those free email accounts you see, and you go into their web server and you compose it on a form and the web server uses a post and sends it out. So the reason that it looks like it came from Read at Local Host is because that's the process. Okay, but it's worthless. There isn't a piece of useful information there except for the time and date. That's the only useful information that message ID number is totally useless. The log that contains that probably had been gone for days by the time I got to this. So what we need to do is we need to understand the system and the system is a web-based system and since it's a web-based system, we hope there's an access log, an HTTP access log, and an HTTP referral log, and I'm running out of things to cross up here. Turns out in this case there was an HTTP access log and we were able to look at Web 03 and see that on Saturday 12th September 1998 at 1825-13, Eastern Daylight Time, don't forget the time, don't forget the time zone, that there was a particular connection. We had about 13 of these and turns out that the connection was owned by the same ISP in every single case. We were able to go to the ISP because we had the source IP addresses which turned out to be dial-up points of presence and we were able to find out who owned the account and we went to the person who owned the account and she said I didn't do it and her husband said I did and we had him. So the trick here is, the question that you asked is forget about who it's from, it doesn't make any difference, they're probably lying about who they are anyway and go for the time of the connect and start working back a step at a time until you get to the dial-in point of presence. When you get there, you've got your best chance, I don't guarantee you're going to win, but you've got your best chance. So in this case we contacted our own security officer, he gave us the connect account name, time, message ID and source IP address, all of which were totally worthless except for the source IP address. We located the ISP, contacted their security officer, got their logs from the source IP which was the dial-in point of presence. It happened to be in a Sun router and found out who was connected at the time of the email. Yes, sir. Cried. I had a lot of time invested in that. It took me 15 minutes to do that trace. No, we'd have been in trouble. If the logs were gone, we'd have been in trouble because we had gotten back to the ISP but we wouldn't have been able to go further. Yes. If the question was if I had broad enough number of instances, could I get telephone company logs? The answer is yes, I could anyway. But you have to, when you write a subpoena, you have to, and your lawyer will know how to do this, you have to write it very specifically. Not as specifically as a warrant. You've got some breadth there. The reason you write a subpoena specifically is to make sure that you get what you really want. Now in this particular case, in this particular case the ISP, and then this would be different from ISP to ISP and from dial-in to dial-in. But in this particular case, the IP address is dynamically assigned to the port and it was log, port, and IP address. So we knew which port it was and we didn't have to do all that. Okay, there's a question over here. How often in real cases did you work in if you encountered a problem of you could order it back to get it to the account and put it at the time, put it on the bottom here and discover that it's an American online user that basically used somebody else's account to get all the tricks that you know. It happens. It happens. And in that particular case, the only way you win is to have enough information that you can begin to start correlating things and start working through the maze. But in the case of American online, the maze is so big and there's so much stuff going on that it becomes almost impossible to get enough because where you really have to go from here is to the telephone company. That's really the next step back and find out what the dial-in was and that can be a massive problem. Massive problem. Okay. Yeah, there's one back here. This is the last one for just a second. We're not finished. I'm sorry. Oh, okay. I'm sorry. I was having trouble hearing. The public makes it. The question is if it was a publicly accessible port such as a cyber café or something like that, well, all I can say is that you're going to need some good old-fashioned cop work. You don't have anything to do with forensics. There's going to be just standard police investigation questioning people who was there at the time, that kind of thing because the forensics isn't going to do a bit of good for you there. The toughest thing to do is to put a human at the machine. We can trace back to the machine by putting a human at the machine and identifying that human is the hardest thing to do. Okay. Evidence collection and preservation. Forensic computer evidence we create. Physical images using a product called SafeBack. There are others. I happen to like SafeBack. It makes a physical image. That is the evidence. Everything else that you do with that is a lead. You protect the evidence, the entire physical backup. Never work on the image. Make a second copy. Forensic computer analysis. I use stuff from NTI. New technologies incorporated, but there are others. And that does DOS on Windows 95 and 98. There's some new stuff and it's free from NTI objectives that will help you do stuff with NT, which is pretty good. Never work on the evidence and always assure a chain of custody and a chain of custody means that you can account of your own knowledge for everybody who has handled that evidence since you collected it. Okay. Now what can we do with Forensics? We can identify email addresses, URLs, graphic and zip files. We can search for key words even in erased files. We can recover deleted logs. We can trace chain of events within computers, devices or networks. And we can locate masqueraders. We can't necessarily identify them, but we can locate them. Remember, every action on a network or on a computer leaves a fingerprint. It's there. It may be real hard to find, but it's there. I can try. Come on. Well, we're not... I'm afraid we're... There we go. That's the one you want? Okay. I hate it when a computer's smarter than I am. Happens all the time. May I proceed? Okay. The places we look for are Slack Space, which is the unused portion of a cluster after the end of File Marker. Mostly in BOS, but some Intel-based Unixes like Linux and FreeBSD, we get something similar to this. An allocated space is the space where nothing is or where erased files are. Swap files, window swap files, the whole treasures, and cache files. Especially internet cache files. We can often find passwords and all sorts of other stuff in there. But you have to do it from the physical backup. No other way to do it. Okay, logical backups. They're the ones for you that you normally do every night before you go home. Everybody backs up based upon the file or whatever file system is reporting. It's not the real file space. It's just the file. The file space includes the attached Slack Space. And that's a physical thing. It's clusters. Okay? Physical is bitstream backup or mirror. A mirror is when you take the bitstream backup and you do a restore to a hard disk. And it contains everything that was on the source, even if it's nothing. That means it actually contains the empty sectors. That's critical in the investigation. All those things I said we could get, we got to do this or we don't. So physical backups for files and windows. What we do is we pull a plug. And the reason we pull a plug is because we don't want to collapse any files. We want them to stay there. Also, windows creates and deletes dozens of files during the boot process and during the shutdown process. So we want to ensure that we don't contribute to the evidence. And then we're going to reboot using a DOS disk with a backup on it. Okay, quickly. I'm going to get to you, Nick. It's two slides forward. Okay? Take the backup for evidence. Take the second one as a work copy. Restore it to a lab computer. Now we have a mirror and we can go and analyze that mirror and preserve our evidence. For NT, we do the same thing. But after we build the mirror, we reboot it with DOS and NTFS DOS. Wonderful tool. Let's just look at the NTFS file system as if it were a DOS system. And then we can get the SAM so that we can go take a look at passwords. We can analyze other files in there using DOS tools. And we can use a tool called NTLast, which is one of the tools from NT Objectives. Now, Intel Unixes. Okay, Intel Unixes retreat pretty much like DOS. Okay? Once we have got a mirror of the Unix though, what's going to happen is it isn't going to boot. The mirror usually won't boot. So we mount it as a second volume. So if we've got a Linux disk that we're working on, we stick it as a second volume on a Linux machine, boot the machine, go in and work on that disk. We can also use the DOS tools to analyze the bit stream, the backup, not the mirror, but the backup. Yes. Yes, it does because it's a physical backup. It doesn't care what's on there. It's just copying sector by sector. Good question. Okay, we've got another question here. Oh, yes. Yes. It's a wonder that I'm not bald from tearing my hair out. Cryptographic file systems and certain types of compression just ruin this whole thing. Quickly. Yes. Yes, it does. Saudex's product. It's one of the, he said the sector translation have anything, have any effect. And what he's talking about is the difference in disk geometry between the source and destination. I assume that's what you're talking about. Yep, yep, yep, yep. In general, in general, and this is a general statement. There are exceptions. Some of the remappers, some of the compressors, some of the things that change the interleaves. This doesn't work right with. But generally speaking, the Saudex product, we'll try to adjust the geometry between the source and destination when it does the restore. It also gives you some ways of doing it directly and ignoring all of that and just moving the sectors. It's not perfect, none of it is. Well, as long as we're not doing anything to the evidence and we're only, yeah, that's right, it's read only. All right, so non-intel Unix is like Solaris. We remove the disk, place it in an Intel computer, be careful because we don't want to contribute to the evidence. Boot it with a floppy, a DOS floppy. Remember, we're taking a physical image so it doesn't matter that the file system isn't even slightly compatible. Take your image and we create the mirror. Mount the mirror as a second volume and another Solaris computer and go to work on it. Yes. Oh, yes, raids. Those are eight discs. We pop them and we do them like this and birthdays pass. There's no other, an eight disc raid would be a blessing from heaven. Trust me. When I walk in and I see a cabinet with about 40 in it, generally speaking, I raise my rates. Okay, authenticating electronic evidence. This is a problem because you've got to be able to show at the time that it's used in court that it hasn't changed. We fingerprint it with a product from MTI called CRC MD5 that does a 32-bit CRC check on every file plus an MD5 message digest hash and you're not going to change that file in any way, shape, or form without it showing up one or the other or both of those places. Then we encrypt it and we don't tell the password to anybody. We keep the key to ourselves and we use a nice meaty encryption algorithm. We use 256-bit blowfish. I'm sure there are those of you who can crack that but not within the useful life of the file. And that's the same as signing and sealing physical evidence, maintain a formal chain of custody. Remember I told you we can get stuff out of routers? This is one way. We can use a tool such as ODS's secure detector and it can find all those things. It uses our mind too and queries the cache. I'm moving faster because we're running out of time. CNDS is the Computer Misuse Detection System. What we do is we feed it logs that would be at least 30 days, the more the better. What it does is it looks for changes in usage patterns of users. What that does for us is help us to determine if perhaps somebody is masqueraded and come in as a legitimate user. Their usage patterns will differ markedly or if we have a normal legitimate user who's gone rogue on us and is going into places they've never gone before, it'll show up. This is a neat program. It's available also from ODS Networks. It costs a lot of money. I'm done. Questions? Thank you. I'm also out of breath. I'll hang out and answer questions, yes.