 As the slide says up here, my name is Robert Stout and I'm currently an employee at IBM. And I'd like to start off by thanking my manager for a foot in the bill to get me out here and approving me to do this talk. I think that's pretty cool. You know, even though it's something that I was working on my own time. Anywho, as you can see up here, I've got a couple of quotes. Can't hear me. Any better? Alright, I'll try talking on the mic. There we go. Game time. I thought I'd do something a little bit different here to try to get everyone a little excited into this. And what I came up with is the stupid ass quotes game. This first quote we got up here is the one that actually was the genesis for my desire to do this talk. Now I'd like to open up to the floor to anyone out there to name either the company or the author of this quote. And for the correct answer, you get one of these fabulous EFF shirts. Oh, come on. Nope, nope. Nope to all of those. I'm going to give this one to you guys. It's you. No. President and CEO of Hortica. I just love this quote. It is important for our customers to know that these tapes cannot be read without specific computer equipment and software. Duh. How about this next one here? The missing tapes require a tape drive to be read. That's a genius. And it cannot be viewed from a PC. That's just plain wrong. Any guesses on this one? Oh, come on. EFF shirt. Free EFF shirt. IBM. IBM. Who said IBM? You get yourself a free shirt. All fairness. I guess you could consider this one of my brothers in arms, if you will. I probably wouldn't, but it really helps demonstrate out to everyone what these spokesmen can do to your company. I mean, throwing out these stupid ass quotes of things that are just inane. And one last quote here. This one was relatively recent. So hopefully someone will be able to get this one also. Continue to maintain that it does not believe the information has been accessed because it would require specific hardware, software, and expertise. Ohio. Who said Ohio? You get yourself a free shirt, sir. And I see our supporter. Excellent. Ohio State, the administration. So today what I'm going to be going through as a summation is some of the reported cases that we have in the media regarding tapes being lost. The cost to a company for losing tapes. We're also going to be going over a bit on data breach laws, how they can affect you, and what we should know on the white side or on the incident handling side as far as the legal laws and when do we need to grab our lawyers. And then the fun part, recovering the data. How are we going to get the stuff off the tape when we find it out in the gutter somewhere? And then protecting it. What can we do to protect that stuff? Here are some of the larger incidences that we've had in the media recently. Ohio State lost almost a half a million employees. State, taxpayers, school. I think it was even lottery winners. Hortica, my good old friend of the CEO with social security numbers. We've got IBM, of course, lost an undisclosed amount of information. IRS, 26 tapes. I could just imagine how many people that holds. That's just plain scary. And then a couple others there. And for anyone who's really interested, down there on the bottom, we've got the data breaches website. They have a great listing of all the different data breaches, information about it, what was lost, how it was lost and such. And what does losing a tape do to your company? Well, there's a whole bunch of things that you can happen. You can be looking at losing your trade secrets. If your company willingly gives up a tape, do you still own those trade secrets? That'd be a good question for a lawyer. Confidential information, financials. Employee data, your SPI, sensitive personal information. That's when we start getting into the data breach laws. And then, of course, the company image. When you get the spokesman going out there saying, you need a tape drive, be able to read a tape. Civil Damages website there has a rather interesting calculator which you can plug in different information as far as how many pieces of data you've lost, how many people. And it can give you a range estimator as far as how much it's going to cost per person. And the basic ranges that they give out there is you can expect to spend between $1,000 and $21,000 per person, at least here in the States. For a case in point, especially with the cost. Akron Beacon Journal, I got these quotes out of. As you can see, Ohio State has spent at least $700,000 to implement some identity protection for the state employees alone. And then, I personally love this one quote there. The tape was stolen on June 10th from an unlocked car of an intern who quote directly out of the article and who quoted a person from the administration who had been designated to take the backup device home as part of a standard security procedure. What can you say about that? If that's your security procedure, so far we've talked about tapes being lost through theft. During transit, the IBM tapes bounced off of the back of the carrier, third-party. No fault to IBM, it was just a bad human error on the third-party. But what about the discarded tapes? What happens to those tapes when you're done using them in your corporation? What do you do with them? Do you burn them? Excellent. Let's see, who are those folks that take care of the environment, the EPA? They're going to be talking to you pretty soon. I'll be showing you why, too. I shred them, that's an excellent way. Digouse them, destroy them. But you'd be surprised how many people eBay them. Corporate auctions. And then dumps are diving. If no one has ever done dumps are diving before, you've got to try it at least once. So a case study here. I've got a big, fat library at home. A nice, gorgeous storage tech timber wolf. Beautiful, I love it. It's fun. Old school, yeah. It's fun. Good stuff to play with. Well, I bought over 100 tapes off of eBay. And then going through all these tapes, I started wondering, you know, what is actually on these things? So I saved about 20 of those tapes, set them aside, and then just started perusing through them. And as I show up here on the screen, one of the tapes was just totally screwed. Yeah, that happens. Get something off of eBay, it happens. Two of them were actually for a DLT-8000. I'm kind of cheap. I still have the 7000 model. I can't touch it. Five of them were short erased. That means that it was a DLT-7000 or older. So I could read the data, except someone did a shorter race on it. And I'll talk about that later. And then 12 of the tapes actually contained corporate backup. Scary. That's a question I do when I ask the EFF folks. When you do purchase a tape containing corporate data and assuming it wasn't stolen in the first place, does that mean that's truly my data now? Does that mean that I can post it out on the Internet? Yeah, being someone that does have some money, I don't want to have the lawyers chewing me all up. So anyway, the question is, do you securely erase your tapes? Do you destroy them? Obviously one gentleman out there does do them. Now getting into the legal. One of the things that I have to mention here, I'm not a lawyer. I never want to be one. I never will be. But I think for incident management perspective, it's really important for us to have a basic understanding of what the laws are like out there. So when we do encounter a situation where we are missing tapes, when we are missing data, we actually understand why we need to engage the lawyers and at what points. So starting off at the lowest levels here for the United States, we have state laws. Of course it all started off with California. They kind of set the path. From there, several of the states, most of the states by now, have enacted similar laws. None of them really as strict as what California has given us. But what we end up having is we have just gobs and gobs of different laws. So depending on where your tape was lost at, what data was on there, what type of person was on there, what state they were located at, it depends on what you have to tell the people. So one thing that I found during my investigation on writing up this paper was a Vigilant Minds has an excellent summarization of all the laws out there. And they've given me permission to put their slide up here. Of course, no one here, even on the front row, is going to be able to see it. But what I wanted to portray to everyone here is on the last slide there is a link out to their PDF file. But what I wanted to give you guys was the idea that they have each and every state listed across there and then the basic breakdown of what the laws are. What are the requirements for data breach notification to the end user? And this, for us at least, I think will give us an excellent footstep, a basic idea of what's required for us when we start having to notify folks. Moving on up, we've got the federal laws. At this point, they pretty much suck. We have this thing called Safe Harbor. Once again, not a lawyer, but from my understanding that it was enacted because the EU was about to say, United States, we're not going to do business with you anymore because your laws suck. And then currently, there's about six or so laws out on the House and Senate floor getting proposed that will hopefully pass and will give us some sort of firm foundation as far as what we have to follow across all the states. Oh, when we get down into all these laws, a lot of the folks out here, I'm sure, are working in companies that are international. So it's one thing to realize you've got a tape that's lost here in the States with just Ohioans' data on it. I think that's a word. But what happens when all of a sudden you also have data from Brazil, Brazil folks, from India, from Russia? China, the list goes on and on. At that point, you start getting really screwed. I guess that's why the lawyers get all their big money. Well, I'm pretty much flying through these slides here, but this is the part where I really wanted to get into, the fun part, recovering the data. Everyone knows, except for the press secretaries, there are several different types of drives. You've got your DLTs, your 8mm, 4mm, what have you. Then you've got different encoding types. Then you've got different models of each of the drives. But does any of that really matter? For our purposes here, I'm going to say no. If you get the most current drive, it's going to be backwards compatible, so you can read whatever tape it was before. Or if you're in my situation, you've got a next-to-current drive for the DLT lower-end DLT series, you can read everything up to the 7000s and back. So you're pretty good. If you get the current stuff, you're pretty good. A couple things I do want to point out is there are two fairly decent papers on forensics on tape. The whole concept of forensic analysis on tape media is nowhere near what hard drive media is. For obvious reasons, being linear, but the tape drives themselves the way that they operate. We don't have as much control over it, so it makes it far more difficult. We do have concepts of slack space on tapes, and the end-of-data, end-of-file markers. You can have some random data that's thrown out there. It could have been from previous rides or what have you. But that's stuff that we can't get ahold of. Perhaps the two-third-party services that I've got there might have some nice, interesting equipment that could pull the stuff off. But for us, that's pretty much out of our hands. Like I was saying, forensics, tapes aren't a hard drive. But for the most part, we can still use DD to pull the data off the tapes and then work on the local copy of it, which is, of course, much faster, much easier to get through. And the end-of-data markers in shorter races. The concept behind a shorter race on DLT drives, at least, keep in mind that other drive formats are different. 8-millimeter drives or 8-millimeter tapes, you actually can get past the end-of-data marker. It's much more relaxed. But on DLTs, at least, the end-of-data marker is something that's hard-coded in the firmware. Once you hit the end-of-data marker on a tape, that's it. On a DLT tape, you cannot go past that. There's a couple of concepts as far as how you can get past them. And I've tried one unsuccessfully. Might eventually do some of the others. But as you can imagine, having five tapes at home where I can see, they put an end-of-data marker, shorter race. A shorter race, let me backtrack here, is just where you start at the very beginning of the tape and you just say, this is the end of the data. So all the rest of the data is still left on that tape. Everything that was on there originally still exists, except the drives firmware will refuse to let you get past that initial end-of-data marker, which is at the very forefront of that tape. Here I'm trying to illustrate a bit of what that end-of-data marker is. Ah, nice good old mouse. So here at the beginning of my pretty little graphic, we've got the beginning of tape marker. Just says, here we are. And then here we're going through an assumption that we've previously used this tape. We've written several files out there all the way to the very end of the tape itself, where the end-of-tape marker is. But then we came back and rewrote these first two files, at which point our drive immediately, as soon as we close out file number two, puts an end-of-data marker on there, which means file five, although it truly and still does exist out there. We cannot get out to it. As soon as we try to forward over to that, the drive's going to say, huh, we're not going to get there. One of the concepts that Bruce brought up in the forensic paper that I pointed out earlier was the idea of starting to write to your tape drive, and then during that write, yank the power off of that drive before it has the opportunity to write that end-of-data marker. Unfortunately, I found this might work, but not if you have the latest respect firmware on there, because as soon as, of course, you know, getting the new drives, and I got brand-news DLT7000 drives for myself, and as soon as I did that, I upgraded them all to the same firmware, because they're all in the same library. And as soon as I do that, I can't get past it. But anyway, I give you a little story of what happened. You know, I write it out. I start writing junk data to the tape, and then I yank the power off of there, you know, all well and good at this point. But as soon as I plug that power back into the tape drive, it realizes there's a tape in there, realizes the write did not finish, so it goes through, plays with the tape a bit, tries to find out where the data should have ended at, and politely puts an end-of-data marker there for me so I can't get past it. Really annoying. But another concept there to bypass that is a physical splice of the tape, actually ripping the thing open, slicing it out, just trying to find exactly where that end-of-data marker is, and just slicing that puppy out. Now, that one I'm kind of afraid of totally wasting my heads on. And then try to unspool it with the power off. But with the DLTs, you've got the little spindle that you spin it up on. But yeah, you might be able to take it out and then just pull it out eventually. But, you know, once again, I didn't pay that much off of eBay for these drives, but I don't want to screw them over. That's a good idea. Using a maintenance tape. I'll tell you about that later. You've got a sec. Basic recovery steps under Lennox. Really rather simple. Very similar to what you're going to do for a hard drive. A couple things different, of course, is you're going to be using the tape devices. The tape info command there is just going to tell you information about the tape itself. It's going to query the firmware off the drive, tell you what type of format it's in, if it's compressed, if so, what kind, and all that good jazz. MT, what you want to do is just set the tape drive itself to a variable block and also with your DD, set your block size to the greatest to ensure that you can pull off whatever block size is on the tape. There's other ways you can figure out exactly what size your block is on the tape, but you know why bother. Just set it to the largest and you're good to go. And then you wash, rinse and repeat. Keep running the DD over and over again for each of the files that are on the tape. A couple of tools out there already that will help automate some of this. We've got tape cap. It will go through. It's probably written for Amanda though, but it can retrieve some information about Amanda that could be useful for some of you folks out there. But to recover the data on the tapes themselves, we can yank off the images from the tape with DD, but to actually recover the binary files off of those tapes, the easiest way that we can do that is grab the original software it was written on. So once we grab some of those files off of the tape, we can start looking through it, you know, with our strings, with our hex dump, and see exactly what format it was written out on and what backup utility we were using or they were using when they originally did it. And just go out there, grab it, use it to restore your tape. And for the most part, it works rather well unless you're looking at something like a... what is that ArcSurf thing? I don't know if they're even still selling that thing, but it could be rather expensive. And of course TSM is not free. One caveat with this though is TSM, and that's where the majority of the rest of my talk is going to be going to. TSM for everyone who has not used it, which I'm going to take a wild guess, is 99.9% of y'all. Could I get a raise of hands for anyone that has used TSM? Except for the IBMers. So we do got a couple out there. So basically, the basic concepts of TSM is completely different off of all the other backup utilities. TSM uses a database to track all files on the tape. In essence, TSM doesn't even care what's on the tape. All it cares about is where the database says the data is on each of the tapes. So it'll just say, you know, tape three has the fifth file, which is a file we want. But that introduces an issue when we find a TSM tape on the side of the street or say on the highway if it bounces off of a truck. Because we can't just go out and grab TSM from our friendly Egghead software place and install it to recover the tape because TSM has no concept of importing a road tape. And here are just some of the highlights of TSM, why it's different, how it's different. One of the unique concepts that TSM has is it's always an incremental backup. Even the very first backup you've ever done on any brand new system is considered incremental. From there, everything is incremental. Such thing as a full backup. Database is the heart. There's no import already mentioned. And the last bullet here brings up an interesting twist for anyone who needs to identify what data was on the tape. Say after it's lost. Because, you know, assuming the tape was just written, then the database knows exactly what was on that tape. But if that tape was written a week, a month, two months ago, then the database has started to expire the information that existed on that tape. So if you core the database, it's going to say the tape was 85% full and these are the 85% of the files out there. But it has no idea of the remaining files that were out there. And here I'm trying to illustrate that concept. Where is my mouse? So here we go. We've got basically two tapes out here. On our first tape, we run through, we do a client backup of files one through six. And then on the second tape, we go through and start picking up the backups again, but from files four through, what have you. Now, making the assumption that the system only uses one set of files, only retains one set of files for the backups, TSM is going to expire the client file four, five, and six on tape one. So if you ask the database in TSM, what files do I have on tape one, because I just lost it and I need to know what's out there? TSM is going to say, well, you've got files one, two, and three, and you have no idea that any of the other files existed on that tape. The way you can get around that is if you happen to have a database backup from the time when the database realized it, you can restore it, big pen in the ass, probably not worth it. Now we're starting to break down into the TSM tape layout itself. When you start looking at the TSM tape, the first file you see on that tape is going to be the TSM label. It's written in the IBM 871 character set, which looks like googly gosh, until you convert it over to something a little more legible C&ISO standard. At which point you can see, I've got it highlighted there, that the tape that I've got in question here that I found out in my library had a volume idea of 100227. Going on, as we look on the next file that's on that tape, we actually start seeing some of the juicy stuff. This is where TSM actually puts the data for the files out there. Here on the slide, I'm sure the folks in the back can't see it, but a quick rundown, which we're looking at here, is we can see up here on top, the host that this file was owned by was Mayo. It was a Linux X86, and it was backed up on the standard domain. And of course down here, we've got the file information, where was it located, the file name itself, some more TSM specific stuff of the management groups. It even tells us who was the owner of that file, the ASCII representation of the owner, which happens to be replay. And if we look at the very bottom here, we can see that the beginning, as we know by looking at the file itself, it's a JPEG, and looking at the bottom of this hex dump, we can see the, can't think of the name for it, but the fingerprint, fingerprint for the JPEG itself. So we can see that the data is actually embedded right there. So if we wanted to, we could go through there, cut it out, dissect it, save it, and then we could view our JPEG file from that way. So as we can see, we can just go through the tape, we can just extract the information we want by using simple commands, DDE, hex dump, strings. We can go through, manually save the files off of our tape if we wanted to, but when you're starting to look at 40 gigs worth of data, do you really want to go out there and save a couple thousand files? No, no, not me. I'm a little lazy. So looking out there, I found this great little program called ADS on tape. It's written by this Icelandic gentleman. And by the way, Icelandic language is not a good thing when it comes to debuggers, especially a community debugger, freaks it out really bad, but long story on that one. But anyway, the problem with ADS on tape is it's written for the predecessor to TSM, which is of course, Adstars ADSM, which was bought out by Tivoli, and only supported version 2 and version 3, and only runs under AIX. So, you know, not having, not wanting to run under AIX and having version 5 of a tape, decided we'd do a little updating for ADS on tape. Again, I give you TSM tape. It's available out on SourceForge. It runs under Linux and can read 5 TSM 5.x tapes. Of course, I only tested on 5.2. That's the only tape that I've got. It's based off of Thor Holler's ADSM tape and is under the GPL license, just as he had his. So anyone interested? Go right ahead. It's out there. Go enjoy yourself. A little screenshot of the command line options you've got. All rather basic. You can either restore specific files. Say if you just want to look at the Etsy Shadow file, you can just dump that bad boy right on out. It also has the ability just to audit the tape, just to go through, find all the information about each of the files, the length, the owner, the size, what have you. Here's the output from running it on the tape that we were showing you earlier. As you can see, the volume label, we get dumped out to the screen. And the first file on there is also the same file that we were looking at the hex dump of. And as you can see, it restored it for me. And it's a bit for a bit coffee. One of the special features you get on the TSM tape software itself is it also creates a restore log CSV file. Through there, it grabs a little bit of additional information that the TSM protocol has embedded into the tape itself. From there you can see the node name that the backup was done on, the OS, the user that it was backed up, that owned it. The permissions of the file when the backup occurred, and even the UID and group ID of the files. Okay, we're now getting into the mitigation portion of it. So now that we know that you can really read those tapes when they're lost, and that special equipment isn't really necessary, stuff you can go pick off of eBay for 50 bucks, not a big deal. Now we're going to take a fun look at protecting the data. First off, no brainer, you've got to know what you've got before you can protect it. If you don't have a good inventory, no holds barred. Probably the easiest way to protect your data is doing client-side encryption, especially so for TSM. It's built in, really simple to set up. And at that point, even the administrators can't even look at your data. It's encrypted. Downside of that, of course, is if you lose your key from the client, you're screwed. You can't look at it either. So you've got some good key management going on. And then, of course, a little plug for IBM, LT04. Got some built-in encryption right into the tape drive itself. Easy whiz bang, going to cost some bucks though. Now for data destruction. You know, degaussers. I love these things. They're really cool. But this one in particular, the Eliminator 4000FS. You know, with a name like that, it's going to kick some ass. It's going to blow everything on there. You're probably going to have a little smoking charge by the time this thing runs through. I have no idea how much they cost, but the name is excellent. Other comical sides. A couple of links here. Do-it-yourself data destruction. The link has an excellent display of bash it, heat it, smelt it, microwave, shred it. And then the fine art of data protection. Pulverize, then liquefy. I hear that the Brits like doing that. But my personal favorite is Thermite. Now getting back to the original comment out there in the audience regarding burning them. This is how I know that tapes make one hell of a snow. Now Thermite. I'm sure everyone is aware of it, but Thermite is actually used for the railroad industry. You can weld, steal together with this stuff. It burns, what was it, 2,000 or 4,000 degrees. Huge sparks. Excellent. As my daughter said, anything that makes fire is cool. And I've got a... I'm not sure if this is going to work, but I'm going to try it. Here we go. There we go. Fire. You just got to love it. Now the ironic part of it was, you could tell at the very bottom I have a huge steel plate that was the basis of a railroad track that I was setting this on. But the very bottom tape, the very bottom of it still existed. But the rest of everything on top was just charred remnants of tape. It was excellent. But anyway, folks, that's my presentation. Thank you for suffering through me. If anyone's got any questions, please do speak up. Otherwise, thank you much.