 Now, this morning we have Wyn Schwartau talking to us about his history, which is really legendary. So, sit back, enjoy, and please give a warm welcome to Wyn. Am I supposed to ask for you to do that again to make me feel really good this— No, no, no, no, no! This'll fuck up the audio guy in the rear, right? Am I fucking you up really bad? This is going to be a rant. This is—I haven't spoken to DefCon, I guess, in 15 years, because I'm not the traditional hacker. Network protocols, I lie, I cheat, I Google, and I have a lot of good friends. A lot smarter than me. But I began, and I didn't really realize that I was hacking. And what I want to show you is some experiences that I had starting in 1958. Anybody alive in 1958? Fuck you, Barry. Really? Okay. Oh, you were two. And I grew up in an engineering family. My father went to MIT at 15 and worked on radar development during World War II. He gave me a non-optional education on electrical engineering early. My mother was the first audio engineer at NBC during World War II and got a whole series of patents on— Yeah, my mother, I wish she could come to DefCon, but she's like, be 140 now, so not going to work out. That was my first DefCon and my first beer. And this is my father giving me every week, my parents split up. So on the weekend, I went down to my father's house in Greenwich Village and watched the tourists get mad at the beatniks. And he gave me non-optional electrical engineering education. And that went on for years and years. So that ended up being weird. But I ended up doing TV repair work at six years old, because how hard was it? You take the tubes out of the TV, you go down to the drugstore, you test them, you take the two dollars, get a new 12AX7, go back, plug them in, and a neighbor in the apartment building gives me 50 cents. I mean, but that was in the 50s, and they thought, oh, how do you know how to do that? Well, my father told me how to use the tube tester. That was really it. That's what our electronics stores used to look like when we had to build shit. And this is the world of computers way back then. So my first hack was in 1958. And anybody know what these are called other than Barry? Rotary telephones, of course. And so we had these, obviously, and some of them were four digit numbers. And this is what a bill looked like back in those days, when you actually had to pay rental fee for a phone and a rental fee for the wire. Phone calls, and this is in New York City, were still 10 cents a minute, even if I'm calling just across the street. So things got expensive very quickly. And apparently I was an unruly child and talked a lot. So my mother locked the phone and, well, I wanted to make phone calls. So what do you do in a non-destructive way? How do you hack a rotary telephone? With no tools, no destruction, whatsoever. Well, here is the old people up here. And they're all giving me this. Now, don't interpret that too sexually, please, from them. That's them. Inside of these are what's called stepper relays, decade relays. And when you dial, basically, it's click, click, three or a seven. And if you get your timing down with the lock on the phone, just right. Hello, operator, may I help you? And please connect me to this long distance number in Greece, or to my friend across the street, a non-destructive hack that my mother then on the next bill came along. And it was too much. And she goes, so you figured it the fuck out, did you? Was it smart? No, it was just a kid not accepting that there was a lock that I could not bypass, even though it was a completely non-technical hack. So I'm screwing around with analog computers and stuff throughout the years. And everybody know who Richard Stallman is? I grew up with him. But I didn't remember that until I hit 60. Because we knew him as Dickie, Dickie Stallman. And so he and I built little transistor radios and all sorts of crap. And his mother was a milf just saying, I can't shut up and out. Now, the neighborhood drug stores back in those days would hire a bunch of us kids for like three hours a week each to spread a little bit of the 80 cents an hour wealth or whatever. And inside of there, of course, you see, there's a phone booth. And those were the phones of the day. And of course, we wanted to make phone calls. But, you know, we're 10 years old. How do you do it? How do you do it for free? Well, that's the way these things used to work. Stick them in the slot. And this is the US system. It was different and you couldn't do this hack in other countries and not everywhere. So here are the pieces to the hack. Chinese laundry cardboard. And a penny. What we did, and it was a design feature or fault by AT&T, depending upon how you look at it, you take a sliver of one of those card boards, slide it down the dime slot all the way to the bottom, pick up the receiver, put a penny in the nickel slot. It registers as a dime and you get your penny back. So if you have a pocket full of pennies, calling Greece with the operate long distance operator on the phone is a no brainer and they're getting a legitimate money trigger. And we lived at that drug store for years making those kind of calls. And again, it was a stupid, stupid, but highly effective way to bypass the security, the features, the controls that were built into that system. So then I go on to high school and they got a computer thing and high school and I didn't get along very well. So I sort of left in 11th grade and decided to go into the family business, the recording industry, music, TV and all that stuff. And I was 16 years old. And so that was the first studio I got to work at, which was fairly overwhelming except for the fact that my father did it. And I had been exposed to the studios and it really wasn't all that scary. Anybody know who that is? My first recording session, shout it out, anybody? Nugent, it was the Nuge. He would get naked in the studio and hump the console. Just saying, please don't repeat that and publicize that. And then this was my first mechanical system. It was a cyber mechanical system for cutting records, vinyl. Some of you may remember vinyl and this is how we used to do that. Now, later on in life, I was at the studios and there was a thing that used to be called telephone credit cards and there's one of them. And once you got the number and you got the number by paying a guy on 8th Avenue and 42nd Street a dollar, you do that until the FBI decides to wrap you around their heads a little bit. So that was my first visit from the FBI for doing some sort of crime. So that was 1969. It was not a hack, it was pure out and out theft. So then I get involved with Hendricks and all these guys doing complex systems. Robert Mogue and that's the first synthesizer down from Electric Lady Studios in New York. So I kept seeing and being involved with complex systems. This is where I learned that assembly language and I do not get along either. And I built it, it semi-sort of worked and that was enough. That was a computer that we used, deck computers to do synchronization for TV shows. And this one was from applause. And trying to make a deck computer back in those days actually do what you wanted to do was an incredibly slow process. Back in, this is the day in our studios in New York, we had our telephone exchanges back in the day. So how do you hack that? Because what we needed to do was get audio from the New York studios into the LA studios. But AT&T wanted $1,500 an hour in 1970 dollars to be able to do that connection. But we had a switchboard. And once you have the switchboard and the PBX system, you can bypass very easily the controls and get directly out to LA and move the audio back and forth. Again, is it a hack by today's standards? From our standpoint, it got the job done. So being broke back in those days and a roommate, we used to get our utility bills on punch cards. Did anybody ever do that? All right, one old guy, two old guys, fair enough. Is that G-Mark? We'll take your mask off so I can see your face. Thank you. Yeah, let's hear it for G-Mark. Come on. So we used to get our bills and what did we want to do? Not pay them. And keep the power and the phones. So we had access, of course, to computers at NYU, what have you. And then we discovered by looking at the punch card, the dollar amount, let's say it was $7.32. There were no fields to the left of the number. So what did we do? Add a minus sign. Every month we got a check for the amount of service that we used the month before. Wait a minute, is this doing a live thing of me? Oh, shit. I wonder if we can say that. Oh, shit, okay, it did. So that's how we, it worked for about two years and they upped their security at that a little bit after that. So then I got hired to do a build a studio complex for some guys and while they wanted to pay me reasonably good money back in the day, they had virtually no budget for equipment. And what we needed to do was tie together all of these studios throughout the facility and there were eight or ten of them. And instead of the big patch bay they couldn't afford, all the wiring associated with they couldn't afford, we decided we have another option here. What picture is this? Yeah, AT&T. AT&T owned the universe. Them and IBM owned the universe back then and the AT&T building was about five blocks away on 11th Avenue. And what did we do? We did what is now called dumpster diving. Looking for old AT&T equipment that they threw out because if there was one little thing wrong, they would just toss it out. Which meant from our standpoint, all we have to do is find the one little thing wrong and fix it. But it was all free. So by putting all of this stuff together and repurposing this equipment, we were able to build recording studios and switching networks using stepper relays. The same way that we used to hack the phone when I was a kid. And the stepper relay was the key to it because you could dial up anything you wanted to do. So there was a rack of dials and it rerouted all of the signals everywhere throughout the recording complex and it didn't cost us a dime. Thank you AT&T for that one. My first home computer where again I learned I'm a really shitty programmer. I'm awful. I got at the war a semi-sort of kind of kluge and that was when my wife first said that she was now a computer widow and that was in 78. Oh, there's my wife. What happened to my wife? Well, thank you, Mike. My wife, she built recording consoles and this was the first computerized, automated recording console 1979. Never fucking worked. I kept blaming my wife for it. I didn't wire those but it was one of these you had to know 700 control commands and do alt phase shift antenna metal in order to make anything attempt to work and it didn't and that was one of the projects that we were trying to make it work and fuck it, just completely gave up. I had a studio in New York for a while so that was the background on which we started or I started learning there's always an answer. It's like that Apollo, one of the greatest scenes in movie history, Apollo 13. This is what they have up there. That's all you get to work with. Nothing else. So back in these days, I was doing a lot of studio work. I was on the road with various musical artists and as we always say at DEF CON something's always gonna fuck up completely. Well, I'm gonna tell you about a couple of those fuck ups and we had to hack our way out of them. 1975, 6, I don't even remember the year. We were in Kingston, Jamaica and it was a thing with the British government and they were trying to make peace because there were troubles down there and Stevie Wonder and Bob Marley were the big acts. So we had to build riot cages into the stage because of the suspected possibility of terrorism that the British Embassy had told us about. Now remember, I think I'm 23 years old here going terrorism, concert, fuck me, get me home. Well, concert goes on. Soccer stadium, 120,000 people in the soccer stadium and keep in mind that Jamaican police don't have police. The ergo, the riot cages built into the stage. So the chaos was suddenly all of the lights and all of the sound went off with 120,000 reggae-ing people in the audience. The fuck do you do? We're in a foreign country. We don't know where the power, it was like give us a plug, it's like what we say, give me clean internet and I'll take it from there. That's all we wanted, give us some good power except in this case there was so much power the pole transformer went. 120,000 people. What do you do? Well, one of the absolutely stoned out of his mind groupies, it was his, not mine, CO2 cools things down really quickly. So we had two of the groupies, not the groupies, roadies, sorry not groupies, roadies get the CO2s that we happen to have because we had tube equipment and it gets hot. Went over to start cooling down the transformer as they start to get arrested for terrorism. And it took a few minutes in order to get them to realize we would like your help keeping this, but we hacked the entire power system for the stadium with several cans of CO2 and the concert finally went on. Sort of like Jeopardy last night. We finally got there, took a while, but we finally... And again, is this a true hack? I don't care what words you want to use for it, it was when the shit really fucking hits the fan. What do you do? And you've got to pull out every single stop and I get into rants about graceful degradation and enterprise networks and things and I talk to the CSOs and I say, where's your graceful degradation and segmentation plan? Say what? All right, well you've never been in rock and roll, dude. So there's a few takeaways and the slides will all be available for you guys later on. I absolutely loved Liza Minnelli. We recorded Liza with a Z and what she did in the studio was rather amazing. She drank a bottle of scotch, took her clothes off and proceeded to sing. All right, that's the way you want to do it, Liza. There's going to be a TV show called Liza with a Z. She was coming back and doing, you know, now going to be famous again. So live shows, live thing went absolutely fine, but we need to go back to the studio. Now in those days, we had film, deck, we had a video, a little bit, but mostly film and an audio machine with a thing called Pylotone. Nobody's probably ever heard. Anybody ever hear of Pylotone? Oh, Masuchi, of course you have. I went to high school with this motherfucker up here. 9th grade, I've known him 56 years. Thanks for coming, Mike. And so you get back to the studio and you expect the Pylotone on the film and the Magstripe to work with the audio, right? Well, not so much. Completely out of sync. We have a live broadcast coming up on NBC. The fuck do you do when you're completely out of sync? Well, you cobble together, this is very sophisticated equipment compared to what we have. We cobble together a set of oscillators and two power amplifiers to drive a set of hysteresis synchronous motors for anybody that may remember what those are before the DC motors came along. That Lissajou pattern, the one that you see there, we had to take dials on the oscillators and the machines because they introduced friction into the system and maintain that Lissajou pattern for four hours by hand. Did it always work? Sometimes the hi-hat in the video was awful a little bit, but in general, it was the best that we could do. We cobbled together some equipment that didn't exist and we went to the shop and did it. Again, is it a hack? We got the job done when otherwise there would not have been a TV broadcast at all. Synchronization failures were a disaster in the early days of the recording industry because everybody in the late 70s and early 80s is saying, we're going to go digital. Digital will fix everything. Except for those of you who remember what digital was like in the late 70s and early 80s, it was really highly experimental. And so we did a lot of experimentation on how do you get, instead of just 24 tracks, how do you get 48 tracks? And can I record 24 of the tracks live in LA, the other 24 live in New York and keep them all synchronized with equipment that doesn't work from England? No offense to my British friends out there. Well, synchronization was awful. Remember the song, Do You Think I'm Sexy? That was with Rod Stewart. We were on that session for 72 hours. 72 straight hours. Just trying to keep the drum tracks from that song, from LA, in sync with the audio and the voice being recorded in New York. That was 72 hours of absolute hell where we had to, again, revert to graceful degradation and a manual mode of working. Completely manual and you could grab a few seconds of sync and it would fall out again. So it was an absolute nightmare. But synchronization in our world is about what makes everything work and getting the timing right. We've all heard the stories of, well, my clock was on CST and their clock was on Central European time. It was eight hours off or I'm using an imperial measurement system versus a metric system and everything completely fucks up. The synchronization issues that we face are very similar to the types of synchronization issues that we face today but are much more profoundly aspects of what can go wrong and that's why thinking synchronization is still incredibly critical. Charlie Daniels, I got flown in from LA to record him in Madison Square Garden. My crew was there. Everything should have gone very smoothly. However, what we were supposed to do was record Charlie Daniels and the event and the bands through a fairly large console onto a 16 track tape machine. Fairly normal activity for us back in those days. So you have 40 microphones go into the console, branch them out and record them properly on the appropriate channel on the audio machine. Except, no offense to any Union people in the room, but back in the day, Madison Square Garden and Afia hired the most incompetent people known to man to assist us in our professional recording endeavors. Every single microphone signal was ground looped so every single input or 40 of them had exactly the same signal. Right. No isolation, no differentiation and we actually had to do it all live with massive amounts of equalization and digital filtering, such as existed in 1981 to be able to get it down to a mono machine and then have to go to the studio and rebuild the whole thing from scratch. So audio quality, if anybody's interested, I actually have the vinyl from that. The audio quality is absolutely awful, but the radio stations didn't seem to give a shit because they were broadcasting on AM and the quality was so bad anyway. So this was about preparation and architecting and never really trusting folks to do it, do the jobs that you expected, especially in that environment back in those days. So there's plenty of realities and I spent a lot of time doing remote work and remote work is where if it can go wrong, it's gonna go fucking wrong and it's gonna go wrong in a big way and these are just a couple of the examples. There was a studio we worked at called R1 and it was a jingle studio. One of the days, the money making for the recording industry that would support artists was from the jingle houses for advertising and so it was pretty straight ahead stuff and so we had the studio was roughly like this and 40 musicians out there, the producer from whatever company it was, I guess this one was Alka Seltzer and the jingle house, they're all sitting there and then they say to me, when is there supposed to be smoke coming out of the console? Oh, fuck! So we got 40 guys and we got a console that just went up in smoke. What do you do? What do you do? Well, you panic and back in those days we had these things and I think some of the AV guys still use them, they're called SM, they're mixers, little tiny sure four channel mixers and so what did we have to do? We had to get all the gophers in the studio, extend all the microphones, wires about 100 feet, wire them into the studio, put them into the back of these little mixers, take those mixers, bypass the console, go into the tape machine where we can manually control the levels throughout the entire session and never be able to hear what was going on until we did a playback to see how good it was. That was sheer chaos. But again, there was a way to do it. We understood the architecture of what we were trying to achieve. We understood what we had in the consoles, how do we replicate the console feature set into something else in under 20 minutes? And it's a bitch, but these are the things that have brought me into the security industry always thinking of what the hell can go wrong. And maybe that's why I wrote several of the books on these topics of what can go wrong when you don't really think about it. Charlie, that's Hank Williams Jr. We were doing a thing in New York, anybody ever hear of the Lone Star Cafe in New York? Yeah, we had a studio in the basement of it, and every week we'd record whatever the act was live to 570 radio stations across the country. Not really a big deal technically, we knew what we were doing, we had the tie lines and everything went well. Hank Williams is up there, and he's playing and a couple of guys whose particular orientation he was not thrilled with decided to have a beer fight and throw beer on him on stage. It's a little tiny stage, smaller than this. And he up and walks out, and we hadn't been through 30 minutes of the live show yet. Gone. No act. What do you do? What do you do? Well, we had the radio announcer, Mike Fitzgerald was his name, he was the New York radio announcer for the station that sponsored all this, and I said, Mike, vamp, vamp. And I said, I need 10 minutes. So what did we do? We were live recording everything, so I rewound the tape to the very beginning of when we began recording when we introduced him live, and pressed play, and replayed for the second half of the show exactly what was done in the first half of the show. Nobody seemed to give a shit. It was, again, in back in those days, and I think it's still the same rules, if you don't fill the airtime, the FCC is going to get on your ass. So whatever it was, fill the air, fill the air, and again, we kind of got through that one as well. The live stuff, when you're doing any technical stuff at all, and you say, oh, this is going to be really easy, the answer is no. How many years of Hacker Jeopardy have we had where we, ah, we got it wired this year until somebody has decided to put Vista on a machine for us, and it was the first time, I think anybody ever yelled at Hacker Jeopardy was, hey, does anybody have a non-Vista machine we can have now? And it finally worked our way through that. As we build our systems, complexity is our enemy, absolute enemy, and we want more features, so we see the increase of sophistication of software. We see the increase of sophistication of hardware, not necessarily so much because we need it, but the vendors want to be able to say, ah, we have 22 new features, and it's worth another thousand dollars for you to give it to us, for us to give it to you. And we're increasing our complexity without due thought, without proper planning. Again, what do you do when the IT hits the fan? And that's why I go back to these presentations and my prior career of it was all analog. We had nothing digital that worked in those days at all, with the exception of a couple little pieces of equipment. Everything was guessing, trial and error, and it was a mess, and today we're still doing this trial and error, but we should learn from history a little bit more planning in when the IT hits the fan. How do you function? What is your minimal, viable, operating capability? Do we plan about that? The businesses go, no, IT you take care of. The business folks don't understand enough. You guys do, and that is part of hopefully the lesson here, is to be able to plan for absolutely everything. One of the things we used to do back in the day, was get a recording console. Let's say a new recording console is in, or we've done an overhaul of a particular part of a studio. One of the things is we called it Turn All the Nobs. What we did, and the reason we did it, we wanted to crank up the volume in every single circuit on the console, and we're looking 4,000 knobs maybe on a good console, because we wanted to stress it to the point. We wanted it to break under our lab conditions versus break in a live situation, the TV situation, or live recording in the studio. So cranking all the knobs for us helped identify problems, similar to where our airplane engine design is done. Stress the hell out of it. And one of the things that I think I've missed in our field, and if somebody's doing it really well, I would love to know about it, is do we have any sort of protocols for stress testing of our networking infrastructures to be able to say, okay, we need to put a limiter, or a compressor, or a restriction control, because it cannot handle all the traffic, all the things. So back in 91, we were building, this is Steve Katz, the world's first CISO. Brilliant guy and a dear friend from J.P. Morgan. They wanted to do the first single sign-on system. 1990. And I was on that team. And we tied together security dynamics, which became RSA. There was Sun Microsystems with Solaris. Delphi Project from MIT was involved. And so we built this whole single sign-on system. We were so proud of ourselves. So proud. We're showing it at a couple of conferences, and it works brilliantly. Our single sign-on is finally a reality until it fucks up. So Steve says, let's deploy it. Deploy it where? We're going to do it at J.P. Morgan. Cool. They deployed it company-wide. Sounds good, right? Well, what we had not considered was the time factor of authentication when 10,000 people try to log on at 9 a.m. at the same time. What happens to your network when you don't have accelerators, you don't have spread bandwidth, you don't have balancing circuits, all the shit we have today we didn't have and we completely fucked it up by spending all this money and designing something that was great for 10 people, not for 10,000 people. So turning all the knobs, stress the shit out of things. I cannot impress enough that we need as an industry to really develop a stress mechanism, a stress protocol for various of the environments that we're in. And let's turn all the V-nobs. We can do the same thing and very easily automate that stress testing process. And I see Dave Fizz is up here, he's designing it right now. Do you have the patent shift for this? Yeah. So incident response, whether it was the incident response with my mother and the rotary telephone, and my response was fuck that, I'm going to make a phone call somehow, all the way up to catastrophic failures with a couple hundred thousand people, I have a lot of TV with all of those and I didn't even show you some of the really nasty ones. It's all about incident response and appropriate planning, understanding your architectures and then making sure that you have a way to degrade and a methodology to recover. And on that note, I wanted to thank you so much. I'm here for some questions if you like, but that was my hacking career beginning in 1958. I am still a really, really shitty programmer. I have no idea how to do it anymore and you guys have been wonderful. Hope to see you tonight at Hackard Jeopardy. And if you have any questions or comments, there's a goon with a microphone out there. Oh, thank you. Alright, if there's no questions, again, thank you very much. See you tonight at Jeopardy. I appreciate it all.