 So yeah, so this is a... Wee! It's exciting. This is War Driving, the smart grid. Talking about a lot of different things about packet radios that relate to utilities. And yeah, this is... Yeah, I'm Nathan Keltner. I work with Sean on our security assessments team. Do penetration testing, stuff like that, as our normal job. Occasionally we get fun engagements like one that sort of led to this talk. I also do some development for the medicine project. Sean gives me a hard time because my hardware hacking skills prior to this project primarily included of modding Xboxes and things along those lines. Yeah, and yeah, I'm a principal consultant in Fishnet's security assessments team. Also, primarily we do network web, physical, social engineering, pen testing and stuff like that. Yeah, I worked at a VCR repair shop in college, so that was my primary qualifications for electrical engineering and such. And I am in fact, provably, remarkably resistant to sustained 110 volt burst. We've proved this numerous times. Now, it doesn't depend on the amperage, so preferably sub 200 amp or so, but yeah. How sustained? How sustained? A few seconds, you know. A lot of the things that we were doing, we were working with live electric meters and other live devices and stuff. A lot of them would have a capacitor on the board that stepped it down to 5 volt and I'm like, oh no, I can JT that. Yeah, it's just right there. That's all 5 volt, you know, and there's this big freaking capacitor right next to it. Oh, not right there. No, that's 110. So yeah, so basically this is about talking about both sort of smart grid and dumb grid radio, which is just sort of all this wireless technology that plugs all of the different types of utility systems together. Not just actually, you know, meters and a lot of people talk a lot about AMI, but this is about sort of all of the things that people are using for the wireless communication channels across the grid. And that also applies to gas, water, lots of other things. Yeah, primarily there's lots of open standards. I mean, this stuff has been around for a long time. You can find documentation from, say, ANSI and some other people. They will sell you the various, you know, similar to like RFCs for these SCADA protocols that will tell you how all this stuff works. But there's a fair amount of, you know, vendor special sauce that kind of gets tacked on all around this and inside this. And so anytime you touch any one of these devices, you always have a mixture of interesting things that you can, you know, look and find anywhere and then random, you know, proprietary stuff from these vendors, primarily in the RF space, which is what we're talking about today. Yeah, and one of the things which is why we want to kind of, you know, point a lot of this stuff out is that these guys are very much still in this model of culturally sort of thinking about infosecond and all of that. It hasn't been as applicable to this particular industry. So they have a lot of the kind of things that you would see in 90s like software security, you know, data sheets and stuff where they describe like proprietary encryption, patented proprietary encryption as an asset, you know, or secret proprietary radio protocols and things like that. And that's actually used as a selling point. You know, you can go out to, you know, AMI vendors and utility vendor sites and see that, you know, as a security asset. And unfortunately, the people in utilities that are buying a lot of this stuff aren't aware that that's, you know, probably not a selling point, you know. So in all of this stuff, you know, a lot of infrastructure that's been there for, right, very long time, for decades. And with the advancement of wireless technology over the last 15 years, it's really become the fastest and easiest way to link up all these remote sites across, say, a city or a state. It's actually really expensive, right, to run lines out to every single device that may be, you know, in a given area. And so what we've seen is that for a lot of actually good reasons for, you know, for overall cost and some of these types of things and ease of deployment, wireless is everywhere. So pretty much anything that you could hook up with a wire has been hooked up with wireless at some utility or another. Yeah, and a lot of this, you know, specifically to where smart grid is going to play is that, you know, and we'll talk about that kind of further in as well, is that there's a lot of rapid push to get a lot of these things out. So there's a lot of things that, you know, traditionally might have been wired or might have just been, you know, something that required physical access to you that they now want to get plugged into everything else and tied in together. And so, yeah, the fastest path to do that is some kind of wireless tech. And so just kind of defining some terms and things like that. You'll hear the term AMI a lot, you know, advanced metering infrastructure. More commonly and actually more pervasive is going to be something called AMR, you know, which is really just meter read type technology, automated meter read. So this is the kind of stuff that most of us probably have, you know, out on our house. And so if you look at your meter and it says sell net or something like that, a lot of those are just one-way reads. And so like in the case of my utility, what they do is they, you know, they just roll trucks around the neighborhood and via pager net stuff will like go out and just pull read data from the meters. And so they're not walking around anymore. The next wave of that obviously and what you're seeing in different parts of the country is being adapted pretty rapidly is more AMI. So then you're talking about two-way communication, you're talking about data coming from the home to the meter and upstream. AMR was what you saw utilities doing say four or five years ago. And that was sort of the natural progression where it was going, where it was the easy way to sort of start stepping into this, you know, more advanced world of digital reads and being able to lessen the number of people you had to actually send out to houses. With the push from say stimulus money, there was a lot of that that was tagged on to smart grid infrastructure. And a lot of that has pushed in various regional markets around the U.S. rapid adoption of advanced metering infrastructure. So like the full scale stuff with all the bells and whistles rather than just the reads. Yeah, and as far as the technologies that are in use, you're going to see a lot of proprietary RF usually on the ISM, Institutional Scientific and Medical Band, which is the same band that you'll see, you know, a lot of other traffic that's kind of like this, but that's the 900 megahertz band. And we got into various discussions with RF engineers and some people that understood why a lot of these decisions were made. And, you know, we like to bash on proprietary protocols from time to time, but outside of the fact that it's closed and we don't really understand always how they work, there's actually were some really good reasons why they did what they did. The type of environment that they were in of, you know, envision you've got to send very, very small amounts of information, right? It's like, you know, a command to do this or that or an acknowledgement or maybe, you know, here's how much electricity I've used or whatever. So very, very small amounts of information that doesn't have to go very often, right? So really what you need is you just need low throughput and you need it to be able to go a long ways and be able to get through, like, brick walls and, you know, all these kinds of awful things. So as a result, most of these guys created their own sort of mesh networking stuff that's all proprietary RF that, you know, didn't exist out there as an open standard so they went to this with a couple of exceptions. Yeah, and a lot of this predates, you know, predates actually smart grid and all these other things. They were around, they were kind of a niche market for some of this kind of stuff before. So they had all this tech that integrated with other types of SCADA systems and all this other stuff. And now those companies are sort of already in that market already equipped, so now they're pushing out this tech. So a lot of the things that we've looked at, you know, on meters and stuff, that's, you know, the actual radio tech in things is sometimes 10 and 15 years old. Yeah, and at the end of the day, right, the whole point of a lot of this stuff is that most of it's all just either an evolution of what was there before or it's literally a bolt-on of what was there before. So at the end, right, you're talking SCADA protocols that are either over IP or over, you know, some sort of custom RF, but it's all still SCADA under the hood and it's usually, you know, you start pulling these devices apart and you can usually find where, like, oh, that's actually a serial connection, like, right there. That's just, you know, I know what that is. I can tap into that. And as far as some of the other tech, you know, obviously some of these things are actually on, you know, currently on nailed wands and so these are actually like, you know, a T1 going, you know, from a substation or a, you know, a plant, you know, back to some central point where they do control of it. And of course that's very expensive, right? A nailed T1 circuit's, you know, say a thousand bucks a month or whatever. But that's what sort of existed, you know, originally and some of these things will be like dial-up lines. What we're seeing more of, you know, lately is actually WiMAX technology. And it's not the same, you know, it's not the same WiMAX that you see with, like, say Clear, you know, typically it's on different frequencies you know, obviously not ClearTex, not designed to be public. It has some of the same problems as other WiMAX technology. The ZigBee stuff, which you'll probably hear, you'll probably hear a fair bit about, you know, ZigBee is essentially a reduced version of 802.11. And then also, like we mentioned earlier, some of these things are on cellular, so EVDO and CDMA. And actually, there are meter vendors out there that, and not just meter actually, but automation vendors and things that support Wi-Fi. So straight-up Wi-Fi, you know. They'll do it. You can plug up. You can plug up a wireless NIC and an AP and send this stuff over that. It shouldn't be a problem at all, I think. It's a fantastic idea. Yeah, and other terms that you'll see, Han and NON down here at the bottom, specifically within the smart grid realm, that's talking about home-area networks and neighborhood-area networks. Home-area networks are the smart grid stuff, that talks to your meter or talks to some other device that'll tell you, you know, you're using this many kilowatt hours and we expect your monthly bill to be blah. And then they can also get messages down from utilities so they can tell you, hey, in the future, when pricing's going to fluctuate more than it does right now, they can tell you, hey, we're anticipating high demand on this day or whatever, so it's going to cost more, right? And the neighborhood-area networks are the stuff that mesh all of that together. So those can be both the things that link up like switching substations and some of those other outside things, but primarily when it's talked about, it's the network that hooks up all the meters in a given area and then pipes that back up to some location. And obviously, backhaul is just, you know, the same thing as your ISP. You might have a bunch of neighborhoods that have, say, cable motives or whatever, and there's a CO or some point where all of those come together to a backhaul. So the neighborhood-area networks, there's some collection point that then runs all that stuff upstream via a little bit bigger pipe. And that's also where some of the actual management traffic and things is going to come out of. Some of that's out of band, some of that's actually in-band, you know, so you're going to see that, you know, if you have access to see meter traffic or other traffic, you'll see the management stuff going across that as well. Like Nathan said, somewhere under the hood here, it typically is something that is really SCADA over IP. So it's at some point you're taking a SCADA protocol, you're turning it into IP, some device, some endpoint is then taking that and turning it back into serial. So it's going to be RS-45 or RS-232 serial under the hood somewhere. Yeah, and just yeah, just a map of the level of adoption and the stuff that we're seeing right now in our part of the this part of the globe. There's the places where you're seeing a lot of the stuff that I, like we mentioned, are in particular a lot of the people that have got the stimulus funding, and that's pushing a lot of that. As I think a lot of people know, Pacific Gas and Electric in California have been the biggest employer of that. But there's lots of other big utilities that are moving this way. So at the end of the day, what do we have? We've got network devices that are talking this RF stuff that then you've given to every single person out there, right? That every person that has this particular technology in their area has these things bolted onto their house and onto random buildings around town and whatnot. So if in that type of scenario you're dealing with a very hard problem to protect, right? You try to think about like some of the people that have had to protect things in that type of space, like I give you a device or I give you a piece of technology and you're not supposed to be able to pull it apart and how hard that is and how well that works, right? Well, in this environment I mean, if you wanted to go like steal one off the side of a house somewhere or off a building and you have physical access to it and you have an extended period of time to work with this, you have access to the network and it really needs to be viewed that way. You've got to expect that at some point, individuals will be able to gain access to that network. It's impossible to be able to adequately block that well. So there's lots of different ways that can be done right? No matter if you've got hardcore encryption and all these other things going on there are too many ways into that type of a system, right? So there's lots of other talks here and previous years talking about how to do that at the hardware level and we referenced Goodspeed, Davis, some of those guys all great work. We make the point a lot of times when we talk to utilities about this that it's a lot like the problem that content providers have with DRM or satellite companies with pay per view and things like that. So these devices have an expected lifetime of 10 and 15 years. And so even if they have like really awesome, super awesome crypto and all these other things, they have the ability to interact with the network by definition. And even if you try to protect some of that, at the end of the day the consumer has the means to join that network and they have access to the device just like with the satellite stuff if one person breaks the encryption and releases the information, right? So if everybody knows I don't know if nobody has a dream box here or does any of that stuff, right? Not at all. If I participate in the sat hacking stuff, I don't have to be the guy who's got the latest codes that work. I just have to go to the forum where somebody else is posting. So the same kind of thing. Once one person in a given area has took apart the protocol or looked at it as soon as other people get a hold of that information potentially, those devices are out there for 10 or 15 years with other people having the means to do that. Yeah, so kind of like what we already said, that a lot of this you get to talk to people and they'll say things like, well but it's proprietary and nobody understands how it works and it gives a very false sense of security. Especially when you're talking to utilities who historically have not really worked in this type of space they don't understand the types of threats that they're running up against. Probably the people in this room who get something like that attached to their house, the first thing you want to do is pull that sucker off and figure out how it works. So it's a very different mindset in a new place for them. There's a real systemic lack of understanding and of really how competent and how aggressive some of these people are going to be. Right? Yeah, and you know on the other side of it related to that too, it's the utilities where they're used to this stuff now is sort of physical fraud. People who steal power. There's always the assumption right that that's what people want. Who's going to do all that trouble to get to not pay 200 bucks a month or something if you have in the summer with your air condition running for electricity? Well it's not really that. If you look back at I think a lot of this stuff as being like the 1960's AT&T world right? It's the guys phone freaks and everything that exploited those networks they weren't necessarily, they didn't care about getting free long distance right? It was about exploring this network and figuring out what it does and this stuff's really fascinating and it does all kinds of nifty stuff so yeah, so people are going to take it apart. I think we're being stopped. Oh, sure. We'll go even faster after this. You good? That's up. Just one year just one year guys. I'd like to not have to stand up here. We made it until two o'clock today which is better than last year. How about next year we do it until Sunday? Somebody, several somebodies are eating a lot of food at Katie's and then not paying their bill. On the order of $100 plus type orders. Where seven people stand up and leave before paying the bill. Not once, not twice several times. Very, very, very uncool. If you know who you are no harm no foul, go pay your bill. You were drunk, you didn't realize what you were doing, whatever. If you know who they are step up put some pressure on them get them to go pay their bill. I don't know about you but I happen to like this hotel and this hotel so far likes us. The Alexis Park is a great hotel. I personally don't really want to go back there. Because I don't like doing CPR on somebody in 110 degree weather. It's not a lot of fun. So please stop being assholes pay your bill and if you walked out without paying your bill go pay your bill. Thank you. And if they don't come forward go over there and give somebody 10 bucks that Katie still adds up to enough. Thank you. So yeah, get back on. You know? Yeah, well, okay, so yeah, so same point actually which is you know what Dynadash is not hacking just like meter theft is not hacking that's just being a dick, right? Yeah, that's theft, it's called being a dick. Same thing, just like the point I was just making is that what the utilities are thinking is going to happen here is that people want to steal power because that's what they're used to dealing with now. And what we think is that it's not going to be about that, it's going to be about getting access to everything inside that network and mapping it out. Just like the guys like Joy Bubbles in the 60s in the phone for days they knew AT&T's network better than AT&T did. It's going to be the same thing. And those individuals may not necessarily be trying to do any harm but they might stumble across something that doesn't do what they expected. See also Mother's Day whatever that was, 98 or 99, right? So yeah, the vendors also are still very new to all the disclosure things and... Yeah, you get into discussions with vendors on this and they've never dealt with people like us before, right? And when they have, it's because usually they've actually brought them in themselves so it's some sort of internal consulting engagement deal. It's very very hard to talk to these people when you are not actually working for them. You know, you try to get on the phone with somebody try to, you know, simple things like trying to ask questions or just clarify on things and, you know, you get on the conference call and then you discover, oh, that's like legal counsel and, you know, some other VP and the engineer that we were trying to talk to is nowhere to be found. And that's a very common thread through, you know, every interaction. Yeah, and we would make a call and ask some little question to somebody in engineering and they would give us an answer and then the next time we'd call with another question they'd say, I'm sorry, I can't you're supposed to sign an NDA, you know, we can't we can't talk to you. And it's very much, yeah, that same kind of mentality that you saw, like the Snowsoft days and the Sue Happy kind of stuff that came out of the software industry when we first started and trying to get them fixed, right? So it's, that makes it pretty tough to talk about this stuff and one of the things that concerns us a lot is that actually as near as we can tell, everybody that we've talked to that's done the stuff that we've been up to has found all the same stuff and none of us can talk to each other. Right, right, at all. We're all under NDAs with separate companies and we all find the same bugs and, you know, you may hear rumors about things over a beer that is actually talking to each other in this industry. And we're fairly confident that actually utilities are all paying people like us and giving those vendors the same bugs, right? And so that's not a very efficient, you know, way to do this. So we talked a little bit about the serial stuff under the hood. One of the big things that, you know, that we find as far as the simplest ways to interact with these networks and something that we think is the most interesting about this is that under the hood, somewhere, whether it's soldered to the board, whether it's under, you know, whether it's literally a board that snaps in, somewhere you're typically talking RS-232 serial. And you talk about smart grid meters, right? Where if you start looking at spec sheets for different meters that utilities can buy, you'll find that each one supports a whole range of types of networking options. So it's all the stuff we listed earlier on that other page. It's either custom RF and what not. And the way, of course, that they're doing that is it's all modular, right? It doesn't make sense to build like 15 different meters just because you want to talk to a different network. So somewhere in there, there is some sort of an interface that, sometimes it's password protected, but it's an interface that's serial. It's not like I can't tap that and just watch that occur. So, you know, it makes these particular devices more interesting and perhaps easier to interact with than some of the other hardware RF stuff that people look at. And there's been a lot of work from you know, say Goodspeed or Josh Wright with some of the ZigBee stuff and some of these other guys that are looking at embedded devices and still seeing the same problems there where, you know, separate chips that are not integrated and so you can tap buses and do things like that. What's even easier, oftentimes, in these cases because it's actually a fully documented, you know, serial protocol with, and you just got to find the right pins, or yeah, it's actually like a snap in. So, yeah, so what we've been able to do, you know, in a couple of different cases is literally just essentially talk to the device that's the radio for, say, a meter as if we're the meter, right, and use that as essentially a network card, you know, to talk to it. So then we don't have to worry about it in encryption, we don't have to worry about any of the other things on the network, we're just, we're there. You know, it's, and they will actually refer to them when you start talking to the engineering guys, they'll call them modems and that's because they really do, they act essentially like a modem. So, you know, to that point, what we think is kind of the future of, to speak to the theft of service side of things is that, you know, rather than, you know, the way that people steal power now is that, you know, you pop a meter out and then jump, you know, the supply side across, and put the meter back in and then essentially no voltage goes to the meter. You know, that's it. You've got three, or, you know, you take the buried cable in your backyard and jump and jump that into your house, which we've heard stories about people doing, is a dig, dig till you hit it and then just tap that and send that to your house. It's probably a good way to run grow lights is what we hear. So, instead of doing that now, right, what you're doing instead is what the utility sees with Smart Grid is the data itself, right, they're not walking to your house, they're not looking at your meter, they're not seeing if the dials are turning, they don't care about any of that, right, that's the whole point is that they're going to pull this data remotely and that's going to populate a database somewhere. So, instead of messing with the actual power, you mess with the measurement of the power, right. So, what we think is the most likely path that a lot of this stuff is going to go down is there are open source stacks in Linux for C-1219, skated, which is the protocol that meters talk under the hood. Basically, you have that and a serial port, you take the card out of your meter and you plug it into your Linux box and then you send whatever data you want to send, you know, that's it. Yeah, and there's, it's more complicated than that, it's not literally just plug and play like that simple, but it is pretty straightforward. I mean, if you look at the traffic coming from there, it's essentially just a number that increments and that's your interval rate. So, you can send that number any way you want, you know. So, that's, yeah, for us, it's the quickest way to get involved in the network. Right, and so, sort of along those lines, maybe controls around any number of these things. In practice, I think, and we talk about this a little bit later as well, but in practice, it seems that many of those controls are not actually turned on. So, one of the things that we've been working through is trying to get, you know, more like configuration hardening or things along those lines where it's like, well, so this thing is probably sending events that say like somebody just unhooked me, right, and that's bad. And that's a sign that somebody is messing with their meter and you might want to send someone out to the house to do something like what we just described. But maybe that message is not actually being replayed up to the utility, or if it is, maybe the utility is not actually monitoring it, right. So, all of those things, of course, have to happen for you to detect that event occurred. Yeah, and that C-12-19 protocol has an entire stack of messages that indicate different things that happen to the meter. You know, this voltage change, and this thing, you know, this thing changed, or this power, you know, this line was unplugged or whatever, but all of those events create more traffic which means buying more relays for that network, which means creating more traffic, you've got to buy more gear and re-scope, you know, how many collection points you have versus how many devices, and so they're kind of off by default, you know, in our experience. Yeah, and one of the probably the most interesting things that we found when we looked at this was when we started actually digging into how utilities are managing those networks at, like, the ground floor. So, you think about how things have been done for a long time. You've got all these texts that are used to being able to have full control of a meter and they understand how it works, and they're able to, you know, go out and actually, like, repair meters and, you know, whatever they need to do. They're used to having a high amount of control of this device. Now we're into this new world where we've got highly advanced things that are a lot of management's being sucked back up into systems that people are interfacing with through, like, web apps and stuff back at, you know, corporate. Well, all these guys they really felt like they needed this ability to be able to interface with these meters and be able to go directly to a meter or go to a neighborhood and be able to talk to these guys. So we started looking at how that works and what that was like. And really what we found was that it appeared that that demand from customers was not necessarily anticipated. So what happened was that providers were taking internal tools that they were using themselves and then sort of retrofitting them and then giving them to a customer. So you've got a field tech who's a guy who's been, you know, a meter technician for 30 years and he's used to being able to go in the field and do all this stuff. And now you go tell him he's got to log into a website and run some commands in his laptop and do all that. He's like, ah, no, no, that's not what we do. So you see from these different vendors a little box, you know, a field tech box that does, you know, disconnects and reconnects and move service around. And by definition, those guys need to be able to talk to like any device that they want to, right? Because they never know what kind of scenario they're going to be hitting. So these little devices are actually fairly dangerous little guys, right? And how they work is interesting. Because it's all the same problems that you see with, you know, hardware and OS and everything from before with like default accounts and that kind of stuff, right? It parallels exactly where you've got these magical God keys that can talk to your entire network. Maybe they can talk to like other customers' networks that just happen to be using the same technology. Maybe there's a, you know, a magical bit you can flip and you can talk to all of them all at once. And, you know, we also found the reason why we think that a lot of these tools were developed in-house and then later turned around and given to customers was that we started actually pulling the software apart. And you find all these extra commands and all these functions that are not hooked up to anything. Like, you know, there's this pretty gooey up front and then all these DLLs in the background that have, you know, a whole wide range of functionality, but not a lot of it wasn't hooked up. So we just kind of said, well, let's hook this back up. Let's see what this does. And then you find all kinds of interesting commands that, you know, nobody is aware of, right? Yeah, and our concern is that basically like, yeah, same thing, fastest path to that network, you know, just like interacting with a network card, the fastest path to that network then is to identify what one of those boxes look like and get it off of the dash of the utility guy that has a truck parked in his front yard. And he doesn't think that it's a big deal. He doesn't know that it has signing keys and all kinds of other scary stuff on it. It's the box that turns meters on. We parallel it a lot with some of the things that have happened in the, like, the GSM or cell phone world where, so there's software from, and I forget who it was. I saw the one of the, like, edge hacking talks here a couple years ago. Yeah, I think Quest is the one that they make a piece of software that'll let you reprogram certain types of G cards, right? So you get the software and it's totally illegal to have. I mean, it's like a proprietary internal software that, like, only their techs are supposed to have. It's only class 1. It's not class 2, I think. So it's not supposed to be out there, but of course it is, right? You can download it, like, on any torrent site out there and then reprogram this stuff and pretend that you're a tech. Right? Well, so in this case you have hardware that also matches that, but it's kind of naive to think of that software that when somebody figures out what you can do with this stuff and that, hey, this is actually kind of valuable and that, oh, there's, you know, I just need to, like, steal a CD in a box off this guy's truck. I mean, so it's entirely a different world. Again, like I said, there were protections built into the software of, you know, functions that were hidden and, you know, there was the user login stuff and all this, but I mean, it's trivial to reverse engineering. I'm not an advanced RE and I did it in about 15 minutes. And so, I mean, it's the same problem of you've got people that are stepping into a very difficult world where they're trying to protect pieces of information when you have full control of it, right? And it's like the, you know, Sony PlayStation problem or something like that, but these guys have been doing it for five years, not, like, 20. So, yeah, so kind of a lot, so we talked about AMI for a while and we're going to just, pretty much going to forget about AMI at this point. So, I kind of have this thing of meters, meters are cute, they're cuddly, you know, they're really super, you know, worms going around, cats and dogs living together, it's bad stuff, right? But at the end of the day, that's not a smart grid. The grid, right, is the whole spiel, right, the whole shooting match. AMI is just the first wave and one of the reasons for that is that it doesn't have a lot of footprint now, right, so that stuff's not wired up. There's a big consumer benefit. There's a lot of things that consumers see out of it and of course, you know, obviously, it's a push to get real-time data, billing data and be able to see that stuff in real-time instead of a dude walking to your house once a month, right? But the longer-term picture here is not about just the AMI side, it's about the actual backside here, distribution automation and energy management systems and all of that. And a lot of the motivation here, you know, the green stuff and all of that has really come about in the last couple of years. The ventures in this space for years have been pushing this to utilities. Prior to that, they didn't call it green energy. What they talked about was that this gave you demand response in real-time and data gathering in real-time. The reason that you do that, if you remember a little company called Enron that kind of sold power short, right, they kind of sold it a little short, oops, and kind of took, say, a state offline for a while, you know, oops. A lot of that comes down to the fact that their data gathering and what they had was at monthly increments. That's how they're predicting this. A lot of you guys probably run networks, right? So imagine if you were trying to deal with what your bandwidth demand would be, but you had, like, no idea how much bandwidth was being used, right? Like, you had some very, very rough, you know, signposts, like once a month or something, you found out that, oh, well, we use this much. So you would buy a whole lot extra and you just hope that you didn't run out, right? So that's kind of where it is right now. But this stuff is going to give you 16 minutes, maybe an hour, but you're going to be able to have very highly specific info on how much energy is being used in a given part of your, you know, your area. Yeah, and so right now, basically the way it works is you either essentially overbuy in capacity or you're hosed. I mean, that's pretty much. Yeah, if you don't overbuy, bad stuff happens, right? So when you can see those curves and you can predict it a lot more, what then happens is that, and yes, you know, at the end of the day, that means electricity because we're going to have better data, but it also means that the utilities trading markets become more like NASDAQ, more like real time. And so that stuff can move around a lot faster, that means profit forever. So that's really where a lot of this stuff came from and the green stuff kind of came after the fact. So yeah, at the end of the day, what that means in order for that to work is that somewhere, power on the grid, decisions about what's going to happen to it are being made in software. And then all that's being implemented over RF down to lots of little devices all over a city or given a state and these little switches that flip on and off and move power around, right? And what we found is that all of that or the majority of that is happening over RF in our experience. And meters are freaking cute. So I mean, that's what, yeah, that was kind of our conclusion. We looked at a bunch of meter stuff for somebody and we're looking at some of that and well, can you look at these, we have probably should talk to you about. So what did they do? And oh, substations, plant control, nothing really important. And the deal with that, and that's, yeah actually I think we have some of this on this next slide, the deal with that is that stuff you don't fuck with. So you take neighborhoods to offline. Do knock on your door if you've got new meters lately and happening to me. No, come back. I'm working on a dock. You can't network off, you know, but it came by and said, hey, can I, you know, I'm just going to take your power off. I'm doing something, you know, or whatever for five minutes and he pulled a meter and popped it back in, right? And then poof, you're on new tech, all new stuff. You can't do that with a recloser. You can't do that with a substation, okay? So what happens is you take the existing stuff that's out there, which is modem lines, which is things that you have to physically walk up to and you bolt something on to it. And yeah, these things are typically they're already networked in some way and now it's about let's get all that stuff up just like the meters and let's get it up in real time and have a real network. Yeah, in a lot of cases that's going to be via something like YMAX, it's going to be cellular and in a lot of cases the vendors that sell the meters are selling the back-end tech too and that's a selling point. It interoperates with your meter tech. It's all the same gear. You've already spent this money on infrastructure to have this blanket network, this fancy mesh stuff everywhere and now, oh, so this is the same thing, right? So I can take advantage of that and I just plug on an extra nick to that and I'm done. Yeah, and in some cases what we've seen that actually mean that there are literally Ethernet drops in boxes and on poles. You have a device that was previously on a nailed circuit over something that mapped out to, you know, there's a router hanging off and there's something that mapped out to, you know, came Ethernet off of there and that's just getting plugged into some wireless device and literally up on a pole or with one of those same cute little locks that you have on your meter outside. Yeah, all these devices that you would use to retrofit old tech like that, they're basically RF with either a serial interface or like you start looking around, you'll see actually like see the little, oh, that's a Cat5 cable on that, like right there. I wonder what happens if I were to tap into that, right? And you talk to people about, oh, so what are the kind of security around these things? And, you know, it's this type of deal and yeah, there's a fence and yeah, there's maybe a camera and they tell you things like, well, yeah, I mean it's in a secure location, but we've got this huge problem, people keep breaking in and stealing all our copper, right? And so you're like, well, okay, so I recognize that there is a fence, but I also recognize that apparently you have a problem with people regularly breaking into this environment and you can't really stop them. So, what does that mean with this stuff, right? Where it literally is a network cable and hey, I could put like a vampire tap on that or I could just unplug it and plug myself in. Where does that drop you? What does that give you? In some cases, it drops you on a subnet with a whole bunch of these other devices, right? Which I'll talk again, Skate Over IP, the unknown protocols that in some cases may or may not have any other controls around the communication. And actually, larger point, if you work for utility, like really simple guys, IPsec is your friend, we use it on the internet, we like it, we've used it for a long time, seems to work pretty good. So, put that on that stuff. And there are vendors that do that. If you're going to take your Ethernet drop or your Skate and everything and hang that over some long range FHSS radio or YMAX whatever the hell it is, put IPsec on it first. And if I break that protocol on the wireless end, I'm still just going to see IPsec traffic. If I get on that network, all I'm going to see is an IPsec endpoint. And yeah, so you've got this, yeah, those are all over the internet and we feel relatively confident about those. They're not fantastic. I'm not going to defend IPsec, but it's better than what we're seeing. Yeah, and just another example, that's a recloser. So these devices, and they usually have actually modems or GPRS or something hanging off from now. These are essentially circuit breakers. So when your load changes you have an outage that say, lightning strike or something else. This is going to essentially just flip a really big stinking breaker and it's going to move power over one direction or the other. So we've spent a lot of time talking about how all this stuff works, how it's put together. We talked about some of the hardware ways to tap into these networks literally on plugging stuff, hooking in tapping on specific places on like a board or something like that. But we'll probably, yeah, we've got 10 minutes left, so we'll go ahead and switch over and just start talking about some of the things that we used to connect to this stuff. So, yeah, ZigBee, we're not going to go into a ton of detail about it. Josh Wright is awesome. We love Josh. We'll buy him beer. The ZigBee stack historically in order to interact with ZigBee you had to buy essentially these developer testbed kits that Digi makes that are called XB. They're relatively expensive, a couple hundred bucks. With Killer B you can buy Atmel cards that are about 40 bucks. You need a JTAG that you can write a custom firmware to, but it's pretty straightforward to do. Basically what that gives you the ability to do then is the same kind of thing you do with wireless where you can do injection, you can sniff, you can monitor traffic and things, and so this gives you the ability to interact with ZigBee gear. A fair bit. It eats up. With specific application to Smart Grid, it does not support the set stack, so there's a I don't know how much you know about ZigBee just real quick, it has these different profiles that kind of block into it, and it hooks down lower and it sets certain configuration parameters stuff like that. There's those set for smart energy. His stack does not actually talk smart energy, so you are somewhat limited in what you can do with his stack. You can also go out and buy like a stack from XB or some of those guys for like $200 or $300, and then you have a full SDK that will allow you to build applications that will interact with that network. Yeah, and actually there's a like we said, ZigBee has kind of reduced data to 11. We're all used to layer 7 model. So ZigBee is kind of, you've got a real basic network layer that's like a much simpler network stack than what you would see and say something that's full 802.11 or like wired Ethernet or something. The larger point about ZigBee is we talk about ZigBee itself a lot as being sort of intrinsically bad. It's not fantastic. Don't get me wrong. But SEP, the smart energy profile that the utilities use does add a lot of extra security bits on top of it. So there's been problems with ZigBee in the past with say like over the air key retrieval and stuff where when they first hook up it'll actually transmit its encryption key, clear text. Well, as an example, SEP says no, no, no. You can't do that. Like you were not allowed to do that in this environment. And so that key management stuff all has to happen outside, you know, in out of band process. So that's one example of ways that it actually locks down the sort of default, you know, ZigBee protocol. And so yeah, overall part with ZigBee is just that we actually kind of like it. As far as, you know, comparing it to what your other options are, it's been looked at by a lot of really smart people. It's by no means fantastic, but if we're grading on a curve compared to the other stuff we see, it's not so bad. So yeah, something else that we talked a lot about this, something that and I know that there's somebody else that's doing some talks about talk about some of these things. The FHS as frequency hopping spread spectrum is what those substations and reclosures and things, what we see a lot of that stuff running on. This is a technology from World War Two that was based on the concept that back then radios were difficult to tune. You had to solder a new crystal on, you know, essentially to get wider bands and stuff. So it's meant for some other, there's some other good reasons and things like interference with radar and all this other stuff that you want to hop channels and it does give you longer range and a lot of other good stuff. But the security argument for it is that you have to be able to follow the hops on an FHSS and it's funny if you go actually to the Wikipedia entry for FHSS, you'll see a bullet point that says FHSS provides security like it's a selling point of FHSS and that it's hard for you to do narrow band interception and like listen and know what channel it's going to be on. All of you go edit that entry and get back to yours. Exactly, but yeah, so how do you fix that right? Well just do wide band interception right? Listen on a band that's wide enough that you can see every single channel and then it's irrelevant where it's going, right? And you can watch that long enough and you can figure out the pattern and most of these guys, their algorithms will repeat themselves so then you can just learn what the pattern is and then you know what that pattern is always for that device that's talking to this other device. And the wide band interception capability didn't really come about until pretty recently for that to be easily accessible to people. That's where the USRP, anybody went to Chris Badges talk, knows all about that stuff. The USRP gives you a much wider band that you can sample off of. So previous stuff that people have done against FHSS with that device, with the older version of the device had to like watch one side and then the other side with the USRP too, you can pull enough that we can see all the samples. We're pretty much out of time. We won't say a whole lot about the USRP Organi Radio which is the API to write to that picture. I would say Organi Radio it's got a fantastic API that'll let you do a whole bunch of stuff. A fantastic get horribly documented. Yeah, and don't ask any questions on IRC or on the mailing list or you just get slapped. Yeah, they'll just beat you if you don't have a Ph.D. in W.E. Did you not go read the docs? Like the big W.E. They'll always send you to, and we have the book now, but there's like this 600 page book on signals design and stuff. It's like, I guess apparently grad level courses in W.E. that's what you read. And so any question you ask in New Radio, they just point you to that book and say go buy that thing. But yeah, it's really neat stuff. Just like we talked about, what we found with things like FHSS is that by doing sampling in a wide range capture we would be able to see those ranges. Something else that we found and we'll actually talk about this again in a minute is looking at things like FCC documentation. A lot of times those docs will have all the detail about how these radios work. They'll talk about the things. In a lot of cases they'll even put the documentation there. Also see that in patents. You'll see it in IC searches. We also found that like finding all the resumes for all the engineering teams for a company would tell us a lot about, like if you look at their background it's like okay, he's got a lot of background in OFDM design. This guy did his PhD thesis about, you know, four GFS we probably know what kind of modulation he did when that company hired him. So that helps us a lot in looking at this and just a random example. This is a Digi radio. It's a serial radio that is in use in a number of places for essentially giving the ability to take a serial port, send it over wireless, really dumb, stupid radio. And if you pull the FCC ID on that, it will tell you all the hop sequences, the channel spacing, the device does lots of other stuff about it. Yeah, it tells you the modulation, the channels, the dwell time, and actually the modulation type that they use. The other thing that... What we're shown here, after you pop off the shielding you can start tracing and looking up what the different chips are doing. This particular chip is the chip that handles basically radio for this guy. We looked at some others that had actually highly detailed data sheets that were available that will tell you explicitly what its options are. It supports this modulation and this one has a lot of options for those. It drastically limits down the number of things you've got to brute force through to do this. Really ease, probably look at this stuff as they put their finger up and they're like, oh, that's GFSK, right? We can't really do that. We cheat. Look up the documentation. It makes it really easy. The guy that's done all the Bluetooth work with breaking Bluetooth, he can actually listen to Bluetooth over audio and hear it and go, oh, I know what that is. They're transferring an image. We're not nearly that smart. So for us, though, these data sheets, the IC data sheets, they're meant for people to buy this chip and they'll tell you essentially the reference spec for this for this chip, for how to write these voltages, do these things on it. That narrows down the number of things that we need to figure out on our own very quickly. We found that to be extremely helpful. Here's just an example of actually taking a wideband sample that was over a large range of frequency hopping. Yeah, so the green line at the top is that's the highest point that it's seen on that channel, right? So each one of those spikes is a different channel within that range and it's hopping between them. The blue line is what it actually is currently being looked at. So the top line is that. It's showing the entire range and what's occurring. The bottom section is showing we're looking at this individual channel right here and then we're doing demodulation based on we looked up from data sheets and we're able to get that out and then all those things actually correspond to ones and zeros in one way or another. There'll be decoding and encoding that might need to take place after this, but basically yeah, that's ones and zeros that you're watching right there. Just as if some of your hardware guys recognize things like that from logic analyzers and things that are happening at that level. Same stuff just over radio with different types of modulation. Yeah, and that's just an example of an individual channel and what in this case FSK might look like. But yeah, the differences in those waves that's just how you get to zeros and ones. If it's just clear checks and there's nothing else in play the mojo here is just getting to the point of figuring out how they got to zeros and ones and building that in software on your end to get to it. So yeah, same process that people have done with Bluetooth and GSM. The suck part about proprietary protocols is you have to do this every time. You've got to go through that process with each radio. So every time you look at a new radio, you've got to go through the same process. We've gotten better at it in what we've done in two weeks, what took us six months before. But it's still a big process. And there's lots of other things. You've got to identify the encoding, the symbol rate, the endiness. A lot of that will come out of the biggest thing we find is using the ICs and the FCC data and stuff like that helps us a lot of time brute forcing the signal until something comes out that looks accurate. And we're way out of time. We had an hour of 15 at Black Hat, so I'm terribly sorry. Yeah, the only thing actually one thing I will say is the other thing we find is when we get under the hood on a lot of these radios, they have something like an SSID for a wireless device. Same kind of thing. They're sort of a network ID. What we've found in a lot of cases is there might also be some undocumented things that will do things like a promiscuous mode. Same thing as a wireless device. There's a channel that will allow you to interact with all other channels. And that would apply to meters or these other kind of radios. So when you get physically a hold of one, sometimes you don't even have to break the protocol. If you use that card and maybe look for some things that aren't on by default, it'll help you out a lot. So we're going to go, what room are we going to be in after this? Capri 112. So you have questions, come see us there, but we've got to get off the stage.