 I guess we'll get started. My name is John McNabb. I want to thank everyone for coming. This is my third DEF CON and the best one yet. As you can tell from the slide, I'm speaking about cyberterrorism and the security of the national drinking water infrastructure. A little about me. By the way, I'm not going to read all the words on every slide. You can read them faster than I can say them, and I'm going to use them at jumping off point to, you know, mention other things on the same topic as the slide. But I've been an IT pro for five years. It's my second or third career after being in environmental advocacy for clean water action and being a lobbyist for the Massachusetts Environmental Agency. In both capacities, I did a lot of work in water supply protection. And my main qualification for talking on this topic is 13 years until a few months ago as a water commissioner in a small town in Massachusetts. I've spoken at one conference so far, and I'm speaking at Freakneck in Nashville later in the year. So I'm trying to make, the purpose of the talk is trying to make a realistic assessment of the potential of a terrorist attack cyber or kinetic against the US public water system. Okay, some of you might be wondering what the hell is a water commissioner? If you're from New England, you might have a glimmer of an idea because most New England towns have a fragmented sort of local government where the executive body is a selectman, the legislature's a town meeting, and then there are various other elected or appointed bodies that do various things. And every town is a little different. In my town's case, and I'm going to try not to mention the name of the town so no one there gets mad at me that I pointed a finger at them. But it's an elected position in my town and in many others in New England. Usually three people for a three-year term each rotate, so one person's up every year. So you can see the picture there of me running for election the first time in 1997. And the pattern usually is for these low-level posts, so to speak, as opposed to the selectmen or school committee, which are the really high-interest posts in small towns in New England, is that you get elected and you're basically not going to be opposed again. And that happened in my case. I had a relatively easy race against one opponent in 97 and then have been unimposed ever since. And most other places, and usually in the rest of the country, where you have a mayor and a city council or a town council, the water department would be under the DPW. But the point is here that I was one of three people for 13 years who were the policy-making body, the managerial staff, and the top of the heap for decision-making. Not day to day. I wasn't the IT department. We didn't have an IT department, but I tried to apply my IT knowledge whenever possible when looking at our SCADA system and these questions. The biggest issues that water department commissioners look at is what's the budget? And it's not that easy to get rate increases, and that makes it hard to upgrade equipment a lot of times and to deal with the decaying infrastructure and the new regulations that come into place. So a brief outline. I'm going to go through some definitions. I'm going to talk about threats drinking water from terrorists or other parties. I'm going to describe the national drinking water infrastructure and some issues relating to that that are pertinent to this topic. Now we'll go through the components of a single water system, the notional water system, and I'll discuss the instrumentation in each case, the pros and the cons of an attack vector on that water system. And then I'll tie it all together in talking about SCADA, which has been a lot of talk at DEF CON this year about, which is good. We'll try to wrap it all together with an assessment of what's really the extent of the risk, what's been done so far, and what needs to be done. So definitions, public water system, the EPA defines it as a system that has at least 15 service connections or regularly serves at least 25 individuals. And I'll explain later that's a lot of systems in the US. SCADA is process control instrumentation and software using manufacturing, infrastructure, utilities, et cetera. I realize there are some control systems that aren't technically SCADA, but I'm going to use SCADA as a generic term to talk about process control. We call it SCADA even our little water system and it really only runs in the plant and some remote systems, but it's the same idea. Critical infrastructure, and that's what this talk is really here for, why is this talk at DEF CON is that it's a concern for security of critical infrastructure. And critical infrastructure is a quick definition, it's infrastructure that's essential for the operation of society and the economy. There's other more technical definitions. There's a number of different definitions in the Bioterrorism Act of 2002. There are presidential commissions on critical infrastructure going back to 1998 and a lot of other legislation and regulation that deal with that topic and there'll be more on that later. And cyberterrorism, now there's no except the definition of terrorism that I've been able to find and there's no, you get even more complicated if you try to talk about cyberterrorism and I've heard other speakers say the user word cyber only because it's convenient and I agree with that because it's got a lot of misconceptions. So without going through the dozen or so definitions that I found in my research, it's not as an attack through the cyberspace or the internet or computer that has an effect in the real world that breaks something, whether it's a person or a water supply or shuts something down or starts something up. It's not hacktivism, it's not defacing a web page what some people call cyberterrorism. We're talking about the stuff that affects the real world from an internet connection in most cases. How important is drinking water? You know in your home you need to be able to drink water on a daily basis, you need it to make food, you need it for your toilet to flush and to have sewage and you need it for fire suppression for the hydrant outside the house. But bigger than that it supports almost every other critical infrastructure that there is. And this is a list I took out of a government report on, you know, looking at the interdependencies of infrastructures and which ones are dependent on drinking water. Agriculture, food, public health, I won't read the whole list. That looks like everything to me but if there's anything missing it's probably not that important. On the right hand side I try to look for the flowcharts that look at the cascading failures from one thing failing and I couldn't find any that started with drinking water but they all seem to start with power, of course. So here are two with power and one talks about water and on the top on there the bottom row talks about, you know, power fails, water fails, agriculture fails. That's not necessarily drinking water but irrigation water and what and it causes a further failure in another sector. And in the bottom one it's power supply disruption, water supply, medical facilities, medical treatment. So you can think about it in almost any category. The water supply fails, something else fails, society becomes less effective, there are less services available in other sectors of society. So the bottom line is water supply, we're not talking about water supply for its own sake necessarily but it's a basic requirement for operation of everything. Drinking water has been the target of attacks for thousands of years and my favorite one is Vlad the Impaler, you know, Dracula. But the real Vlad the Impaler did poison his own wells when he was attacked by the Turks. He did a lot of other nasty things too. But there's a long history in warfare but also in terrorism and of water supplies being poisoned. Some more recent but some go back back centuries. So the point is that water supplies are a natural target in war and terrorism. We shouldn't be surprised when someone's threatening it to attack a water supply or actually attacking it. There have been terrorists that have poisoned the US public water supplies and through history there's been some domestic folks who ordered the rising sun in Chicago, Typhoid, the Covenant Arm and sort of the Lord in Arkansas they were threatening to poison water in New York, Chicago and somewhere else. In some cases they were caught before they could do anything and the evaluation was they didn't really know what they were doing and they didn't have enough of the substance to really make a difference. But it was taken seriously and there were arrests and prosecutions there. There were a number of ones about al-Qaeda and modern terror groups threatening us. This is a declassified document from a raid at the Tarnac farms in Afghanistan where if you can read it it's handwritten of one of the al-Qaeda operatives there where they had a training session talking about what sort of poisons to use for poisoning water supplies. So there's realistic evidence supporting the fact that al-Qaeda they've threatened to poison the US water supply and there's a piece of paper showing some planning. Now we're talking about cyber too. In the talk I'm going to bounce between cyber and kinetic or real life attacks because either one they could happen at the same time actually. So looking through the literature and research and what other people have done in this topic it boils down to they've got people by T-skills since after 9-11 the US broke up a lot of their centers and they've been using the internet to coordinate communications the evidence it seems to suggest that they use the internet for the same thing almost everyone else does for communication, organizing, propaganda information, disinformation for managing the organization there's evidence that they use it to conduct cybercrime to raise money but not for cyber terrorism as I defined it I corresponded with Dorothy Denning a well-known authority on the subject who's testified it for congress and she testified 2002-ish that well they do have cyber capabilities for doing these type of things that they don't seem to be actually using it for terrorism to create to attacks in the kinetic world and she emailed me a few months ago saying she still has that same conclusion and I can't find any evidence that they've done that but again we don't know everything they're doing. This is the Cybernet Cafe in Britain where a number of Al-Qaeda fellow travelers were arrested in Britain a while ago that they use for communications there have been some cyber attacks on water systems now this is a short list because there isn't a long list none of these could be called a cyber terrorism but it's interesting if you go through the details the Salt River Project was attacked via modem let me find my notes excuse me this system is the most popular one we're a consultant who was not hired for a job got very upset and he used this laptop to communicate wirelessly with the pump centers at the wastewater treatment facilities and he caused at least 46 releases of untreated sewage into the environment so we have a modem, we have wireless this was Harrisburg, a lot of news coverage an employee came into the plant with a laptop that got infected with spyware that then infected the computers at the water plant so it wasn't anyone actually attacking the water plant but again it was a an intrusion Tehima Kalooza Canal Authority was an employee who installed some improper software and was alleged to have caused damage so that's an employee insider job now Rissy a repository of industrial security incidents which is the authority I guess worldwide for keeping track of these things cyber incidents and water systems and they mean water and wastewater and agriculture it's a broad term have increased 30% but we'll look at the numbers later those still aren't big numbers okay now the good news is if you want to, if some bad guy wants to attack the U.S. water infrastructure it's almost impossible to attack the whole thing because there's 155,000 public water systems they're varied my water plant is unique every water plant is unique they're not cookie cutters for the most part they're individually designed and built depending upon the exact characteristics of the water that they use that they treat their demand the history of the plant where it started a lot of plants were built 30, 40 years ago and they're sort of organically growing from that even if they were all connected you'd have a devil of a time trying to have the same cyber attack affect each one so again I won't read all of this but again the contrast is with the electrical infrastructure down here is that there are three grids a recent study by a Chinese grad student alleges that one or two attacks on attack on one or two nodes in any of the grids can cause a cascading failure in the electrical grid and shut the whole thing down that's what the report said anyway you can't do that in the water systems you shut one plant down and the rest of them aren't affected at all so that's sort of the good news that they're isolated except that there are some water conglomerates the biggest one is American water which is the largest private US water provider which runs or owns facilities or both in 35 states not everything in each of those states so it's 15 million people that's a big chunk of the US population in 1600 communities so I didn't look into their networking but if they're networking through all their or most of their systems that's a potential cyber intrusion that you could get a bigger bang for the buck than going after individual systems that's a question that I I'll just leave out there but it's still a question you still can't attack the whole thing at once or otherwise but that's a way to still create an impact again terrorists don't always have to cause mass casualties they basically the word terrorist means creating fear so they may only have to say that they've done it and have it occur in a couple places United Waters the second biggest with 7 million in aquariums in eastern US is in three states so that's the exception to the fragmentation but again likely that the only mass attack could be through cyber if you can do it but there are water system interdependencies that bring them together as well and the biggest one is treatment chemicals I won't read all the boxes around here but the biggest one is treatment chemicals and you can see that chlorine gas is used by most systems and flip side of the fragmentation of the water industry the water infrastructure is the concentration of chlorine production 38% of it is in Louisiana and a short list of other states so and if we looked at the rail lines if I did a detailed study that looks at the rail lines in Louisiana and some of those plants you could find some nodes where some kinetic attacks could slow down the system or do it or doce it we know that LKDA has considered directly attacking US rail lines other reasons to attack rail lines but this could be one reason as well now I presented this at the American Water Works Association National Security Conference last year and I was a much smaller crowd than this for some reason and I was in a group I was talking on a day after the IT talks about some of the things about SCADA that I'll be talking about later and I was in a group of folks talking about chlorine and I don't know if anybody in the audience got it that hey even this isn't the easiest thing to do it may be impractical but it's there and somebody needs to be thinking about it and I haven't found any evidence that anyone in charge has been looking at this problem because it's not so you've got the concentration of the production facilities meaning you could doce the system you could shut them all down if you attack the plants or the rail lines that might only work for a short period of time because some plants stockpile my little plants stockpile for a month bigger plants though get shipments every day, bigger water plants so that's something it should be thought of but the strangest thought I had which I can't get anybody to confirm that that they're thinking about it is chlorine's already a poison it's used to kill bacteria and to disinfect the water my thought was gee why couldn't you poison the chlorine when the water comes into the plant no one tests the chlorine it's in these cylinders and you put it in a special room so no one can breathe it and won't kill people in the building but you're not sampling it for anything and actually there is a substance that I'll withhold that could be used it's bulky, it's expensive it does exist in the US it is a chemical that is known to be held by Iran and North Korea because it's used to make other weapons of mass destruction so hopefully it's a far-fetched idea but I brought it up at the first Water Security Congress right after 9-11 when their panel of experts and their chemists and whatever and I'm not a chemist I asked them that I said tell me is this crazy and they didn't say it was crazy so I'm worried but I think what they say in Mythbusters is it may be plausible but impractical but again somebody should be thinking about looking into that so types of attacks in general if you're going after individual systems broken down into chemical, biological, physical disruption and disruptions of SCADA honestly the good old fashioned sabotage might work best but again if you're attacking water systems in the US how many are you going to attack you know that needs a large manpower base to do kinetic attacks and do bombings and stuff and Al-Qaeda didn't say they were going to cut down the water they said they're going to poison it and that's much harder to do but these are the options but if you're going to have chemical, biological or radiological contamination it's got to meet four criteria it's got to be weaponized so it's got to be soluble in the water or not soluble in the water depending on what it is it's got to be infectious or toxic from drinking water it's got to be stable meaning it doesn't isn't consumed by being transported in the water and it's going to be chlorine resistant now I'll go through the components of the water system later obviously it's going to be chlorine resistant if it goes in before if it goes like in the reservoir it goes into the plant but it's still going to be chlorine resistant if it goes in later because it's always going to be chlorine in the water too it's going to be some chlorine residual in the water when it gets to your house or at the end of the system to make sure that you don't have bacterial growth in the pipes so there's a short list of CBR that are plausible I'm not going to go into those details as much as other experts on that subject might but it's also going to be a combination attack so for example some of you may have heard in the news that in May there was a boiled water water in the Boston area and what happened was that there was a single point of failure in the 26 inch it was a gigantic water pipe from the quabin reservoir into the Boston system and it just broke and water was spewing hundreds of feet in the air until they were able to stop it and that was the water that was treated water going into the mass water resources authority is a wholesaler and they distributed it to 68 towns in the Boston area so they couldn't get treated water to people but they still had to get water out to the system to keep the for fire suppression and also to run sewer systems so they still ran the water and then the public campaigns that tell people boil the water don't drink the water do this here's where we'll get boiled water and so on so what if an adversary did the same thing and blew up a pipe and then disrupted the communications then people wouldn't know or not as many people would know that they had to boil the water so again there's a lot of scenarios like that where you don't necessarily have to poison the water to cause a problem if it's not chlorinated and it's not treated it's going to cause problems okay so here are the public water system components and I used there's a lot of different diagrams I could get and this one attempts to show some of the instrumentation and the SCADA components in security we've got CIA confidentiality confidentiality, integrity, availability they've got PSA you have to sufficient pressure sufficient pressure of the water it's got to be safe to drink and it's got to be available now the pressure I won't talk about much more but the pressure comes from the pumps at the treatment plant that push the water out at a certain pressure and it's got to be the same as at the top of the water column in the water tanks so pressure can come from either way and of course you lose pressure if you have a break in a pipe which happens it's a routine thing safe to drink of course is the quality of the water in the reservoir or the well and then after it is treated but also how it is affected by the pipes the condition of the pipes isn't something you usually think of for safe to drink but it is and I'll it's not a big topic for this talk but it's a gigantic topic for making sure that the water coming out of the tap is safe to drink and actually good to drink and you want to drink it and it's not cloudy and it's clear and available and demand of course is the key thing it doesn't do any good to do the rest if you can't get it to the people's taps so source of water supply after 9-11 people were scrambling before at water plants anyway water systems before we got any direction and I remember the MWRA the mass water resources authority water resources authority the biggest one in Massachusetts spent millions of dollars with the armed guards and building fences and so on around the Coven Reservoir and around reservoirs and we had similar thoughts in my town and that's my reservoir there actually that's the pond there's another reservoir that we share with the state and we were concerned with immediate what do we do now and so I try to block vehicular access as much as we could to the reservoir but you can't in many cases because they're big then as we got more details and things filtered down from the EPA and the state environmental agency it became clear that it's not impossible to poison a large water supply but it's again very impractical and the example I got in my research is the Dillon Reservoir someone did a calculation of what it would take to contaminate it and it's a lot of it's a lot of contaminant so someone would have to dump you know you'd have to have tanker trucks backing into the place again not impossible but again if you're an adversary you're going to want to do that in 10 or 100 places unlikely so it's good to be vigilant about it but again we're not really worried that this will be the first place someone would attack now on the other hand I've done a lot of work in protecting water supplies for clean water action in the state and in my town there are examples of well fields getting out of commission because they're contaminated from leaking underground storage tanks petroleum etc also nitrates if septic systems are too close together superfund sites there's a water supply I worked with a citizens group in Wilmington mass to deal with a superfund site that knocked out half the town's water supply that were wells so it's not unusual at all for a well field to be knocked out of commission if it's what's called a confined aquifer so it's an underground pond basically with clay on the boundaries say or rock or something relatively impermeable so something spills in it it stays in it and the concentrations go up and you can't drink it anymore that's not that uncommon so well fields are most vulnerable again but it's the same problem with dilution so you've got a superfund site in Wilmington for example for decades was dumping god knows what PCE all sorts of industrial chemicals into the ground and over decades it killed the water supply so again that's difficult for an adversary to do on purpose now instrumentation there's a wide variety of instrumentation well maybe not that wide but depending upon the source in a surface water supply a pond, lake, or reservoir you often have stream flow controls where you try to keep track of the stream flow going in and out of the pond to control the water balance you might in some cases you have a real-time monitoring of water quality although that's fairly rare and in many cases those that feed into the SCADA system so you'd have a route potentially for data or an attack from these instrumentation systems into the SCADA system the biggest sort of impact would be say a dam obviously if you had a dam that was radio controlled or SCADA controlled or some other remote control for it you could open the floodgates in a dam or close it so it would spill over or combine that with a kinetic attack that's an area for causing serious damage okay the water treatment plant that's where the heart of the SCADA system is and I won't go through the entire treatment process but it's you have to intake the water you coagulate it to take out the bigger pieces of solid matter flocculation further takes out the solid matter chemical addition you have to control the pH and some other parameters filtration takes out everything else and chlorination is used for the most part for disinfection but also when New England anyway to oxidize iron and manganese which is a common contaminant in New England and then it goes to the clear well something called the clear well which could be inside the plant could be under the plant could be a tank outside the plant and that's probably the most vulnerable part of the plant because it's after all the processes and this is where the heart of the SCADA system is and again I'm used to a small system with one plant you could be a moderate size system and have four or five treatment plants and maybe have one SCADA system that you know works for all of them now usually they're in a building and the building is alarmed it's got fences, etc but here's a picture of a plant in Florida that I took a few years ago that's those are all those are the tanks that's the processing facility you know it's not not really that well protected it's an island that's has a gate and it's very hard to get to but again it's they don't have much weather there I guess so it's relatively exposed so what could you do with a plant if you were attacking it a kinetic attack I think would boil down of this plain old sabotage you'd blow something up and stop the production and that's the most dangerous thing actually because every plant is individually built and designed and the pumps that create the pressure don't you don't get them off the shelf in my plant it took 18 months to get a replacement and we one go out when we didn't you know we didn't plan on it and we had to run in one pump for like 18 months then we made the water commissioners made a rule that let's buy another one and keep it on stock so that's probably the most likely one if they're going after one one plant if you were if you were going to if you had the capability to do a SCADA attack on it and I'll be getting to that a little more later you could perhaps change the dosing of the chemicals and it's been alleged that the most likely thing to do would be increase the chlorine because too much chlorine in the water can be harmful to health you could also reduce the chlorine and then then shut off the sensors that monitor it going out and that would cause bacterial growth in the in the water which doesn't sound that dangerous but that would cause diarrhea and other gastrointestinal problems which can be fatal so that's poisoning of a type finished water storage again water tanks this is an old tank that we had in my town before we repainted it the high school it was behind the high school and the high school kids wrote a derogatory term about the principle up there in 1965 so that's the racing stripe because we called it for decades a lot of water systems have single points of failure is what a lot of the studies say and from 1965 when this was built to 2000 that was the only water tank in the town and it was such a single point such a problem that we couldn't clean it because when you take down the water tank you have no place to store water and it affects the pressure and everything in the town so it wasn't until we built the second tank that we were able to actually clean out the tank which is an indication of the noise that we have in water supply terrorists would have to do quite a bit to get over the noise where pipes break, water gets dirty, water tanks get contaminated with bacteria there's a lot going on that we don't need terrorists to have problems with water tanks for example Wolburn mass which has a history of again wells being contaminated in the movie Civil Action you've all probably heard about two years ago they had boiled water water because one of their tanks got polluted with bird shit because they had allowed the top of the tank to be corroded and the birds would perch on it and defecate into the water and they didn't find that out until it was too late and they had to boil water water then they had to take that tank out they had more than one tank fortunately drain it, clean it, fix the top put it back in service so there's a lot going on in water systems without terrorists getting in the way yes no, no they were never identified either and so the level of the water in the tank is very important and here's a readout from our SCATUS screen for the water tank level for example now the boil will talk all down this is the most vulnerable part of the system in all the studies and research since 9-11 and before when people looking at this question the bottom line is the most vulnerable part are the pipes and the fire hydrants because it's after treatment in most cases there's no monitoring of the pipes or the water in the pipes there are federal regulations for monitoring certain areas of the pipes for bacteria, chlorine residual and then four times a year for trial of methane and highly acidic acids which are formed by the interaction of chlorine and solid matter in the water but the most likely attack that would cause human casualties from all the studies that I've seen from experts is putting water in through a hydrant and it's usually a targeted attack because if you there were studies showing that if you did it within a half mile of say a targeted building that you'd get a high concentration in nine minutes I don't know the details of that but it's a possibility you could put water in a hydrant with the intention of contaminating all the water in the distribution system and it might work except that again the noise the things that always go wrong it's been a constant problem in my system where valves in intersections you have valves and you should have valves and all the hydrants and in most cases the valves are decades old they haven't been exercised no one knows if they work or if they might be broken closed so it's very hard even for the water operators to know what are the hydraulics of the system so an attacker from outside is not really going to want to know that so if you're reliably trying to poison a community by putting water even in the distribution system it's a good chance it'll work but again you've got a delusion to worry about and you don't really know where the water is going to go so the most likely scenario is that there could be a targeted attack by either pumping water from one building into the system and at least with some assurance that it's going in a certain direction to another building that's your target or in a hydrant now there's 1.8 million miles of water pipes in the US that's an estimate no one really knows for sure there are between 6 and 12 million fire hydrants no one seems to know I checked with the National Association of Fire Hydrants fire was called fire protection association e-mails and checking their websites and I dug down with one analyst and he said well no one knows no one's counting it the point is there's a lot of fire hydrants and even in one community the rule is supposed to have one every thousand feet that isn't usually always followed but in a city there's going to be hundreds in a small town so there's a large potential for those to be used as vectors for contamination now some communities have put in locks on the hydrants which of course is a practical issue with fighting a fire so it's one more thing for the firemen to do when they're racing up to put the fire out so I personally balanced that against the very remote chance of the fire hydrant being used as a vector and not do that the other option is that there are backflow preventers I'm not sure of the term a check valve that you can put in it that prevents you from putting anything into the hydrant but again that's a major cost and the national cost for even 6 million hydrants could be quite high so there is a counter measure but the best counter measure is one that's been recommended by a number of studies which is unlikely to be done nationwide because of the cost and until something really bad happens is having constant water monitoring there are there's technology now for 24-7 monitoring of the quality of the water in the system of course you only test for what you're looking for so you'd have to have a short list of things that you'd be expecting would be the likely contaminant and they could get you with something that isn't classes of chemicals you could look at markers so it's better than what we have now we really have no idea now I was talking about the noise in the system and honestly if you're running a water department terrorism is not the first thing you're worrying about and the only time you do anything is when someone tells you to do it again you got water main breaks not all the time but fairly often discolored water because the buildup of sediment in the pipes or treatment plant breakdown or the tank hasn't been cleaned in a while you got to deal with unpaid water bills aging infrastructure it's a national issue the water infrastructure gets a D minus from a national association for the poor condition of the water pipes the water treatment plants the entire water infrastructure in the country it's in some cases a hundred years old in my part of the country the east and a lot of things are still original equipment maintenance new regulations create costs and require work vandalism is probably more of an issue than terrorism of course and getting more money to do all these things so we only did something in my town after 9-11 we talked about it but we waited until we told what are the three things the ten things you're supposed to do and I'll get to that but it's tough justifying spending money for things like this unless you have an actual event or threat or direction from a government entity now let's talk about SCADA we changed from to SCADA a few years ago from the old control center here electromechanical controls Alan Bradley, PLCs you saw in the earlier slide the SCADA screen for the level of water in the tank on this one it's a pen recorder here that we have to check every day now this is semi-automatic but what happens is when the operator comes in in the morning in our plant we didn't run 24-7 we didn't really need to most places do so we only had it manned when we were operating it which is maybe 8-12 hours a day so they'd come in and they'd set the speed of the water coming in and that would set then they'd have a formula and they'd set the chlorine injection and what the concentration of the chemicals should be so it was a lot of hands-on stuff which meant they were paying attention and it's a small plant so they could walk around and check everything now we went to a SCADA system where we've got a very nice self-explanatory HMI human human machine interface that's like pushing so instead of pushing the buttons up here you see it all here but then you only have to set it once and then everything else happens automatically and it's an ethernet connection throughout the plant but it builds upon the old relays and PLCs in the system through an old relay box the water commissioners liked it because we got more control over the quality of the water we were able to have better assurance that it was being operated properly and we got reports on what happened the bad news is that no one was thinking about what exposure does this give us and we have remote facilities we have a well field we have the two tanks we have a dam and those are all connected electronically either radio or cable we get all of the above or ethernet to the plant so that's a big problem so studies show the McAfee did a study called in the crossfire that came out recently that they surveyed a number of SCADA people with SCADA, not just water throughout the country and 76% said they were connected to an IP network of the internet so there's an exposure there and 47% realized that that created a security issue surprise Team Simru a security research firm produced this heat map in their darknet project they scanned for well receiving scans for SCADA ports and they found a lot of activity and it's a good question who's scanning our SCADA ports and why we don't know there are test bed studies by various researchers that show that external contactors can penetrate systems like most of us here would know and I mentioned the power grade issue ok so RISC which I mentioned earlier the repository of industrial security incidents is showing a general trend of increase in incidents which again are not all intentional but it's instructive that they're increasing and that's where we got the 367% increase for water again the numbers are pretty small 22% were targeted attacks the rest are mostly malware there's been debatable stories but some confirmation that there's been cyber attacks that shut down power systems outside the US and also reports that the US electric grid has been repeatedly penetrated now you've probably heard about stuck net the it just came out July 12th I believe and there's been a lot of news on it I did some research this seems to be the first malware that targeted a SCADA system since most SCADA systems are on windows XP for the most part is advertised by this vendor as being a robust platform for running the SCADA system and that's what we use in my plant but it was particularly targeted for Siemens Sematic, WinCC and PCS-7 SCADA software which is using a wide variety of applications but also water the American Water Works Association put out a bulletin a few days ago alerting their members to this and telling them whatever they knew at the time about what to do Microsoft has a patch which doesn't seem to be doing too much but Siemens has SysClean and I believe Trend Micro has one as well the first guesses were that it was industrial espionage but then some analysts are saying it could also be used to control the systems give the malware attacker the same administrative rights as the user of the system so it looks like the toothpaste is out of the tube there could be copycats for the other SCADA manufacturers this could be the beginning of a bad trend of SCADA specific software now there are studies on the potential impacts of a cyberattack in a public water system SCADA I won't read through the whole list but it's not out of the question if you can get into the SCADA system through the Windows platform or Linux or anything else you can interfere with the operations and cause changes it's not that easy as you'll notice in all my examples earlier I couldn't come up with an actual terrorist or other attack that actually did that but the Siemens Stucknet malware shows that the bad guys are going that direction so trying to wrap up here what's the extent of the cyberkinetic risk to public water systems on the one hand it's too big and exposed to protect but on the other hand the fragmentation makes it hard to attack there are some single points of failure I mentioned I mentioned the crumbling infrastructure this is estimated to be a 350 billion national shortfall and funds needed for that so before you even get to attacks by terrorists or other folks the infrastructure is already in bad shape there's frequent unintentional contamination water main breaks other things so again a terrorist can have to do quite a bit to get through the noise of the problems that occur anyway what's being done to protect public drinking water systems and I listed the government things that have been happening so far there's some research Gate of Hunting net testbeds SANS has been working on methods to use to harden and strengthen the security the bioterrorism act 2002 is the one that has had the largest impact from where I was sitting that required us to do vulnerability assessments and send that to the EPA and then to do a emergency response plan and those had to be done by 2004-2005 and that's it so on my water system we got the vulnerability report we hired a consultant who was in the business of doing that gave us a lot of very good recommendations and we followed some of them and because no one tells us to do you know here's the 10 things you have to do so I'm not aware of any study that says here's the status of every water system in the United States on doing minimum required or recommended security for kinetic or cyber and that's part of the big problem it's local control of water systems and it's a wide variety of security what still needs to be done estimate of 1-1.6 billion to implement the security recommendations of the government but also other bodies the bottom line is since the most vulnerable part of the system is the distribution system is real time monitoring of water in the distribution system don't hold your breath I don't have a cost estimate for that but it's probably not going to happen until it's too late or there's a major attack somewhere but that's your really only security from all the other possible attack vectors is checking the quality of the water as it gets to the customers in my opinion is there should be some EPA required standards and all the post 9-11 changes in governance of DHS and so on the EPA is still left in charge of everything to do with water supplies so they've got to they give strict regulations for quality training of operators blah blah blah for running the plant I think my personal recommendation is they should take say the 21 steps to improve cybersecurity of SCADA networks apply to water systems and require it to be universally adopted they have the authority to do it and as we've seen there's a possibility of exposure if they don't and so conclusion this mostly wraps up everything I've been saying so far water systems are attractive targets they're historically been targets in warfare and by terrorists there have been cyber attacks on water systems there's been cyber incidents and attacks on SCADA systems in general so we have to assume there's a vulnerability there I wrote this before the Stuxnet malware came out so the toothpaste may be out of the tube there may be other ones going on we haven't heard about yet it's possible for a kinetic or cyber attack to degrade water quality or shut down the system or poison the water so this needs to be taken more seriously by the government and it's got to be the government that comes down on the private and public water systems and makes universal requirements until that happens a cyber terrorist or a regular terrorist means can wreak havoc on the water systems thank you