 I'm Raven Alder and I'm here to talk to you guys about denial of service and distributed denial of service attacks and a handy little pearl script that I've written to track them through major ISP backbones. Now I know most of you are probably pretty familiar with this but just a quick background on denial of service and what we normally see to let you know what the problem is with ISPs and why a lot of them don't slash can't address it. A lot of denial of service attacks that we're seeing are... Big pardon? Can you hear? Okay. Better? All right. There are two major types of denial of service attacks that we see at my job. I work for a major ISP. We'll see denial of service attacks where they're just coming from maybe one or two clients with the source IP address is spoofed or we'll see denial of service attacks that are coming from like 200 clients with legitimate IP addresses. There was actually a script called DOS Tracker that someone at MCI wrote a couple of years ago that tracked the distributed denial of service attacks but it only tracked one IP address at a time so you'd have to run it repeatedly in order to get anything approaching and tracking for modern DDoS attacks. Now the script that I've written is called Eyeball and you can find it on sourceforge.net slash project slash eyeball and it's capable of tracking many IPs. It's also capable of tracking spoofed addresses. Currently it only works on Cisco platforms. It does work okay on GSRs. We hope to add support for Juniper and Zebra routers in the near future. www.sourceforge.net slash project slash eyeball and the script as you can happily see is open source so you're totally free to take the source code, modify it, change it. If you find any bugs or problems please let me know. You can also send email to me at ravengeek at users.sourceforge.net. Now the problem that most ISPs have in dealing with denial of service attacks is that actually tracking down anything with a spoofed address takes absolute ages. You have to log into each router in the path so you usually start at the router closest to the customer being attacked. You can log in, throw an access list on there on the Cisco platform with the log input keyword which will log where the actual source interface is of the traffic coming into the router and then track it back. Most links are point to point so you don't really have a lot of problems figuring out what's on the other side of that link. Now doing this manually takes hours. Doing this manually when you have 200 source addresses most ISPs won't even bother and the script obviously is not going to change that. What it can do is automate the process and give you a list of here are the IPs at which this attack traffic entered your network. Licensing for the script we covered already. It's GPL. Have at it. Also a quick thank you to Ben Stern, Tiffany Mork, Steve Ains and most of all my husband Raven Black for much help in getting this up and running and tested and all that. In order to run this script, it's a pearl script you obviously need a pearl environment wherever you're set up. You will have to set up little access files with your username and passwords. On Cisco's, you usually require a regular password to just log on via Telnet, your VTY passwords and an enable password or enable secret. The file that you'll set up can handle attack X or things where you have an actual username whether it's just attack X for the login and then regular local database passwords for the enable or whether it's attack X the whole way. If you have your router set up so that you can have more than one authentication method, i.e. attack X server and then fall back to local bases, you just run the little script to set up your file twice and then append the second onto there. Bundled with the package which you can get on SourceForge is a script called router access in code. You invoke that with output, the little angle bracket output, towards a file called router access dash default, dot router access dash default. It will prompt you for your username, your login password and your enable password. This uses the net telnet Cisco module in Perl which will intelligently handle logins. So if you give the username and the router happens to be configured to only authenticate from a local database and doesn't use usernames, then it won't return the username for the password or anything like that. It should work just fine. If you have a backup method, run the script twice but the second time you run the script, append the output to router access dash default and that will put in, if it fails on the first authentication, try again with the second set of username and passwords. This should handle radius, I haven't tried it with that, but it works okay on attack X and local passwords. All right. Now, once you've set up this script there, the router access default file has to be located in your home directory to actually work. All these files that eyeball uses should be located in the same directory but that doesn't matter whether it's your home directory or not. The only thing that has to be in your home directory is the dot router access default. Also, the encryption that's done on the dot router access default is really, really weak. It's almost like the, you know, ROT 13 or something. So it's only there to prevent a casual observer who happens to, you know, look at your console or something from knowing your passwords. Do not store this with permission so that anyone else can read it because they can decrypt it with a napkin and a pen. All right. So, after you have created the proper router access default file, you will also have to set an environment variable for the eyeball access list. Now, the way that this script works is it will go into each router and your network and create an access list with the same number. The way that Cisco does access lists, they have their access list grouped by number and the number represents the type of access list it is. So anything between 101 and 199 is going to be an extended IP access list. That's what you want to use. You have to make sure that this access list is not used on any other router in your network. Otherwise, when the script logs in tries to create the access list, depending on the version of iOS and the bugs they're in, it may overwrite, it may change, it may just totally delete your other access list and screw up your routing, otherwise. So, bad idea. The script, if you do not have this set, will actually kick up an error message and say you need to set an environment variable for eyeball access list. So, just find an unused number between 101 and 199. You're all set. Also, one of the things that was originally somewhat challenging is the router needs to know what is your network. Assuming that you're a ISP or a provider, otherwise, you don't want the script attempting to log into the routers of your peers, et cetera, et cetera. So, just crawling through the network and going interface by interface until your passwords fail is not really a good idea and tends to annoy people. So, you have to create a list of IP addresses or of host names that you own, essentially, that the script is allowed to attempt to log into. And we've also included here a script that will create that list for you. Most ISPs that I know have a list of their routers so that when they want to make some sort of policy change, they can just go through a script to change to all the routers. So, the eyeball make interface list .pl script will take care of that for you. It takes the list of your routers as input and you have to direct the output to a file called eyeball valid routers .txt. If you are constantly adding things to your network and removing things from your network, you may want to cron this script so you run it once a day, at night, during your off peak hours whenever so that you always have a current list of IP addresses that your routers, that the script can log into for your routers. Milderrouter list .txt there is just whatever you call your input list. All right. So, once you've run these two scripts, you'll have everything to set up, set the environment variable for your eyeball access list. Oh, one other thing that should be set, the eyeball wait time variable. It's set to default to 10 seconds and it's just how long that access list will remain on the router before being pulled off. So, how long it'll be there trying to log the attack packets. 10 seconds is a default but if your routers run any sort of fast switching, Cisco express forwarding, things like that, we have seen problems with Cisco's implementation because fast switch packets do not get written into the system log. So, if you're dealing with an attack where it's like one packet a minute or something, then you'll want to expand the eyeball wait time variable to leave the list on there for maybe two minutes or something to catch each packet. Now, this is going to slow down the run of the script because every router you log into, you'll have to wait that much time for it to catch the packets as well as all the logging in, shooting through the output time, et cetera. There are two more debug variables that you can set if you want to actually see output. Eyeball, Ultra Debug and Eyeball Insane Debug. Eyeball Ultra Debug, if you just set it to one, it will turn on debugging output so that you can see what the commands you put into each router are. You can also see some helpful comments that I've thrown in the script. Eyeball Insane Debug will turn on what the router sends back to you. That's a whole lot of output. It's like a show log of each router that you go through. You'll see the show ins and the show frame, or show frame of the map, et cetera, et cetera. Lots and lots of stuff. So that pretty much covers the prerequisites. Oh, also, on your routers for this to work. Most people do, but you will have to have logging, buffered debugging, or logging, buffered informational turned on so that there is a local syslog on the router for the script to write to. If there isn't, it does not go out to your syslog and then parse through the logs or any such things. All right. Using the script itself, it takes as command line arguments the router that you want to start tracking on, which will almost always be the router right next to the customer being attacked. The IP address that is being attacked. Currently, the script only supports one IP address being attacked at a time. It could expand that in the future. And you can also specify the type of attack traffic. IP, TCP, ICMP, UDP, what have you. We will probably be including specifying the port numbers in future, but the script doesn't do that at the moment. IP is the default type of attack traffic if you do not specify this. This has caused problems in some implementations if the customer is being ping flooded and you just look for IP traffic. Oh, thank you. So if it's a ping flood attack, make sure you set it to ICMP. All right. While the script is running, you should be seeing messages printed to your display, letting you know which routers that the script is logging into, what it's seeing in the output, whether it's found packets or not, and what the routers that it's going through. The script is multi-threaded. So if you are coming in through several different places off of one router, it will branch at that router and start running the script, or rather, start logging into the several routers from each individual point. Oh, and blame for this is mine. So comments, questions, suggestions, bug reports, etc. Send to ravengeek at usersourceforge.net. All right. Let's get to the script. It uses a module called Router Access which is included with the distribution of the script that, and all that Router Access does is throws the login functions on there. You'll see them called a bit later. It will take the router they attacked to address in the packet type as arguments and you can specify the wait time as an environment variable. It does not check from the command line. You'll get an error message there if you don't have iBallAccessList set as an environment variable. You can see we have a function here to actually throw on, or rather, a array there to throw on the access lists on the router and to remove it. Simple Cisco commands that you guys are probably really familiar with. You'll notice the host keyword in the access list. This is because it only goes through one attacked IP address at a time. If you have several addresses being attacked, then I would suggest running it, at least until we put support in for that, running it once for each IP being attacked. The script will output to iBallOutput.txt in the directory that you have invoked it from with all your iBall stuff in it. I would move that after I have done each individual address being attacked so it doesn't overwrite append any such thing. All right. We also see the array here to actually take commands off the router. Now, when it first tries to login to a router, it will print an error message to the console if it actually can't login to the router. Otherwise, we'll give you I've logged in. It took this many seconds to this particular router. It uses the showver command to distinguish between a regular Cisco router and a Cisco GIG switch router because the login format and a couple of other things are different on the GIG switch routers. If you actually look at the show log on the output for a GIG switch router, you will see at the beginning of the line slot 7 and then later on pause 0 for pause 7-0 on a line card. So showver identifies the difference between a GIG switch router and a regular one. I believe that I'm not real savvy with junipers but I've been told that showver will also work on a juniper so when we start including those in we'll probably also differentiate being junipers that way. So, now in order to determine which way the switch interface the attack traffic is going out it will do a showip route on the address that's being attacked. If this is not something directly connected, then it will just do a showip route on the next hop repeatedly until it gets to something that is directly connected. This is because when we put the access list on the interface we're going to apply the access list outbound on the interface that is facing the router being attacked. We can use the login put command on there to log where it's coming in from. This gives you a relatively foolproof way of tracing spoofed packets. The script does treat every packet that it looks for as a spoofed packet. So it doesn't attempt to determine whether the packet header is spoofed or not. It will just do a showip route and log input on that interface regardless. Here we see the access list actually being applied IP access group, the access list number to the interface. It will sleep for whatever the wait time is set to which is whatever you set your eyeball wait time environment variable to defaults to 10 otherwise. One thing that you may want to look at, if you do not have fast switching available if you have a packet sniffer anywhere near the guy being attacked or you can just throw an access list that permits IP NANI or permits ICMP NANI and logs, try and diagnose before you actually run eyeball the type of traffic that's coming through and how often you're seeing it. It won't really help you very much to use the default wait time of 10 seconds if you're only getting a packet a minute. You've got a 1 in 6 chance of your 10 seconds being that 10 seconds. One bug that we have encountered what you should be seeing in the show log of your router, after you run the script in each router that the script passes through, you will see three configured from whatever interface you've come in on whether it's console or VTY or what have you by username if you have TACX with the username enabled three separate times. Now the way that the script works is it will log into the router, create the access list, get out of global config mode, go back into global config mode, apply the access list to the interface with access group whatever out, wait for the eyeball wait time, remove the access list, get out of global config mode, and then later gets back into global config mode to remove the access list from the router entirely and it gets out again. So you should see configured from console by raven, etc. And the packets that are logged should be after the second iteration of that. There seems to be some problem with the timing of this by Cisco and I've talked to their TACC about this but there are a good number of times where you'll see configured from console, configured from console, configured from console, anything that happens in the next minute or so and then all the logged packets. So it appears that the packets are being logged after the access list has not only been removed from the interface but has been removed from the router entirely, it's kind of strange. So we put an extra little sleep function in there so that if the access list is removed you still wait a bit before you actually check the logs to see if anything happens. If you have this problem recurring then check your version of iOS, it was happening a good bit in 12016 but seems to be fixed in 12016st. You may also want to talk to Cisco about it where the script exists. Here's the log parsing bit. The sex6ip access you should see in anything that is reporting the logged packets. You'll see the access list number there and the interface that it came in on and pardon. You will also see the interface so if you have a ATM interface then you may see VPI, VCI information. If you have a framerilla interface you'll see Dulces, things like that. All that gets pulled by the script. The dreaded error message that you see all too often if you have fast switching and so all packets are not being logged. No packets seen in router. If this happens the script will just stop at that router and not attempt to search anything further upstream, it would know where to look. The only fix that I have found so far for that if the fast switching is enabled and so all packets aren't being logged is you can try and run the script again or you can increase the wait time. Sometimes that helps. It will then do a lookup on the correspondence between the IP address and the ATM or framerilla address is there. It defaults to getting all sorts of layer 2 information regardless of what is actually on the router. It's a lot easier than actually trying to determine this is an ATM interface. We'll treat it like that. It just gets all the information. If you see lines in your output of show ATM map and there's an Erroneous command because you haven't got any ATM interfaces on that particular router that's okay. It's in the script. That's supposed to happen. It will then log out of the router and move on to the next one. I've done some testing of this script and it seems to work fairly well, but the fast switching is the biggest problem that we have. If any of you got... It has to be run by someone with enable access to the backbone obviously because you need the enable scripts to be put into the input file. This means that end users cannot run this script. Pretty much only ISPs are capable of doing that. If any of you feel like giving it a try, please let me know how it works, what happens, etc, etc. I have tested this on Linux platforms and on Solaris platforms. Everything works okay on both of those as long as you have the proper Perl modules installed. I haven't tried it under the Windows implementation of Perl. I'm not much of a Windows chick, but it should in theory work. Any questions? You mean if your customer is multi-homed from one particular router? If the customer has multiple links into your core, then you should run eyeball twice or for as many times as each link starting at each router that they have a link to you on. If they have two T1s out of one router in, you can start it on that one router and it will be fine. If they have two T1s into different routers, run it once starting at each of those different routers and it should pick up everything. The output in the log will tell you that packets are going to the customer's IP address. You don't give it the customer's MAC address or anything like that. If you have multiple routes to the customer from your one particular network, it doesn't matter which one. It will just do show IP route and determine the interface there. BGP gets kind of strange there because if you're using BGP, which like most everyone on the internet is, then BGP will only see one quote-unquote best path but that's the path that the traffic would be taking. I don't think it's a huge problem there. Any other questions? Yes. Obviously, you shouldn't have this sitting around on your home machine sending things totally unencrypted telnet over the internet. Cisco unfortunately doesn't support SecureShell yet but it's very important to protect the file that has your the router access default, etc. You should set it so that it's permissions if you're on a Unix system are only readable to you. It is a concern of mine, yeah. As far as the entire network is concerned, BGP is not actually saved in the eyeball valid routers. It's just a list of IP addresses. It doesn't tell you what connects to what. It just says, these are the IP addresses that I am allowed to try and connect to. No, I haven't had that come up. The question was, have you run into problems where you are unable to remove the access list in the same telnet session? It may be in iOS specific thing but it seems like if it actually manages to log into the router, it can throw traffic. The biggest problem I've seen is when it doesn't actually log any packets because of the fast switching stuff but I've never actually had it die like mid-session. If it does though, one of the things that would be fairly helpful is with the output that it puts to the screen you can see which routers it's logging into and you will have one particular access list that you would only use for this. That's why it needs the eyeball access list variable. So you could, at worst comes to worst, go back through the routers that's output on the screen and make sure that those IP access lists have been removed from the routers. Now I haven't seen that the output, this is console on the Cisco or console on your Unix box on the Cisco because the script is run from a Unix box and just does the telnet into each so it doesn't display the entire output on your console. Hopefully all it would do is if you have a particular length of your log buffer on the router then it can just it would, worst comes to worst, it gets wrapped. Is that what you're saying freezes up your TTY? No. No terminal monitoring at all. You just do the show log, you parse whatever's in the log so if it's overwritten its log buffer 18 times you're only going to get that last screen of information. So in case of like a truly massive DDoS attack or something like that then you probably have to take that list that you got of the attacking addresses, try and deal with those whether calling up your ISPs and you're peering ISPs and telling them about them shutting off those that are your customers if you are the ISP etc. and then running the script again to get the next lump of them because it's only going to pull what's in the sys log or what's in the buffered log. Yeah, if it gets to the point that it's killing your router then the script will not, will probably aid in your router's death slightly but you're already killed. We did try running it on some routers that were short on memory and indeed in one of the initial sketches out for the layout of the script I was going to have it do like a couple of show runs, pull out secondary IP addresses for the interfaces so that if you had an ethernet interface with two IPs assigned to it it would include both of those in the allowed list as it is now it will only include whatever is in a show IP interface brief so that's only the primary IP address on any interface but it will get IPs of subinterfaces and such. But we have a couple of routers in the network that I was testing on that are very short on memory and so avoiding doing a show run was generally a good idea so as to avoid causing malach failures. Yes, it does do the ARP addresses that's why it pulls the show frame relay map etc etc to get the ARP and then when it goes through the logging it pulls both the interface number and the interface type so it will get DILCI, VPI, VCI what have you. I haven't actually tried it on any multi-point links it works fine pulling the information I get the information there but the network that I tested it on only has point to points in its core so if anyone has multi-point links you know some sort of funky ATM setup or what have you I would love to hear how it works. Yes, I don't have it with me but it's really simple looking it's just a list of IP addresses and the interface on your router where it came from so new line separated text file yes good traffic and bad traffic yeah that's all in the creation of the initial access list which is why I was saying if you can possibly try and figure out what type of attack traffic they're sending and that's one of the reasons why I want to include a port filtering in the future the access list that's created is just a two-line access list the first one will be permit traffic of the traffic type that you've specified so if you said TCP then it will only look for TCP etc then it will look for coming from anywhere to the attacked address and it will log the input so yeah this does mean that if you have a server that is being attacked on its web port and legitimate users trying to FTP to it then you are going to get those legitimate users FTP traffic reported in there so the port filtering will be out in version 0.4 anyone else yes well I haven't seen like any performance problems with it it seems to work okay but it hasn't been extensively tested yet so okay okay so if the router is one of our routers then it will open up a temp file if it can't open the temp file dies with an error message and it will go through and it will fork and create children processes for each individual router and and it will just create a separate little session for each one of those it will wait to actually close out the whole process it counts how many children processes it has at any given point and it will wait for that to close out the whole thing but essentially I am not the queen of multithreading this is essentially the only thing I know more about routers than I do about Pearl so if you have a suggestion to approve efficiency or technical performance then send it to me an email anyone else yes well I've tested it on routers that have interfaces up to OC48 speeds that were at about the OC48s were on 20% utilization but on the same routers we also had OC3s that were at like 80 or 90% utilization and it didn't seem to cause a noticeable performance lag on the routers or anything thank you as far as attack traffic volumes I haven't seen anything above about I didn't test it with high level traffic volumes with more than about 200 attacking addresses in the last like couple of versions of the script simply because the ISP that I work for has not had an attack of that level going through it in the later versions of the script it seemed to work alright in earlier versions but a good number of things have been changed I've gone up to about 10-15 attacking hosts sending about 100 megs of traffic or so to the attacked address and the script handled that okay other questions alright we're done early