 This meeting was held in exciting Las Vegas, Nevada from July 9th through the 11th, 1999. This is video tape number 6, Firewall appliance brand number 4. I'm Deramo. I'm from Columbus, Ohio, 614, and I'm the alpha doll to the wolf pack up there. We just completed a comprehensive review. It took about six and a half months of some 23 appliance firewalls. And we're going to talk about that today, some of the things that we found. If you were at Black Hat, you covered, we covered what we found as successes and failures. Today, we're going to talk about a little bit more about the failures and some of the things that we found kind of hard to believe. So go ahead and next slide. I guess right off the bat we should talk about what is an appliance firewall. When we started this whole process, we went out to the web and just did a search on the keywords appliance and firewall. We came up with several hundred products and what we immediately determined was that our definition of an appliance firewall and vendors definitions of appliance firewalls very greatly. So we're going to talk a little bit about what we kind of thought about an appliance firewall. We'll talk about some of the technologies that these products used. And we'll talk about how you in the field can know these devices when you see them and why that's advantageous to you. And theoretically, we'll talk about what weaknesses they possibly might have. And yeah, the lights would be good. You can, yeah, that's great. That's even better. Let there be like fantastic ambiance, ambiance. And following the weaknesses part, we'll talk about how you can test yours or others firewalls and where you can find out more about these types of appliances. Next slide. First of all, we talked about what appliance firewalls are. And we were looking for devices that were integrated hardware solutions. We wanted all the software, including the OS, to be preloaded on the platform. Early in this process, we came up with a term that we borrowed from the MS folks again. We kind of took on the plug and pray attitude with these firewalls. We were looking for something that was easily deployable. We didn't want to have to hold hands to a bunch of folks who weren't skilled administrators in the field once these were deployed. And we were looking for something that could provide dual home network access. That's pretty much where we started with this whole process. We weren't really looking for something to firewall off a modem connection or something like that. We were looking for something with dedicated network access. And immediately what we discovered was most of these, the majority of these products, the 23 that we actually reviewed, and there were about 34 to 37 that we tried to review. Quite frankly, when we told some of the vendors that we were doing the review for Defcon, they just kind of laughed and didn't want to give us any products to review. So you guys might kind of carry that around with you. Maybe they're a little afraid of you. You can drop that in some interviews and stuff. There won't be any names, unfortunately. Attorneys have been already knocking. Yes, there will be a white paper available, and I'll talk more about that at the end. Okay, the very first thing we discovered is the majority of these little devices are based upon packet filtering technologies. And they were very simple access control lists based. They simply compared the source and destination IP address as well as what protocol you were speaking, and then decided whether or not to let it pass through the firewall and into the network. Some of them were active state or I guess they called it active content types of devices. And so we looked at some of those. And what that did is it built a state table in memory and it said okay, I shouldn't be getting a SIN or an AC from this area or this IP address speaking this protocol because I haven't initiated a session. I don't know how much that bought you, but it was a little bit better, I guess. There were a couple of them that did application proxies. Most of them were basic services only, with basic web, FTP. And those kind of fell apart as soon as we started talking about real audio, real video or anything else that a real company might be using. And a couple of them allowed you to create custom proxies on the fly. You could do your own proxy from this address to this address speaking to this protocol. There were a couple that were combination. And the last thing all of these devices do is network address translation. In fact, in some of them it wasn't even an option to allow or disallow network address translation. You simply were netting whether you liked it or not. Next slide. Some of the things we discovered is that we looked at these for some cable modem type devices and some did support DHCP on the external interfaces. Also, you might want to notice that DHCP on the external interfaces makes for some interesting types of spoofing attacks. And the ability to possibly convince the firewall to release its IP address mid-stream. We kind of played with that stuff. All of them did port and address redirection. And there were two or three of them that had a possible DMZ deployment with a third network interface on the device. Many of them offered offloading of the logging. And that was a good feature that let you move all the log files to some other device. Of those that we tested, the majority of that information was passed in the clear between the firewall and the logging device. The interesting thing is the ones that didn't give the offloading of the device. Because these are solid state machines, they only had a limited amount of memory to store those logs and support flooding this thing would cause it to run right through the log files and hide all your earlier attacks. And many of them also interesting to notice. A lot of them did alerting via email, paging, SNMP, and just basic protections against spoofing. Basically it wouldn't allow you to, many of them wouldn't allow you to have internal IP addresses on the external interface because of the NAT. Now comes the good stuff. Now that everybody knows what an appliance is and I can stop being product boy. This is how you recognize them in the field. Right off the bat, because of the nature of the way these folks deployed packet filtering, you will find many open high ports, ports above 1024. And these are not usually prevalent or existent in proxy based firewalls. So if you find those above 1024's open or filtered VNM map scan, you'll know for example that this is probably a packet filter. You can also, many of these systems identify themselves via the web browser. Simply connecting to port 80 or port 8080 on the devices would pop up this great little banner telling you how to buy the product. Fantastic. By the way, if you're in marketing and you think that this is a great way to market a product, stand up because we all love you. Tell that in SSH. We're also in recent use devices and amazingly once again, marketing, marketing, marketing, tell that in and it tells you what kind of device it is. Many of these were solid state as I mentioned. So you couldn't even change or disallow those banners. Gotta love that. The basic services may identify the types of devices. In this case, we were able to actually guess which holes or excuse me, which filters were in place. And we would use that to identify the devices that were being punched through to the backside of the firewall. And we'll talk a little bit more about that in a minute. Also, NMAP and CASO, we use those quite extensively in the testing process. And on many of them, we were actually able to identify the underlying operating system that was laying under the application. Amazingly, most of these are based on this thing called Red Hat Linux. You might have heard of it. Next slide. The real weakness is that we're going to talk about today. First of all, the general lack of logging. Firewalls in general, most platforms, do quite a bit of logging. So we were kind of amazed at the general lack of logging that was taking place in these little devices. Possible holes in the filtering rules, we'll talk about those and how to figure those out. DNS problems, address spoofing, some session hijacking. By the way, this is a great form of attack. This is fantastic in this case. Possible exposure of internal systems and denial of service. Right off the bat, almost all of these firewalls, all they did was log some type of connectivity. In many cases, they didn't even log dropped packets, only successful connections in and out of the firewall. Well, this is fantastic. We can port scan these devices all day, they won't notice, the admins never know. We can all kinds of play with some of these. The problem was, a couple of them did. They did record port scans, but when we looked at the dropped packet logs, unless you were a skilled administrator and saw that there was a series of packets from this port to this port, and every period of time, you wouldn't realize that it was a port scan. None of these devices just came out and said, okay, a port scan just happened. So we kind of liked that feature. You might want to know if you are going to play with one of these devices and you don't want your core administrator to know that you've been doing that, you might want to use some type of stealth attack or stealth port scan, like an end map thing scan or something like that. Be careful out there. Few log repeated connection attempts. In fact, we couldn't find any of these devices who would log the repeated connection attempts such as in a brute force against a telemet prompt from a system that was exposed to the outside world. They just seemed to ignore it. The traffic was going there, so it must have been okay. So that's pretty useful. The brute forcing is kind of lame, but at the same time, it is kind of nice to know that you can go ahead and play those games. None alerted against attempts to overflow exposed processes. And the big thing here that we discovered is these are like half firewalls and they aren't really intrusion detection devices. Again, part of this reporting process is that there was no clear way to identify what attacks had occurred and we kind of liked that. That was pretty fun. Almost none of them, maybe with the exception of two out of the 23, made any log at all of our attempts to denial or service the box. We hit these things pretty hard. And of the two that did, many of them only just dumped basic Linux log files which you would have to go through and grab to pick out. Now, how do we take advantage of some of this stuff? Well, the first thing right off the bat are these filtering holes. If you were at Black Hat and caught the extreme hacking from E&Y guys, you learned a lot about how this packet filtering stuff can help and hinder you. And I've got just some words for you there. Netcat is your friend. Got to know Netcat. FireWalk is a great tool as well. And it can be used to help determine what filtering is in place as well as locate possible holes in the filtering holes that allow you to map devices on the inside of the firewall. And that's very useful. NMAP will display the ports open by status as far as are they open, are they filtered or are they firewalled. Got to know NMAP as well. It is a great tool. Most of these devices, as I said, allowed some thin scanning of the internal network. And a couple of them, we also noticed some things like ARP leakage where the device would, for some apparent reason, start to ARP addresses on the external interface that belonged to devices on the internal interface. Also could be useful. Once holes are discovered and you've found ways to touch those internal systems, just exploit them using normal packets with forward source ports to reach the internal apps. And you won't even have to do that for some of them. Some of them, like I said, expose systems to the outside. The next real fun one for discovery were DNS problems. And because of the state of these devices and the way that they are created, they try to get these to market as a pretty cheap price. And I'll answer that question right off the bat. They range in value anywhere from $600 to $20,000. And it seemed to be that the cooler the case was in the outside, the more expensive it was. So the mileage may vary. But misconfiguration of these solid state DNS servers was very common. And here we showed a kind of a little example of using NS Lookup where you set type to authoritative, set the devices, your DNS server, and then just simply do an LS minus D with the domain name. And voila, usually it would dump out all of the internal IP addresses. Thank you. That's always fun to play with. Address spoofing. Spoofing the source address may allow you to pass through the specific packet filtering rules. And spoofing addresses on the internal interface may allow you to access to internal hosts. Now, what you have to be careful of here is that the folks on the inside are not using non-routable IPs. If they are, obviously you're not going to be able to spoof those internal addresses in the Shana local segment. So keep that in mind. Spoofing the addresses of other perimeter systems may allow you to access host applications. We found in some cases that as long as you were in the same class C as what the external interface in the firewall was, there were some inherent rules that weren't defined in the ACLs that applied to you. And you might want to be able to play with some of that stuff. My personal favorite here, session hijacking. This worked really well against these appliance devices. And all we did was we gathered connected users, in many cases, simply by fingering the device, finger just at the IP address. And it would tell us many, many times how many or what users were logged in. So we would wait until we saw somebody connected to the firewall and we would use a tool like Hunt to grab their session. And voila, we've already authenticated, we're through. We don't have to hack anything else. We've already taken control of sessions. This is probably the big one here. Those that didn't deploy proxy-based traversal of the firewall just simply said any connections on this port, such as port 80, route to this internal device. And what that did was it allowed us to carefully probe and examine those systems that were exposed. For example, we could look at banners of the systems and oftentimes we could query them such as send mail or other types of systems to find out exactly what those operating systems were on the inside. Obviously, if you scan the firewall and it comes back, NMAP determines it as a WinxBax or a Winx-based OS and you see this fantastic email system called Exchange, well, you know what's going on there. You know, it doesn't run on the same platform. So query those applications for system and version information and you'll be able to figure out exactly how to exploit those devices while passing through the filter. Next was denial of service or DOS attacks. And right off the bat, I want to tell you these are pretty lame unless you have something to gain. And so all of you folks who are DOSing all these servers around the country, I say thank you because I get to travel and see the world and they write checks and that's very cool. We did run into a few things here. Obviously, historically, there's been this problem where if you could DOS a firewall and sometimes in some rare occasions those older firewalls in some moment would fail in a completely open state. And the application would simply crash and it would start to route traffic between the interfaces. We couldn't get that to occur in any of these devices that we tested. However, just be aware that that sometimes does exist. Also, we talked about and we weren't able to exploit but we really think that this is the case in some of these devices as well as other firewalls in the market where you could DOS the box and wait for it to reboot and as it started to reboot it would load routing before it would load the ACLs. So there was a period of time there where you could route traffic across the device without passing through the access control lists and that could be very, very fun to play with at some point. Again, we couldn't find any time frames in those devices that we tested but we have heard that they are out there and so we might play with those. Typical denial of service rules apply. Malformed packet attacks. We were able to DOS a couple of the firewalls that way just simply crash the stack. Normally what would happen, the firewalls wouldn't reboot simply the external interface would just stop functioning and stop responding. Bandwidth socket attacks. Obviously, if you have more bandwidth than your target you can suck up all the sockets, nobody else gets connection. Very lame but I guess it might buy you something if you're really pissed. Floods and smurfs. That just general kind of stuff. Go ahead. Some of the other issues that we kind of discovered in this whole process was that some of the appliances passed management information in the clear and we discovered in some cases that you could actually remotely manage the firewall once you caught those sessions and intercepted passwords. That's very, very useful and some of these packages came with browsers or custom applications that you would load on your system and it might be possible to spoof management information whereas you might be able to change the ACL remote way. What percentage of those devices allowed you to have managed those scenarios? I know some of them. You can't configure the external interface. I'd say about half. About 50% of them allowed remote management from the external interface. How many of them have fun by default? On by default. Off the top of my head, I'm guessing maybe another half of that so maybe a quarter of the total deployment. Do you repeat the question? Yeah, I'm sorry. The question was how many of those devices allowed external, or excuse me, management from the external interface and then how many of them allowed that on by default and about 12 of them out of the 23 allowed you to manage the firewall from the external interface or remote device and of that about six of them had that on by default. In fact, there were two or three of them that you had to manage from the external interface when you first plugged the device in. You couldn't manage it from the internal interface of the firewall. The other question, the other issue that we ran into is the same as in any other firewall. If you're a firewall administrator, I pity you because misconfiguration of these devices as well as others is incredibly simple to do. The ACL rules are oftentimes pretty difficult to understand and even harder to figure out how to deploy in the order of significance. We ran into some that read bottom to top, top to bottom. It wasn't documented anywhere and half the time we'd call support and they wouldn't even know. So if you're dealing with software vendors too, the one thing I have learned through this process as I explained at Black Hat is I really pity you guys. The vendors and support people and it's just... I couldn't believe it. I'll get off my soapbox now. The misconfiguration issue, we found out that some of them allowed all the interfaces to be set to the same devices or the same IP address. Think about no sanity check. Here it is. These things would become bridges and route traffic everywhere. All three interfaces. So we kind of like that too. Now, if you own your own appliance and you want to test it, here's how to do it. First of all, take a sanity check on your access rules. Take a look at what you're doing. Figure out if you're going to pump data from the external interface to the internal interface. Where's it going? What's it doing? That's right off the bat. Use some of the mentioned tools that we talked about to test your own firewall. If you have this appliance firewall you deploy these without testing them yourself. Hey, all we can say is come to DefConn again next year and we'll give you a beer or something. We thank you. Use an automated scanning tool if you can find those out on the net. There are some commercial products if you have a budget. And lastly, I had to mention this one just for all my friends who are doing this. Hire a security professional to audit the firewall for you. We'll be happy to do that. Next slide. If you want to know more about these there's going to be a white paper that we will put out. It will be on www.microsolved that's slved.com We're hoping to have that up by the end of the month. We had meant to release that here but we've been having some problems with attorneys and getting people to sign off on some of the NDRs. Search. Can you repeat the domain? Yeah, www.microsolved that's slved.com Yes, and here's how. The question is if support can't help you on that on the question of importance of the ACL order how can you do that yourself? The first one is don't reinvent the wheel check with the firewall wizards mailing list some of those folks out there have used these devices. If you want to test it yourself though set up a couple of rules such as the first rule in the set allow access to port 80 on this device the second rule would be deny access to port 80 to this device and then see if you can access it try switching them around and you'll figure out which one is in order of importance so do that in your testing phase Did that answer your question? Yes Hopefully by the end of this month if the attorneys get everything done www.greatcircle.com we'll have a lot of the information about the firewalls the appliance firewalls there's some folks out there that are working on that stuff and there were several security and networking magazines that have done a couple of reviews on these so check those out and lastly if you have other questions ask me and I want to thank you guys for coming I'll answer any questions that you guys have Do you classify the Nokia device as a firewall appliance? Yes, yes we do and we did look at that device and there will be information about it in the right paper Yes sir We did everything from just pick two firewalls from Sonic Wall the question was what firewalls were included and I'm not going to go into a specific list but we went everything from Sonic Wall all the way up to $25,000 appliance firewalls such as the PIX and things like that And most of you have questions about the PIX The only thing that we could figure out price correlated to was how cool the case was I mean literally some of these devices that we looked at were $600 and they did a much better job than some of the devices that were $15,000 So, yeah and I want to answer another question that came up at Black Hat right off the bat before I take another some other folks have asked me what if there were a number of devices that were $15,000 and the folks have asked me what if there will be a current another review or an intended review to this and if we will continue this process and the answer to that is probably not and what we want to do is turn this over to the community and we want to hear from folks who are testing these firewall devices and if you want to add stuff in we'd like to create a living document to let you do that so that will be coming up as well How many devices it took us about six months to test all the firewalls the question is how long and how many it took about six months to do that we tested 23 we started with a field of 34 some vendors were long cooperative we only included in the white paper whether or not they included VPN technology the question was what did we do about the VPN technology did we test that do we not do any crypto analysis or any type of other application analysis of the VPN stuff if you see a reason why this couldn't be done right and if you think somebody will good question well the question is do I think someone else could do this right and do I think someone else will I think that there are folks out there who are testing firewalls I think we needed to do it in a manner that these folks can understand and that suited our needs so as far as that is I think we did it right but beyond that if you guys want to do it go out and test some firewalls do the appliance firewall right yes yes I think after looking at these 23 firewalls I think what we did find was that there is a future here these plug and play devices that are incredibly robust for what we were expecting and I think in the future you will see that I'd say within the next year or two years you'll see a very robust application proxy based appliance that is affordable for folks that are using small and home offices and cable modems so yeah I'd say you can reference it in the packet inspection techniques for quality did we notice I'm sorry one more time did you notice any difference in the packet inspection techniques the question is did we notice any difference in the packet inspection technologies whether it was stateful or not stateful inspection did help a little bit we weren't able to do some of the less ankle biter attacks against them but otherwise we really didn't see too much benefit from it and obviously there are some cases out there where it will help some attacks and it will halt some attacks so as far as performance went we didn't see any implication of any problems either from that as you know a lot of sophisticated firewalls pose a lot of problems and keep the accesses how many of them actually matter do you push the denial as I meant the average the question is encountered by firewalls and firewall administrators so how many of these came with an initial deny all state that was at the end of the rule set the answer was less than half yeah did any of these firewalls have like certification from ICSN and if they did did it relate to how good it was wow the question is did any of these firewalls have ICSN certification and did that correlate to how good it was yeah I'm going to answer that no some of them did have ICSN certification and I couldn't honestly tell you how some of them determined if that was ICSN certified and if they're ICSN folks here in the audience I'm sorry I'm just calling it like I see it the question is do I care if it's ICSN certified because they just simply write a check and along comes a certification is what he's implying and I would say no I don't care if it's ICSN certified in fact it means absolutely less to me if it's ICSN certified if it's ICSN certified if it's ICSN certified it means absolutely less to me if it's ICSN certified when I deploy a security device for a client or for my own use I want to know how it works and then it does what it's supposed to do and I could care less about any of the other stuff another misconfiguration question I do a lot of work at the checkpoint and there's just a few things you get to know that you're leaving by default we're not even in the rule base we're in the legal properties if you don't change them you leave aside from that it's pretty good the question was that the gentleman works with a certain X brand of firewall and that there are things that you inherently must know in order to secure the device did we find things like that yes we did we found some ACLs that were just completely unexplained as to how to configure them correctly and the whole thing is I heard my little rant there that I got off on and I'm not going to do that today just about how bad support sucked on these products we call up and ask folks that didn't even seem to know what packet filtering was let alone what to do about it so I apologize so go ahead that's perfect the gentleman was just letting you folks know that you can find the default admin passwords on 4point.com thank you thanks yes sir I got a question excuse me is there a firewall right if I have certain lines between the firewall and I left my CPU on and I left work now somebody point to me the question is if he leaves his device his computer as he goes home and he leaves himself logged into the network and his computer is turned on and he's protected by one of these devices is it possible for someone to access data on your device the answer to that is yes determine how to determine these folks out here in the audience are so those of you who have some of these data could you give it back he just realized what happened great question the question was did we see any open BSD based appliances there were two out of the 23 the open BSD can't name any names I'm sorry they did pretty well actually the question was how did they do in the process they did do pretty well and the question was that many of the devices that were UNIX based or Linux based amazing and BSD based were much more resistant to denial of service attacks than those that were using custom implementations for some of these folks they still if you're writing an appliance firewall and you're still using a proprietary like operating system we like you and you are a friend keep coming and make new revisions that you mentioned some of them are doing application proxies what's the level of content inspection and then if that brings stuff up to like viruses and stuff like that the question is if they are doing content inspection across a proxy how deep is that and are you able to offload virus scanning and things like that we didn't find any of them that allowed you to offload virus scanning we found a couple that some toys that you could kind of place in a DMZ situation that would allow virus scanning of content as far as the actual checking of the data content within the proxy that seemed to be fairly robust although we didn't find any that were configurable in that manner anything that was in a configuration issue kind of fell under that content blocking filtering I'm keeping you from going to xyzsite.com kind of stuff so are you going to be listing the names of the virus in the the question is are we going to list the names of the devices that we tested in the white paper the answer is yes for those whose attorneys will sign off and allow us to do so that's what we're dealing with now the whole end of this process turned into a big nightmare is like some of these vendors freaked out when they found out we were coming to DEF CON and you guys about it the question was how robust was the creation of the proxy process and what could you specify what data signatures or data loads you were looking to prevent we didn't find any that were configurable to that manner they simply said you could build these custom TCP's that were kind of like packet filters and it would make sure that the originating service was indeed you know xyz and this target so yeah it was a little more than graceful packet filtering any other questions one for you the question was some of these we've heard that they would possibly use stripped down versions of linux as code and how would we find out we were not able to determine that in fact some of them our favorite one seemed to be a full implementation of linux in this appliance device with a really cool front end and it did some really awesome things and we'll see some of that in the white paper the question is were any of them any of the appliance firewalls that we dealt with capable of handling H323 and I'm not sure we didn't we didn't test I want to mention to you to you guys as well while talking about some of these appliance firewalls that we discovered some things here that you really had to play with and I'm not sure that you were able to find things here that you really had to play your cards pretty close to the vest when you were talking to the vendors we found some that were significantly less than what we would classify as even a firewall a couple of them were simply NAT devices that did no logging or anything it just simply NATed your internal and external addresses and then we found this one which I'll kind of read about again that when we asked for the firewall to test they sent us too and we couldn't figure that out until we set them up and realized they were just point to point encryptors they were not firewalls so one more here the question was did we test throughput in capacity of the firewalls almost all of them came with either 10 megabit ether or 10100 megabit ether we tested those we didn't test any of the others we asked for some token ring implementations and we got back like what's a token ring you have to keep in mind that the majority of these devices are for small and home office users outside of this room not many people have T1s hanging out at home that could happen here I really want to thank you guys I'm sorry I can't take any more questions my time's up thanks for coming out to DC7 we'll see you soon