 Hello everyone. Can you hear me in back? Yep. Good, good. My name is Fyodor from Insecure.org and the M-Map project. I'm here with N-Map's co-maintenor, David Fifield. I'd like to thank you all for coming and of course DEF CON for having me. I've been going to DEF CON for 12 years and it's one of my favorite conferences. Basically I don't think there's really anywhere else in the world that you get such a concentration of smart people, passion for technology and energy as you do at DEF CON. There's also the parties as well. So I think I'm going to start out this presentation with a couple questions for the audience that will help us give us an idea of how much introductory material we need to cover. Could I get a show of hands for anyone here who has heard of and used the N-Map security scanner? Awesome. All right. Well, we'll definitely skip N-Map for dummies then. What about a little harder question? Who has used a newer feature on N-Map, the N-Map scripting engine? Okay. Well, we've got at least a third of you, maybe a half. That's great. And for those who haven't, we're going to really show the highlights and features of the N-Map scripting engine and maybe we'll be able to inspire you to give it a try. I want to mention that all of the evil hackers in this room, I think we'll take Wired please. All right. So these features that we're going to show are all available in N-Map version 5.35 DC1, the DEF CON release we made earlier this month. And also in 5.35 DC18 for some of the newer ones, which is our latest subversion repository code. So let's get started. The slides are kind of cut off on the left a bit, but I'm hoping that none of that will be so critical and we'll be able to work around it. So I'm going to start with just a little outline so you know what we're going to cover in this talk. First, we're going to give a quick introduction to the N-Map scripting engine, what it is, how to use it, why we built it, and then we're going to actually give you a demo to show you the power of the N-Map scripting engine. And we're going to do that by a large scale scan of more than a million IPs of a major corporation. We're going to hide this panel. Okay. So after we show you the large scale demo, we're going to then go into taking the next step and actually writing your own NSE scripts and how to do that. And then David has a really great demo to really demonstrate the ease of writing an NSE script by writing one from scratch right in front of you, in front of 500 people. And hopefully he doesn't have any syntax errors. So then after that we'll do the final notes and then we'll move to the Q&A room for the questions and answers. So the N-Map scripting engine is one of N-Map's most powerful and flexible features. It allows users to write and share simple scripts that they can use to automate networking tasks from enhanced network discovery to vulnerability detection or even exploitation. And the good news is you don't have to write all these scripts yourself. N-Map ships with a library of 131 NSE scripts which already can handle a wide variety of tasks. I feel that NSE really completes N-Map scanning mission. We already had host discovery where N-Map scans the networks and tries to find the machines that are up and available. We had OS detection where N-Map would figure out what operating system those machines are running. We had port scanning followed by version detection in order to find all the services there, what application and version numbers they're running. And then NSE is basically the glue that binds all of this knowledge together about the network. You can use it to interrogate the host in pretty much any way you desire. We're going to look at an example on this slide. I don't know if you can read it in the back but we're going to go over what it says anyhow. Basically here we're doing a scan of a machine scan me dot N-Map dot org. That's when I run for the purpose of people testing their N-Map, make sure it's working well. And basically you'll find out in this talk that I'm not really shy about scanning other people's machines and so I feel it's only fair to provide one of my own that people can use to scan me back. We're also giving the capital A option which tells N-Map to use its advanced features such as OS detection, version detection and the N-Map scripting engine. And then we get these results. Now most of it is going to be familiar to typical N-Map users, regular N-Map users. But I want to highlight the N-Map scripting engine results from this. First we have the under SSH, the SSH host key script which just grabs the cryptographic key hash that the server uses to identify itself. And while that might be useful for say a script that you would use to scan the network and find bogus keys that could be indicative of an attack or Trojan SSH, what I like to use it for is machines, a form of unique identification that is independent of IP address. So say I'm scanning a machine, I find one, I'm really interested into it, maybe I managed to brute force some passwords, the next day I try to get into it further, but it's gone. Maybe it's a DHCP client, it could have changed address to anything else. Well with SSH host key, what I do is scan the network again and try and find that key. Similarly, if I'm doing a big scan and I see a bunch of machines that seem to be configured similarly, are those really different machines or is it just one machine with a bunch of IP aliases? Well with host key script, that's a good way to figure it out. You'll also see some scripts below the HTTP port. We've got HTML title which is a super simple script which just grabs the root web page from a web server and tells you the title. Very easy to do, but useful information in the context of a scan. We also have HTTP methods which basically says what methods does the server support, get, post, whatever. In this case it says the trace method is potentially a little bit risky. Here's a URL you can use to find out more. If it had like the delete method or the put method then it would be sounding even bigger alarm bells. So the nice thing is that with NSE it puts the results down here by the port or host that they related to so you can see them all in one place. Now I mentioned that we already have a library of scripts here. Is this zoomed enough that you guys can read it in the back? Yes? Good deal. So this script, we have 131 of them and I'm certainly not going to go through them all right now, but I want to mention on the left side these categories that we have. So you can get an idea of the breadth of tasks that NSE can be useful for. The auth category is for authorization related scripts like we see up here the AFP brute force script which does brute force authentication cracking for the apple filing protocol. We have default scripts which is a hand curated set. We select to find the ones that are most useful to people and least likely to cause network disruption or to be perceived as attack or otherwise undesirable. We have discovery tests. I mean that's kind of end maps bread and butter. So if you click on a category you can see all of the scripts that are in that category. And so for discovery we have a bunch of information. If a DB2 database is found tell us what's up there, grab information from DHCP, DNS, HTTP and it goes on and on. We have the DOS category for denial of service attacks. There's only one of those. I think we only have one in the exploit category for exploits now. External, that's kind of a policy related category. Fuzzers. We have intrusive and safe categories are two important ones to know. Basically the safe ones we fill those are the ones you can run with little risk. They're not usually going to use a lot of network resources. They're not likely to crash servers. They're not likely to be perceived as really hostile by a network admin. Whereas the ones that aren't safe that we think do have some of those problems we put in the intrusive category so that you can consider more carefully why is it intrusive and does that matter to me before you decide on your scan. We have a malware category. For example, Nmap was one of the first scanners to remotely detect the config or worm last year. And the malware category is where that would belong. Version detection. Nmap's version detection engine now has more than 6,000 version detection signatures in it so we can detect a huge number of applications. But there are some protocols that they're basically designed to hide from people trying to detect them on the network. Skype is an example. And for those where our version detection isn't strong enough, well, we found methods with NSE with its ability to interrogate multiple probes and analyze the responses more carefully is able to do those. And then we have vulnerability detection related scripts. So this NSE doc page which you can visit at any time at nmap.org.nse doc is really an easy way to find the scripts that you think are going to be useful. And when you find the script you want, you can actually take it a little bit further. We're going to look at a script real quickly called NFS-LS. That basically says if Nmap detects an NFS server on the network, you can use NFS-LS to say what shares are available, what list the file names in those shares as well as the permission information. And yes, this is stuff you could already get from running other tools separately. But the nice thing is to have it all in your scan results all run automatically together. And so you find that through NSE doc and then you can actually click on it to get information about that script itself. So the very first link is to get the source code of the script. If you really want to figure out what it's doing, that's where you're going to go. The next is the description of the script. Basically tells you what the script does. You get all the arguments for the script. We have the wrong script here, obviously. This is the show-mount script instead of the LS script. This one has a bigger summary because it's more useful of a script. You can see the arguments. It takes, like, maximum number of files you want it to show, whether you want the modification time or access time or what. You get example usage so you know how to use it. We see some of the arguments are from a helper library we have for RPC and NFS. So you can see these arguments aren't specific to this script, but they're specific to a library that the script uses and so they might be useful. You get an output and as you can see, the author of this tried hard to make it look basically like a directory listing you'd get from LS and then you get things like what categories it's in. In this case, for listing NFS directories, it's in discovery because you're figuring out information about the network and it's in safe because you're not really doing anything intrusive. You're basically using this service as it was intended. So we have a lot of scripts now, 131 I mentioned and it's always growing. Here's an example of the data behind that growth. We started out with about 20 scripts. It took us some time to get our review process in place to get all the infrastructure there to easily write and process scripts, but then lately it's just been growing like we can't hardly believe. I mean, we've been working four years on this system and had it in end map for three and a half, but more than half of our scripts have been written in the last year and I won't be surprised if when DEF CON rolls around next year, it's doubled again. So now we've basically shown you the raw basics of NSE, basically what it is, what it can do. Now let's actually show you an example of what it can do. And to do this, I want to introduce you to a set of scripts written by a fellow named Ron Bose, who spent months researching the SMB and MSRPC protocols, which is not something I would really wish on anyone, but I'm glad he did it and these 13 scripts are a great gift to the end map community. Basically, there's informational scripts to just query the SMB server and say give us the detailed OS information, including service pack and such, the server stats, the system info, the security mode. We have enumeration scripts to say contact the server and give us a list of users or domains or groups or processes, sessions or shares. And then we have three more intrusive scripts. We have SMB Brute, which does brute force authentication cracking against the server to try and figure out passwords. And it can work in conjunction with the SMB enum users so that you already have a user list. We have SMB Check Volums, which checks for a number of remotely exploitable MSRPC bugs. And then we have SMB PS exec, which allows you to execute arbitrary commands on the server with some canned ones available for you for things like dumping password hashes. You can look at the different, you might want to start a remote display server, that sort of thing. So Ron sent me all these scripts and I'm like these things are awesome, but I don't run very many Windows machines on my home network. And so I was thinking, how am I going to test these? What sort of company might have a lot of these to run against? So I thought, you know, it would be pretty interesting, since it's their own darn protocol, you would think if anyone can secure it, Microsoft would be able to. And I thought it might be interesting to see what sort of policies they take. Do they just ban it at the perimeter? Do they lock the servers down so that they don't allow the guest access and the IPC share and that sort of thing? What is it that they really do? So I designed a scan. Step one was finding the target IP addresses allocated to Microsoft. And I used typical things, you know, look up the air and database to find allocations. And eventually I found more than a million of them and decided to scan them all. I started with a broad version detection scan. So basically this scan, I didn't want it to be super, super intrusive and detailed for the first scan. I kind of wanted to have an idea of what was there and then I could scan further on once I've seen the initial results. That gives me faster results and makes it less likely to raise a lot of eyebrows. So we say end map, do aggressive timing, scan what we've empirically found to be the 50 most likely ports to be open, do version detection, OS detection, do a minimum host group size of 128, which makes it faster because it scans more in parallel, set a host time out of 10 minutes so that we don't waste too much time on any single host, send the output to files, and I give it the list of IP addresses. And this sort of scan not many years ago could have taken a month to complete. And then if you wanted it any faster, you would have to use a bunch of really detailed performance options to tune it so that it goes faster but not too fast that you get inaccurate results. Fortunately, end maps gotten a lot smarter in the last few years, developed a lot of more clever algorithms, and in this case we were able to scan the million IP addresses in about one day, 26 hours, and we have 74,293 hosts up. So let's take a quick look at the results here, and I'll zoom this all on the screen. Good. And so basically this file is a big one, and that's actually the problem with it. I mean, more than a third of a million lines, this file is intimidating on its face, and for us to go through all these, it would basically be next DEF CON by the time we got it all done. So I want to show you a little trick I like to use in those sorts of cases. And so here's a scan, or a simple UNIX command that I'm going to run, which basically says take all of the open ports found during the scan, and look at the different versions running, and give me a reverse sorted list of how often every sort of service is running. So we take a look at the results, we see the highest is IIS-6. And by the way, I did give these results, I did these scans last year and gave these results to MSRC, so I hope, hopefully they fixed it by now. If not, it's going to be some long nights in Redmond coming up. We have IIS-6, so at least we can give them eating their own dog food here, running some of their own stuff. But rather than look at the top of the list, what I like to do is go down near to the bottom, and look at the more obscure services that they're running, because those are the ones you're more likely to be able to find a little bug in. I found little printers that I could get into the administration port on, check their toner levels, make sure everything's okay there. Various teleconferencing systems, there's just all sorts of bizarre stuff on here. But we're kind of getting distracted. As much fun as it would be to take a voyeuristic view into Microsoft's network right now, it's really NSE, NSMB that we're most interested in. So let's take a look at those results. So the vast majority of Microsoft's network blocks, I'm happy to say they basically filtered all of the MSRPC and SMB ports at their gateways. And so that's something that enterprises and businesses should really take in mind, although you should have already. I mean, if Microsoft feels that it should be blocked at the network and can't be really secured, it's probably a good idea for other businesses to do the same. But note that I said the vast majority of Microsoft's networks blocked these ports. There were actually some that didn't, and I found dozens of hosts that had port 445, the more modern of the ports available. So from that I designed a new scan. And basically I said Nmap and the key new option we're going to look at is hyphen hyphen script equals to say run Nmap scripting engine. And instead of doing the default scripts I showed you, we're going to do SMB enum domains, enum processes. Basically all of the scripts that I showed you before that are in the less intrusive categories like the enumeration scripts, the information gathering scripts. Even I'm not crazy enough to do brute force authentication cracking and crack their passwords and then present it at DEF CON. But even from there we ran the scan and got the results. And I would like to get some commitment from the audience here before we look at the results. Who here thinks Microsoft was totally secure? We've got one person, but I think there's probably several Microsoft people in the room who still aren't raising their hands. But let's take a look. Maybe Microsoft will surprise us here. So we're going to look at the results and we don't have time to go through all that many of them. So I'm going to go through one of my favorite machines. Can you guys, can you read this? Okay. So we have 976 closed ports. So that means they've defaulted the ports to be accessible so you can tell if they're open or closed rather than filtered. And so instead of default deny and allow what you want, they default allow and they filtered a few ports like telnet and off that they specifically wanted to prohibit. So that's the less secure way to do it. So we'll deduct two points for that. Next we look at M-Maps open port list. And as you can see, there are quite a few of them. More than is really easy to secure. And so for having that many ports open, I think another two or three point deduction is warranted. But the part we're really interested in is the host script results. This is where our NSE action comes in. First we have their shares to enumerate. They have some sort of admin share, a C drive, a D drive. Now these ones are restricted shares so you need a password which you could get thanks to maybe our SMB brute script. But one can argue that it's not best practice to share your hard drive over the internet even with a password. And so we're going to give them some points off for that, maybe four or five. Now we're going to continue on and we get to the SMB enum user script, which is one I talked about. And here we get the list of user names on their domain controller. And we have, some of them are actually pretty interesting. Here's the administrator user. This guy I like, I took off some of the last names to give him a modicum of privacy. But Richard, his username is the boss man, which I thought was pretty cool. Let's see what else. There are a few other ones that are worth pointing out here. We've got this MSB 50 net. That's the building 50 networking team. A nice thing about this script is that in the verbose mode hyphen v, it shows you all this extra information rather than just the list of user names. There's a user, not a guest exclamation point. And then, yeah, there's this support user, Microsoft Corporation, Redwood Washington. But my favorite is the T-shirt ho. So that's the sort of information that you can get by doing large-scale scans with these SMB scripts and with NSE in general. So as you can see, they're a lot of fun. I'd like to mention again that these scripts were actually written by a fellow named Ron Bose. And all I really did was take them and point them at Microsoft and pull the trigger. So Ron really deserves most of the credit. And I think if he was able to get in the room, yeah, there he is. Yes, thank you, Ron. These scripts are awesome. We also have a fellow named Drozan Popovic who's been working on us this summer to improve our SMB stuff even further. So now we've gone on how to use SMB scripts, basically what they do. But the next question is, what about writing NSE scripts? If you think about it, using them is probably as much as your average casual M-map user is likely to do. But really, any script kitty could do that. What do you do if you want a networking task, which we don't already have a canned script for? What if we have a script but it doesn't behave in exactly the way that you believe it should? This next section is going to show you what we provide to let you go to the next step. And even if you don't consider yourself a great programmer, how you can build these scripts to run your own tasks. And we're going to start basically with these scripts, the language that they use is called Lua. It's a great little language. It's easy to learn. If you know other languages like Python or Perl or compiled languages like C, you can probably figure it out. It's tiny to embed because we didn't want to bloat your N-map. The Lua book says, the complete distribution, source code, manual, plus binaries for some platforms fits comfortably on a floppy disk. So for those of you young people in the audience, a floppy disk, it's a small storage technology. Now, Lua is also widely used, known, and debugged. So we weren't taking something brand new that they might stop developing the next day. It's been in use since 1993. It's best known use is in the game industry with games like World of Warcraft or Crisis. But what we've been seeing more and more is that security tools have been picking it up. We're talking in this talk about how N-map uses it for NSE. Wireshark can use it now for protocol disectors. And the Snort 3.0 beta has its own Lua interpreter to extend Snort as well. Basically, the only holdout now on the Lua bandwagon is probably Metasploit, which is in Ruby. But whenever I see HD more at these conferences, I talk his ear off about why he should rewrite it yet again in Lua this time. I think that's why he's been avoiding me the last couple of days. It's also extensible. We can hook it up to N-map's fast parallel scripting engine. It's safe and secure. No buffer overflows, format strings. I mean, Wireshark showed us the problem with doing their disectors in C. They got a lot of contributions, but not all of them were secure. And it seemed like every week you got a new vulnerability in a disector that someone contributed. It's portable, works on all the systems N-map does, and it's interpreted so you don't have to keep recompiling. N-map adds a few capabilities to that. We have protocol and helper libraries. They help you make, say, an SMB query or an HTTP request without getting bogged down in the details of the protocol for making those requests. We have protocol brute forces like we looked at SMB Brute so that it can find the passwords for you in those times when you really need them. We have SSL and we have the dependency system. We talked about how SMB and NUM users can then take that user name list and SMB Brute knows to have users run fast, take those results and crack them. So that's a really quick overview of what we've added. Let's look at an actual example script. So I think the script that I'm going to show you this time is called RPCInfo.nse and it basically does exactly what you would expect. It, N-map users and pen testers in general who've been around a long time you kind of get used to seeing port 111, RPC bind open so you just run RPCInfo hyphen e hostname to see what's there and what RPC services are listening and what ports. Well it would be nice if N-map could do that for you whenever it detects the port open and it could put the results right there in your N-map results so you don't have to keep it all separate. And the nice thing is thanks to these protocol helper libraries and the like writing such a script took us, it is 46 lines long. So let's take a quick look at what it does. We have a description, we have a sample output, these are for the NSE dock page I showed you. We have a port rule tells you when it should run because it would be terribly inefficient if RPC info.nse was running against every port. This says only run if it's port 111 or if version detection detects it as RPC bind and for either TCP or UDP protocols. If that matches it runs this action. The key line in this whole script is this one that says rpc.helper.rpcinfo. It grabs that information if it failed it gives you an error message it sorts it into a pretty table sorts the results and prints it out to N-maps output. I'm sorry that was a real quick overview but as you can see you can do a script that actually does really useful things half of it is just documentation and the other half is pretty simple. So I could go on with all 131 of these scripts and show you them one by one but like I said before it seemed like it would be much more fun to have a live demonstration to show just how easy it is and so David is going to write one right in front of us that actually does useful things and execute it and hope that it works. Thank you David. Alright so to demonstrate how easy it is to write NSC scripts I set up a little example here and we're going to work our way through it. Anyone in this audience a programmer? If you're a programmer you can write NSC scripts. So before I left home I set up a webcam pointing out my window at downtown Denver and I want to find it and see what's on the webcam picture. The problem is I'm on a dynamic IP address so I don't know exactly where it is but we're going to write a script to find it. Before I left I found out some attributes of this webcam that can help us fingerprint it. It uses T-H-G-T-P-E to serve and it serves up a file called cam.jpeg. So we're going to write a script that looks for those two things and finds my webcam. So we're going to start let's call this H-G-T-P webcam.N-S-E Sorry? Oh. Let's like redo this. Alright so the customer extension is NSC. Every script needs a description. Syntax highlighting. Sexy. So this script is going to be in the safe category because we're not doing anything weird also I would put this in discovery. Every script needs a rule. In this case we're going to use something called a port rule which decides when the script is going to run and in this case we just want it to run when the port number is 80. So this is real simple. Return port number equals equals 80. And then finally the action is where the script really does its work. Again it takes two parameters the host and the port. Now we're going to be using H-G-T-P functions so we need to require that library. This is just like an import in any other language. Let's define a local variable here for our response and we want to retrieve cam.jpg. Now let's just test what we got from the response here if response.status we're just going to make sure that it worked and response.status is not 404. And remember what else we're looking for? Response.header we want to check that the server header exists just a minute. And string.match is a standard Lua function and we're going to match it against the pattern thtpd. Then it's easy to return results from an NSC script you just return a string. In case we don't have any results we just return nothing and then the script isn't going to report anything. So let's run it. I'm going to add a couple of flags on here to make this a little faster. So pay attention if you have trouble with the speed of your NMAP scans. Turn off reverse name resolution. Turn off ping scanning. We only want port 80. Only tell me about hosts with an open port. We want to run HTTP webcam. The address range I'm going to scan. The address range I'm going to scan is this one. Oh, do you have to zoom it? Oh, I got you. Okay, thanks. And we're going to scan these addresses. Let's write the results to a file in case it scrolls off the screen. And when you're testing your scripts for the first time, you always want to add the debug flag in case you're a programmer who makes mistakes. It will tell you where it'll give you a back trace to your syntax errors. Yeah, right. Oh no. So what is that? One of the virtues of a programmer is hubris. All right, there we go. Oh, that's the problem. We need to update the voice recognition. Okay, so Nmap is fast. Let's look at the log. There we go. This is the address of the webcam 66.7171.5. And that's how easy it is to write an NSC script. Let's take a look at it. Okay, so let's not let this scare us. This is just an opportunity for us Nmap developers to do what we do best, which is scan. I want to show you a script that I've been working on. All right, you follow me? All right, so let's go through this one real quick. You're gonna, the structure is gonna be familiar to you. Description guesses HTTP passwords. Categories, I can't really, with a good conscience, put this in safe anymore. Put it in auth category. We need the HTTP library again. This is another password. This is our username password database library, which all our brute scripts use. And I'll talk to you a little bit more about that in a second. Port rule is exactly the same. And the action is pretty simple. Just a few technical details here. We have a username iterator, a password iterator. We have a nested loop here. We're getting cam.jpeg. The only difference here is we're passing it a username and password. And if we get a result that is not 401 authentication required, we're gonna return username colon password. So let's run that scan. It's gonna be very similar. Just take off a few of the stuff, some of the stuff we don't need. And, oh, change the name of the script. Thank you. And letter rip. So, the thing about password cracking is it tends to take a long time. Let me tell you a little bit about our username password database library. Whenever we're shipping databases with NMAP, we like to base them on real results. You know, it's easy to go download somebody's random password database, but you don't know how good it is. You don't really know where it came from. You don't know if it's gonna help you in your situation. So we tried to base this on, hey, that was too fast. I'm not done talking about the password database. I wanna say that our password database is based on some real measured results from some public information, some data breaches, password disclosures, and things like that. We've curated them and built them into a pretty decent list that you'll get if you download NMAP. So that's what we've used here. So now in 25.90 seconds, it has cracked the password. Username web, password monkey, that's an honor of the goon who introduced us. Let's give it a try. And there we go. All right, so I had to, this was, I made the script very small so it could be understood in a short time. Before I add this script to NMAP, there are a few changes I'm gonna make. The port rule has to be more generic. You know, it's lame to just match port 80. You wanna match all the common HTTP ports and also any ports that version detection is found to be HTTP. We have a library called short port that makes that very easy. I would add script arguments so we don't have to hard code cam.jpeg. You could just specify that on the command line. Add documentation for usage and output so it can go on our online documentation portal. And finally, this is very important. I want it to be able to cache credentials in our registry so that other scripts that run later through our dependency system have access to these. Now let me turn it back over to Feodor with some final notes. All right, we're gonna just run through these last slides really quickly. First of all, we have a lot of stuff coming in NSE. We have scripts in the queue. We have an idea for scripts that can run before and actually accumulate targets so you could do like a zone transfer or a broadcast ping or the like and a hand those targets straight to NMAP. We have ZenMap NSE integration. That was gonna be on the coming soon slide, but instead it's here because we actually put it in the repository just about three days ago. David was working with a fellow named Kuru Bakhan Sampath to create this extra panel in ZenMap which shows you all the scripts we have available, what categories they're in, the descriptions, what arguments they can take and lets you set those. So it's a nice feature if you don't have all the scripts memorized. I wanted to mention credit where credit is due. I wrote NMAP 13 years ago and sometimes get way more of the credit than I deserve. It's really an open source project in the true sense with kinds of contributors all over the world and this is just a list of the people who've written scripts for NSE which is just a subset of the contributor set as a whole. Final notes, the slides are up now at insecure.org slash presentations. You've gotta scroll down a little bit to get to the 2010 part. Here's the URLs for downloading NMAP, the NSE doc portal, the system docs and for Q and A we're gonna take that right across the whole in 118. Thank you very much.