 All right. Thank you. Sorry for the delay. My name is David Feifield. I'm an end map developer for about two years, and I'm the maintainer of the end map scripting engine subsystem. That's what we're going to talk about today is end map scripting engine or we call it NSE for short. I hope you all were able to pick up one of these. We're going to refer to that. If you haven't and you have your laptop, you can go to this URL and download one. But do pick one up. I mean, I've got links on here for resources and things if you want to get into script development. All right. So I want to start by assuming that not everyone in the room is an end map expert. There have got to be a few newbies in here. And even those of you who think you are experts may not be as expert as you think. That's something that happens to me just about every day. So let's look at what we're going to do for this talk. Okay. Don't do that again. About 10 minutes we're going to spend on NSE for end users. How do you use the scripting engine? How to interpret the results? About 20 minutes we're going to talk about how to write scripts, how to be a developer, and we'll walk through maybe writing a simple script. And then finally, I have an application of kind of a neat application of NSE that I want to show you and stick around. I think you're going to like that. Oh, and do pay attention. There are only two jokes in this presentation. And that was the first joke. Okay. Okay. So for those of you who are not familiar with end map, please go to look at this highlighted section of the handout about end map. I would run a live scan for you. This isn't my computer, so we're just going to show you the results that are here. These are typical end map results. scanme.nmap.org is a website that's set up just for people to play with end map or to test other kind of port scanners. So if we run end map scanme.nmap.org, this is the output we'll see. It scans in about nine seconds. It has scanned the 1,000 ports that we found empirically are most likely to be open. And it has found two open ports, four closed ports, and 994 filtered ports. Filtered ports are those that didn't send back any reply at all, not a SINAC, not a reset or anything like that. So this is great. And this is what people know end map is for. And end map is really super at this. We're always looking for ways to improve end maps results, and NSE is part of that. So let's talk about why NSE exists. How many of you are familiar with end maps version detection? Okay, yeah. So you just give it a little dash S capital V, and end map will, for every open port, it will send probes to that port and try and get a response. And then based on the response, it matches it against a database of regular expressions. And it'll tell you, you know, this is probably Apache. This is probably IIS. We have some pretty crazy ones. Like this is probably a web thermometer. This is probably a car wash, you know. Yeah. Version detection is great, but it can't do everything. There are some network tasks where you need to send more than a single probe. You need to send the response, get something back, and then depending on the response, you may need to send, you know, one of three things or whatever. You need some degree of programmability. I wasn't around when NSE got started, but what I have heard is that one of the original use cases for it was detection of the Skype protocol, Skype version 2. It's really a tricky protocol. I think one of the reasons is because they try to make it hard to detect. If you send something that looks like an HTTP get request to Skype, it will send back something that looks like an HTTP 404. If you send anything else to Skype, it will send back what looks like random data. It's probably encrypted or something. I don't know much about the protocol. I think people have tried to reverse engineer it. But anyway, you really need two probes to detect it accurately. So, I mean, that's one of the original motivations. Another thing that we hope NSE is used for is exploit research and development. Say that you are a security researcher and you found a new exploit and you want to demonstrate it. You might write a proof of concept, you know, in Python or Perl and it runs against one host and it exploits this. Or maybe it doesn't exploit something, but it tests for a vulnerability and you've got this tester that runs against one host and you say, okay, now let's make this real. Let's bring in some threads or something to make it fast. Let's make it run against lots of hosts. We'll add in name resolution, everything like that. All of that is sort of nuts and bolts coding. It's separate from your exploit or from your vulnerability checker. NSE is sort of designed to free you from that. So, if you write your original script in NSE, that's your proof of concept and that also becomes your production scanner because it automatically becomes parallelizable. It automatically can run against multiple hosts. It automatically is fast because it's using NMAP scanning engine. So, how to use NSE? Simplest use, it's very easy. We just add a new option. You say dash s capital C. I guess that sort of looks like script. It's not that hard to remember. We're running out of letters to name NMAP options. Now, these results are almost exactly the same as before. You'll notice it took quite a bit longer, about 27 seconds. And we only have one extra line. It gave us this HTML title. That's the title of that web page. So, unfortunately, that's not very inspiring just because ScanMe doesn't respond to a lot of the scripts that we have currently. But let's adjust the command line a little bit. Now I'm using the dash dash script equals safe option. What that's going to do is run all the scripts in the safe category, which is more than you get with just the default scan. And now you see we've got a whole lot more to look at. We've got the HTML title. And if you went to that web page, you would see that's the title of that web page. Another script called HTTP headers has downloaded all the headers. A third script called HTTP date has got the date, which is obviously a subset of the headers. And it's also showed you how far the remote clock is off from the local clock. And I don't know why. ScanMe is always like 5 to 10 seconds off. I have no idea why that is. And some extra information here. ASN, ASN query script didn't return any answers, but we also got who is results. So this is sort of the who is script is like a replacement for the who is command. So now this is a lot more interesting. And this is just a taste of what NSE can do for you. Okay, now I'd like you to turn to this highlighted section using NSE in your handout. And this is going to give you a quick rundown of how to activate NSE in various modes. We mentioned dash SC dash dash script equals default. Those are actually synonyms. And NMAP dash capital A. Capital A is advanced and aggressive scan. Can anyone tell me what else that option turns on besides script scanning? This, this option here, sorry. Yeah, OS detections one. Turns on script scanning OS detection, version detection and tracer. So if you want a lot of results and you're not afraid of, you know, raising a few snort eyebrows or whatever, that's the option to use. If you have a specific script in mind you want to run, you just give the script's name SSH dash host key. We follow the convention that NSE scripts have a file name that ends in dot NSE, but you can omit that. You can use shell like wildcards. So this will run all the HTTP scripts and all the SNMP scripts. And obviously you have to quote that to protect it from the shell. We try to follow common naming conventions so that you can, for example, run all the HTTP scripts like that. Scripts are divided into categories. So that first example I showed you, script equals safe. That's just an example of category scan. You can separate multiple scans, multiple categories by commas. So here we're going to run all the scripts that are safe default or that check for malware infections. And then finally Boolean selection you can use and or parentheses and not to make anything you want. So for example, you want to run all the default scripts and all the discovery scripts, but nothing that could be considered intrusive. That could possibly crash a machine or things like that. Now I'd like to see if I can open this in a tab. Oh, never mind. That's not going to work. Yeah, that's right. Sorry. I would like to show you this, but definitely if you're interested at all in script development, write down this URL. This is what I would show you nmap.org slash nse doc. This is where we have documentation for all the scripts. We have 94 scripts I checked this morning and a couple dozen libraries. All the documentation is there online. When I'm writing scripts, I consult that just about daily. I'm not going to read all the script categories to you. You can read them and also in the online documentation we've got more information. Just a few notes on some of them. External means that it could contact a third party. And so if you're trying to be very private in your scanning, you want to just put and not external at the end of your script selection. Intrusive is also worth a look. When it says may crash systems, that's not a joke. We have some scripts that test for denial of service. And the best way we figured it out how to test for it is to deny the service. And so there are scripts that do that. So don't run intrusive scripts. Well, you know, read the documentation before you do that. And one last category is sort of a special one, the version category. Those scripts are run when you do normal nmap version detection. So I mentioned that Skype detection script. That's going to run if you just ask for version detection. Some scripts can take arguments. All you do is just give dash dash script args. Let me give you some of the examples here. Like for all the SNMP scripts, they take an argument called SNMP community. So you can set the community string. Otherwise it defaults to public. This is kind of a neat script PGL ready message for certain HP printers. This will set the message that's displayed on the little LCD screen. So you give it the script arg to whatever you want it to say. And if you're running the SNB scripts, for some of them, they can do better work if they have a username and password. So you give the SNB user SNB pass args. Again, look at the online NSC doc, and that tells you all the arguments that all the scripts can take. If you are running the SNB scripts and you don't have a username and password, don't worry, just run the SNB script first. We got started a little behind schedule, so I'm not going to show you the NSC doc for these scripts. I guess I couldn't get on the net anyway. But these are just a few that I like that I think you could start using to automate some of the network scans that you're already using. Banner, for example, just connects to a socket, sends nothing and listens, and prints out whatever it receives. Sometimes that can be useful if version detection isn't working for you. This is kind of a fun one. HTTP favicon downloads the favicon, hashes it, matches it against the database, and from that it can fingerprint some web applications. So that's a summary of NSC for users. Let's talk about NSC for developers. In your handout, it's on the back part here. It's a section labeled anatomy of a script. This is the HTTP date script, and we're going to walk through it really fast. This script is written by me. It's the first script I wrote. At the time, I was sort of embarrassed just because the script doesn't do very much, but now I'm glad I wrote it because it's almost small enough to fit on one page, and it makes a great example. And of course, it's impeccably coded. So at the top, there's a variable called description, and this, again, will go into the automatically generated documentation. This can be long. It can be several paragraphs. It can include hyperlinks, and we encourage you to be as descriptive as possible there. This square bracket syntax, that's just Lua syntax for a string. Boy, I almost forgot to mention. NSC is written in Lua. Okay. It's used by a lot of other security tools too, so there are more and more people learning how to use Lua. But I think, I mean, you can look at this. Any programmer is going to be able to understand what's going on here. Lua comments begin with two dashes. Special comments that begin with three dashes start NSC doc sections. So you see this little at output. This is going to go into the online documentation as a special output section to show you what the script looks like when you run. So for example, here is me running this against, this was probably scanned me, again, off by a few seconds, and here's me running it against my DSL router, which apparently has a date two years ago that's set when it boots. A few other variables have script metadata, author and license. If your script is under a suitable open source license, we can consider including it with the Nmap distribution. This is where we define the categories that a script belongs in, and a script can be in any number of categories. So this one's discovery just because it discovers the clock setting in the remote system and safe because it's not going to do anything. I mean, this is what web servers are designed to do. We use three external modules, HTTP, short port and standard NSC. Now, here's the real interesting part, and this is what makes NSC what makes it itself. The port rule is a function, and this function is run against every open port that Nmap finds. Script scanning is done after the port scan, so we have a list of open ports and a list of scripts that the user asks to run. Each one of those has a port rule, and that port rule is run against all the open ports. For all those ports that it returns true, the script is going to run. It sounds kind of complicated, but we have a library to hide most of that. It's called short port, and basically what this means, this function call is going to return a function that returns true whenever it's running on one of these port numbers or whenever version detection has found one of these version names. So basically, we want to run this for anything that looks like HTTP. And finally, the action function defines what the script is going to do when it runs. So the action function gets two arguments. It's a host and a port, and I mean, this is pretty simple. Anybody could figure out how to get the date from an HTTP server. We have an HTTP library. We get the root document, parse the date from it. We have a standard library function to format the difference of two times, and we return it. The action function returns a string, and then that string is what appears in the output. So it's pretty easy to use. So if you want to get started writing scripts, my recommendation is to start with an existing script and modify it. There are a few required fields. You need to have a description. You need to define categories. You need a port rule, and you need an action. And once you have all those, your script can run. But NMAP is going to complain to you if one of those is missing, so it's not like you could make a mistake there. Now, I mentioned port rules. There are also scripts that can use what's called a host rule. And these are a little bit different. These are only run once per host, and these are scripts that don't belong necessarily to a single port, but they belong to a host. So like who is is an example of that. We don't run who is against a port. We run it against a host. So who is has a host rule and not a port rule. The difference here, let's look at some examples. banner.nse is a port rule script. You see port rule. banner. We like to run for any TCP port. And the action function takes two arguments, a host and a port. ASN query, which gets ASN numbers, or sorry, gets AS numbers, has a host rule. It runs for any non-private IP address, and we have a library for defining what's a private IP address. And this is important. The action function only takes one argument. It doesn't get a part table. There's a technical thing that is useful for script writers to know. How does NSE handle parallelism? I've mentioned that NSE is good for parallelism, and it is. You have all these scripts running at once. How do you control which is running when? The way this works is your script yields control whenever it does a socket operation. So when your script does a connect, another script gets to run until that connection is completed. And as soon as that connection is completed, your script gets called back and it continues. Now this makes writing scripts really easy because to the script, it looks like a normal blocking socket call for all the scripts. Same thing when you do like a send or a receive. Your script gives up control, other scripts get to run, and then transparently your script will come back as soon as that send or receive is finished. So again, if you're writing your scanner, you don't have to worry about writing non-blocking sockets and you don't have to write a threading engine or anything like that. It's just going to happen when you use NSE. Okay, the next section down below tips for script development. There are some files in the end map source you're going to want to know about. NSE main.lua is the actual core of the scripting engine. It is itself written in Lua just like the scripts. That was one of the summer of code projects last summer, was to rewrite the scripting engine from C++ to Lua. We had a hacker named Patrick Donnelly do that. NSE lib, all these start.lua files, these are the standard library. And again, look at the online NSE doc to learn all about those libraries. If you're going to write a script that, for example, uses DNS, don't write your own DNS engine because we have a library for that. There are some libraries that are written in C++ and not in Lua. So some of those NSE.cc files are those. And if you're interested, like, for example, our open SSL library is written in C++. If you're interested in that, it's not that much harder than writing a pure Lua library. But you are going to want to consult the documentation and probably read the programming in Lua Book. And finally, there's one script in the scripts directory that is not a script, script.db. And this is just a database of what category each script belongs in so that we can load them quickly without having to actually run an interpreter on them. So, you script writers. I don't know, has anyone thought about writing an NSE script or found an application for that? When you're developing, there are a few things that I have found useful. One is to know about the data dir option and give it an argument of dot, the current directory. That's going to run scripts in the current directory and not those that are installed in the system directory. You know, it'll be a few times before you modify your script and you say, why isn't the output changing until you learn to do this? After you install a script, you have to run script update db and that's going to update that category database. Otherwise, again, you're going to be saying, why isn't my script running? And this is super important. If you use the nmap debug option, dash d, you're going to get back traces and it's going to inform you whenever a script fails to load and that's definitely very useful for development. Okay, you've seen my talk. You all know how to write Lua and NSE now. So, you've written your script and how do you get it in the nmap distribution? Well, the way to do it is to write to our mailing list, our dev mailing list, which is nmapdev at insecure.org. Recently, we had a volunteer, Patrick Carlson, came in and wrote just a whole bunch of great scripts. He's committed, I don't know, probably about 10 scripts in just the past few months, been really prolific. And this is just an exchange that took place on the mailing list to show you kind of what you can expect when you submit a script. So, first of all, he says, I finished two scripts. I've got one that shows NFS mounts and one that shows Citric published applications. And I wrote back, I said, okay, you've got, you know, I like it, but you've just got undocumented packet contents here. I'd like you to, you know, figure out what those bytes are. And he said, okay, well, I had to reverse engineer some of this stuff but I've documented the packets and then, okay, we added the script and that's how it works. And now we have scripts to show NFS mounts and Citric's applications. Okay, finally, well, not finally, there's one more exciting section after this. Some resources to know about if you want to get into script scanning. If you weren't aware, there is a book on nmap. It's called nmap network scanning. It's called nmap.org slash book. And there's a chapter on NSE and that whole chapter is available online. So you've got no excuse for not reading it. Programming in Lua is the official Lua manual and it's actually decent programming text even if, you know, you're not interested in Lua. And again, it's all online. Well, not the latest edition, but one edition back is all online. Definitely the online NSE doc has got documentation for all our scripts and libraries. And finally, the nmap dev at nsecure.org mailing list is where you ask questions, report bugs, and send your script when it's finished. All right. We have a script I mentioned earlier, http-favicon, and this is part of the NSE doc. Gets the favorites icon from a webpage and matches it against the database of the icons of known web applications. If there is a match, the name of the application is printed. Otherwise, the md5 hash of the icon data is printed. We got this script submitted actually a long time ago by a guy, and I hope I pronounce his name right, Vladko Kosturyak. And we didn't accept the script for a long time. He said, okay, this is a good script. It's a simple idea. I like it. But we need a good database. There was a database that had maybe a dozen entries and we don't know how relevant the entries are. We don't know how common these things actually are. So we solved that problem as we solve all problems in the NMAP project by scanning. We said, well, let's set up a scan and start scanning web servers and downloading icons and we'll do a frequency count and then we'll just document whatever the top 50 icons or whatever. So I wrote a script, faviconsurvey.nse, and a colleague of mine, Brandon Enright, who is a super NMAP scanner. He's a guy who really knows how to make scans go fast, especially huge scans like this where we're scanning about 10 million sites, ran these scans for us, and we downloaded about 10 million icons and about 30 gigabytes of data altogether with, you know, including host lists and everything like that, and including file system overhead because file systems aren't really happy about storing 10 million tiny, tiny files. And we made a database. So if you want to look at this, you know what the most common icon on the web is? The empty file, the zero byte file. It's not very exciting. That one's commented out. That's not useful for identification. But we have this database of hashes and matched up to applications. And okay, good. The only thing is it seems like kind of a waste to get 30 gigabytes of icons and only come undone but with a one kilobyte data file, don't you think? So what would you do if you had a million tiny icons? Well, we had one idea. But the ransom notes weren't really paying off, so we changed that. We said, let's be more constructive. Let's create some art. Okay, I intended to... Can I get a volunteer? There's a couple of posters right there in the corner. Do you want to hold those up so everybody can see? I meant to hang those up earlier. So we took all of those roughly 10 million icons and we sorted them by how often they appeared and packed them semi-randomly into this graphic and created this work of art, which is, I don't know, I think it's awesome. And we had one of those hung up. I'm going to have some posters here in the hall so if you can find them, come take a look, but they're pretty cool. If you want... Yeah, let's unroll that and show everybody. It turns into a pretty gigantic poster. Thanks, guys. So if you want more information about this, if you want the data files and you want to download... Oh, I should say something about the poster. You can download this graphic and there's a PDF of it. The graphic is roughly 21,000 pixels wide and 24,000 pixels tall. It's 900 megabytes. If you're thinking of having it printed, I should warn you that it's been known to commit denial of service attacks against commercial printers. I got email this week and they said, I'm sorry we can't print your poster, it keeps crashing our printer. Jodor told me this is the first time Nmap has crashed a printer without sending any packets to it. But if you want some more information, we set up a quick website for this project, nmap.org.com. Please take a look, let us know what you think. This is my name, this is my email address. Nmap is at nmap.org. Are there any questions? Okay, so if I understand the correct... The question is, has there been thought to use Nmap for more than just running after a port is found open or running after more information is found? Scripting engine does run last, so it should have... It has as much information as it's going to get besides what's known. There are ways to coordinate the running of scripts. You can say one script depends on another, so if both scripts are run, one of them will run before the other. For example, our brute force password guesser scripts run early because other scripts have the use of the passwords that they find. Do I answer your question? Okay. You can find me after, yes. I don't know, and that would be a good thing to see. The question is, let's have some... What do you think of some integration between nmap scripts and metasploit scripts? And I think there are a lot of overlaps. There are things that both of them can kind of do well, and I have to admit that I don't know much about metasploit development. So I wish I could tell you more information, but I can at least thank you for the idea. Yeah, any other questions? Yeah. The question is, are there any problems, any legal problems with using nmap in certain countries? And... I'm not a lawyer, obviously. I think there may be a problem in Germany, although I did do a talk on nmap in Germany, and there seem to be a lot of nmap users there. You might have to talk to someone who has more experience with that, and there may be some information in the man page about, like, exports to terrorism-supporting countries or something like that. I really don't know much about that. Anything else? Yes. So the question is, how intelligent is the parallelism? He gave the example of a who-is server, where you don't want to hit the who-is server for every single host, especially if all the hosts are in the same net block because they're all going to have the same who-is. So that's an excellent question. And that exact example drove part of our library. So there are ways to disable the parallelism. We use a mutex so that the first who-is script that runs gets access to the server and the others wait until it finishes, and then once it's finished, they check a cache of who-is entries, and then if the entry is there, they don't have to hit the who-is server. So that does require some intelligence on the part of the script. It has to be built into the script, but there are ways to enforce that. All right, I think maybe one more question. Yes. When your script gets, when it runs the action function and action receives a host table on the port table, the host table should contain that you need. So it's got OS scan results and it's got the DNS name and it's got the IP address and everything like that. So it really should have access to everything that Nmap has access to, and I'm sorry, the question was, do scripts have access to all of Nmap's information? And they should, and if they don't, it's a bug and we can probably fix it. In fact, I think there's a patch pending to include the MAC address in the script now. Thank you, everyone.