nmap XML output for "down" hosts on -F scans

[Andrew Smith]

Hi folks,

I'm currently working on a monitoring project using nmap as the
scanning/discovery engine.
The XML output is being parsed and updated on a MySQL database with a
PHP front end to display/filter the results.

Running nmap with -sP provides output of both hosts that are up and
down, for example:

nmap -sP 192.168.2.0/24 -oX nmap-sP-192.168.2.0-24.xml

....
<host><status state="down" reason="host-unreach"/>
<address addr="192.168.2.6" addrtype="ipv4" />
</host>
....

However, if I run nmap as:

nmap -F --script smb-os-discovery 192.168.2.0/24 -oX
nmap-F-smb-os-discovery-192.168.2.0-24.xml

then I only get results for hosts that are "up".

I have tried using debug and verbosity (-d9 -vvv) but this doesn't
provide any information about "down" hosts.

I'm working around this at the moment by running a -sP first pass
followed by a -F second pass but it would be more efficient (and reduce
the load on the system) by being able to get the status of all hosts in
the -F pass.

If this feature isn't available, is it possible to request it to be
added to the next version of nmap please?

Thanks for your time,
Andrew
Send instant messages to your online friends http://au.messenger.yahoo.com

[David Fifield]

Post by Andrew Smith
I'm currently working on a monitoring project using nmap as the
scanning/discovery engine.
The XML output is being parsed and updated on a MySQL database with a
PHP front end to display/filter the results.
Running nmap with -sP provides output of both hosts that are up and
nmap -sP 192.168.2.0/24 -oX nmap-sP-192.168.2.0-24.xml
....
<host><status state="down" reason="host-unreach"/>
<address addr="192.168.2.6" addrtype="ipv4" />
</host>
....
nmap -F --script smb-os-discovery 192.168.2.0/24 -oX
nmap-F-smb-os-discovery-192.168.2.0-24.xml
then I only get results for hosts that are "up".
I have tried using debug and verbosity (-d9 -vvv) but this doesn't
provide any information about "down" hosts.
I'm working around this at the moment by running a -sP first pass
followed by a -F second pass but it would be more efficient (and reduce
the load on the system) by being able to get the status of all hosts in
the -F pass.

This is the same problem that was reported at
http://seclists.org/nmap-dev/2009/q3/1081. Unfortunately there's
currently no way to get down hosts in the output when a scan includes a
port scan, script scan, or traceroute. The reason is that Nmap removes
the down hosts from its internal data structures before doing the
further scanning and output.

Depending on how much control you have over the parsing, you may be able
to work around the problem by doing the first pass with -sL rather than
-sP. That will add an entry for every host, with a state of "unknown".
You can then subtract the up hosts from the second pass to get the set
of down hosts.

David Fifield

[David Fifield]

Post by David Fifield

Post by Andrew Smith
I'm currently working on a monitoring project using nmap as the
scanning/discovery engine.
The XML output is being parsed and updated on a MySQL database with a
PHP front end to display/filter the results.
Running nmap with -sP provides output of both hosts that are up and
nmap -sP 192.168.2.0/24 -oX nmap-sP-192.168.2.0-24.xml
....
<host><status state="down" reason="host-unreach"/>
<address addr="192.168.2.6" addrtype="ipv4" />
</host>
....
nmap -F --script smb-os-discovery 192.168.2.0/24 -oX
nmap-F-smb-os-discovery-192.168.2.0-24.xml
then I only get results for hosts that are "up".
I have tried using debug and verbosity (-d9 -vvv) but this doesn't
provide any information about "down" hosts.
I'm working around this at the moment by running a -sP first pass
followed by a -F second pass but it would be more efficient (and reduce
the load on the system) by being able to get the status of all hosts in
the -F pass.

This is the same problem that was reported at
http://seclists.org/nmap-dev/2009/q3/1081. Unfortunately there's
currently no way to get down hosts in the output when a scan includes a
port scan, script scan, or traceroute. The reason is that Nmap removes
the down hosts from its internal data structures before doing the
further scanning and output.
Depending on how much control you have over the parsing, you may be able
to work around the problem by doing the first pass with -sL rather than
-sP. That will add an entry for every host, with a state of "unknown".
You can then subtract the up hosts from the second pass to get the set
of down hosts.

Here's a patch for everybody's consideration. What it does is print out
a host record for down hosts before they are removed from the internal
host lists. The output for down hosts (in normal, grepable, and XML
forms) will come all in a block at the top of a host group, not
interleaved with up hosts as with ping scan. The output always goes to
XML and grepable formats, but requires -v to go to normal output.

Index: nmap.cc
===================================================================
--- nmap.cc (revision 15952)
+++ nmap.cc (working copy)
@@ -1747,6 +1747,9 @@
/* I used to check that !currenths->weird_responses, but in some
rare cases, such IPs CAN be port successfully scanned and even connected to */
if (!(currenths->flags & HOST_UP)) {
+ log_write(LOG_XML, "<host>");
+ write_host_header(currenths);
+ log_write(LOG_XML, "</host>\n");
delete currenths;
o.numhosts_scanned++;
continue;

David Fifield

[Andrew Smith]

Post by David Fifield

Post by David Fifield

Post by Andrew Smith
I'm currently working on a monitoring project using nmap as the
scanning/discovery engine.
The XML output is being parsed and updated on a MySQL database with a
PHP front end to display/filter the results.
Running nmap with -sP provides output of both hosts that are up and
nmap -sP 192.168.2.0/24 -oX nmap-sP-192.168.2.0-24.xml
....
<host><status state="down" reason="host-unreach"/>
<address addr="192.168.2.6" addrtype="ipv4" />
</host>
....
nmap -F --script smb-os-discovery 192.168.2.0/24 -oX
nmap-F-smb-os-discovery-192.168.2.0-24.xml
then I only get results for hosts that are "up".
I have tried using debug and verbosity (-d9 -vvv) but this doesn't
provide any information about "down" hosts.
I'm working around this at the moment by running a -sP first pass
followed by a -F second pass but it would be more efficient (and reduce
the load on the system) by being able to get the status of all hosts in
the -F pass.

This is the same problem that was reported at
http://seclists.org/nmap-dev/2009/q3/1081. Unfortunately there's
currently no way to get down hosts in the output when a scan includes a
port scan, script scan, or traceroute. The reason is that Nmap removes
the down hosts from its internal data structures before doing the
further scanning and output.
Depending on how much control you have over the parsing, you may be able
to work around the problem by doing the first pass with -sL rather than
-sP. That will add an entry for every host, with a state of "unknown".
You can then subtract the up hosts from the second pass to get the set
of down hosts.

Here's a patch for everybody's consideration. What it does is print out
a host record for down hosts before they are removed from the internal
host lists. The output for down hosts (in normal, grepable, and XML
forms) will come all in a block at the top of a host group, not
interleaved with up hosts as with ping scan. The output always goes to
XML and grepable formats, but requires -v to go to normal output.
Index: nmap.cc
===================================================================
--- nmap.cc (revision 15952)
+++ nmap.cc (working copy)
@@ -1747,6 +1747,9 @@
/* I used to check that !currenths->weird_responses, but in some
rare cases, such IPs CAN be port successfully scanned and even connected to */
if (!(currenths->flags& HOST_UP)) {
+ log_write(LOG_XML, "<host>");
+ write_host_header(currenths);
+ log_write(LOG_XML, "</host>\n");
delete currenths;
o.numhosts_scanned++;
continue;
David Fifield

Excellent, this is exactly what I was after - I can now scan and parse
the XML and get the state of the network in one pass.

It would be great if this could be committed to svn.

Many thanks,
Andrew

[Fyodor]

Post by David Fifield
Here's a patch for everybody's consideration. What it does is print out
a host record for down hosts before they are removed from the internal
host lists. The output for down hosts (in normal, grepable, and XML
forms) will come all in a block at the top of a host group, not
interleaved with up hosts as with ping scan. The output always goes to
XML and grepable formats, but requires -v to go to normal output.

Hi David. This is an interesting idea. I tested your simple patch,
and it seems to work. Here are my notes:

o In interactive mode, I see a bunch of lines like

Nmap scan report for 64.13.134.93
Host is down.
Nmap scan report for 64.13.134.95
Host is down.
Nmap scan report for 64.13.134.96
Host is down.

This is a similar issue to the one we dealt with for -sL where it was
using two lines per skipped host. In that case we removed the "Host
not scanned" lines, but I'm not sure what we should do here. Leaving
two lines per down host might be OK, though I'm tempted to add the
information to the scan report line like:

Nmap scan report for 64.13.134.95 [host down]
Nmap scan report for 64.13.134.96 [host down]

Up hosts could still be handled as they currently are:

Nmap scan report for gw.cust-cedera.svcolo.com (64.13.134.81)
Host is up (0.26s latency).

Requiring verbose mode, as you do, sounds reasonable for
normal/interactive output.

o In normal output (-oN), I get:
Nmap scan report for 64.13.134.158
Nmap scan report for 64.13.134.159
Nmap scan report for 64.13.134.160
Nmap scan report for 64.13.134.161

Here we only have one line per host, but it doesn't mention the host
state. I'd rather this be handled the same way as interactive output.

o XML output

A down host entry looks like:

<host><status state="down" reason="no-response"/>
<address addr="64.13.134.207" addrtype="ipv4" />
</host>

That is about 100 bytes, so we could have 10K down hosts per megabyte
of log file size. So a class B address space would take abotu 7 MB if
all the hosts are down. That sounds reasonable. I think your idea of
writing these to the XML files in all cases (e.g. without requiring
-v) is a good one. If people complain about space usage when they
scan sparse networks, we could always add an option to omit those.

o The grepable output looks like:
Host: 64.13.134.77 () Status: Down
Host: 64.13.134.78 () Status: Down
Host: 64.13.134.79 () Status: Down

That is fine.

o As you noted, the down hosts are presented in the "wrong" order. I
don't think that is a big problem.

So, overall I think it is a good patch. But befor it is applied I
think normal mode should be fixed to show the down status and we
should figure out what to do about interactive mode (one or two
lines).

Cheers,
Fyodor

[David Fifield]

Post by Fyodor

Post by David Fifield
Here's a patch for everybody's consideration. What it does is print out
a host record for down hosts before they are removed from the internal
host lists. The output for down hosts (in normal, grepable, and XML
forms) will come all in a block at the top of a host group, not
interleaved with up hosts as with ping scan. The output always goes to
XML and grepable formats, but requires -v to go to normal output.

Hi David. This is an interesting idea. I tested your simple patch,
o In interactive mode, I see a bunch of lines like
Nmap scan report for 64.13.134.93
Host is down.
Nmap scan report for 64.13.134.95
Host is down.
Nmap scan report for 64.13.134.96
Host is down.
This is a similar issue to the one we dealt with for -sL where it was
using two lines per skipped host. In that case we removed the "Host
not scanned" lines, but I'm not sure what we should do here. Leaving
two lines per down host might be OK, though I'm tempted to add the
Nmap scan report for 64.13.134.95 [host down]
Nmap scan report for 64.13.134.96 [host down]
Nmap scan report for gw.cust-cedera.svcolo.com (64.13.134.81)
Host is up (0.26s latency).
Requiring verbose mode, as you do, sounds reasonable for
normal/interactive output.
Nmap scan report for 64.13.134.158
Nmap scan report for 64.13.134.159
Nmap scan report for 64.13.134.160
Nmap scan report for 64.13.134.161
Here we only have one line per host, but it doesn't mention the host
state. I'd rather this be handled the same way as interactive output.
o XML output
<host><status state="down" reason="no-response"/>
<address addr="64.13.134.207" addrtype="ipv4" />
</host>
That is about 100 bytes, so we could have 10K down hosts per megabyte
of log file size. So a class B address space would take abotu 7 MB if
all the hosts are down. That sounds reasonable. I think your idea of
writing these to the XML files in all cases (e.g. without requiring
-v) is a good one. If people complain about space usage when they
scan sparse networks, we could always add an option to omit those.
Host: 64.13.134.77 () Status: Down
Host: 64.13.134.78 () Status: Down
Host: 64.13.134.79 () Status: Down
That is fine.
o As you noted, the down hosts are presented in the "wrong" order. I
don't think that is a big problem.
So, overall I think it is a good patch. But befor it is applied I
think normal mode should be fixed to show the down status and we
should figure out what to do about interactive mode (one or two
lines).

All right, this is committed now. Normal output shows down hosts with
the "[host down]" notation in one line.

David Fifield