[RFC] --exclude-ports option for Nmap

[Jay Bosamiya]

Hi All!

The --exclude-ports option would be a major boon to many Nmap users and
it has been in demand for quite a while now. However, there are some
things that need to be thought about before we add it in.

1. How does it interact with -p? More specifically, what does "-p 80
--exclude-ports 80" do (since user specifically included it as well as
excluded it)?
I think that we should follow a "exclude has higher priority than
include" ideology and NOT scan 80 in this case. However, we could show a
warning to a user if he has included a port individually and then
excluded it (i.e. not using ranges). The warning thing could be added
later on, as a follow up.

2. How does it interact with --top-ports? More specifically, how many
ports does "--top-ports 2 --exclude-ports 80" scan? 1 or 2?
I think that it should scan 2 ports excluding port 80 (i.e. it should
scan ports 23 and 443). This seems to be the logical way that anyone
would read the command too.

3. How should the arguments for it be taken? Similar to -p, or only
individual ports should be taken?
I think that we should allow for ranges of ports to also be
specified, very similar to the way we take input for -p (kind of like
"--exclude-ports 80-90,666"). This would make it very easy for users to
use this option.

4. Should there be a one-letter flag for it? If so, what should it be?
I don't think that it needs a one-letter flag but this depends on how
much it will be used.


I would request you all to comment on the above questions (whether you
agree with my point of view or not) and also add in other
questions/points that you find necessary/relevant.

Cheers,
Jay

[Jacek Wielemborek]

Post by Jay Bosamiya
Hi All!
The --exclude-ports option would be a major boon to many Nmap
users and it has been in demand for quite a while now.
However, there are some things that need to be thought about
before we add it in.
1. How does it interact with -p? More specifically, what does
"-p 80 --exclude-ports 80" do (since user specifically
included it as well as excluded it)?
I think that we should follow a "exclude has higher priority
than include" ideology and NOT scan 80 in this case. However,
we could show a warning to a user if he has included a port
individually and then excluded it (i.e. not using ranges). The
warning thing could be added later on, as a follow up.

IMHO the first case should be an error as there are no ports to
scan. I would personally find it perfect if it defaulted to
signalling an error instead of guessing. I was thinking of
suggesting that it should panic anytime the -p and --exclude
lists overlap, but that would make "-p- --exclude smtp"
impossible, so it's probably not a good idea.

Also, I believe that the warning message is critical to this
feature.

[John]

Hi Jay,

I like the suggestion and i think your logic on how to handle the flags
is what i would expect.

Post by Jacek Wielemborek

Post by Jay Bosamiya
Hi All!
The --exclude-ports option would be a major boon to many Nmap
users and it has been in demand for quite a while now.
However, there are some things that need to be thought about
before we add it in.
1. How does it interact with -p? More specifically, what does
"-p 80 --exclude-ports 80" do (since user specifically
included it as well as excluded it)?
I think that we should follow a "exclude has higher priority
than include" ideology and NOT scan 80 in this case. However,
we could show a warning to a user if he has included a port
individually and then excluded it (i.e. not using ranges). The
warning thing could be added later on, as a follow up.

IMHO the first case should be an error as there are no ports to
scan. I would personally find it perfect if it defaulted to
signalling an error instead of guessing. I was thinking of
suggesting that it should panic anytime the -p and --exclude
lists overlap, but that would make "-p- --exclude smtp"
impossible, so it's probably not a good idea.

To me the main use for this feature is doing things like nmap -p-
--exlude 9100 or nmap -p1-1000 --exclude 9. So i would be against
making this an error.

[Jay Bosamiya]

I think that this point may require some clear explanation.

Post by Jay Bosamiya
1. How does it interact with -p? More specifically, what does "-p 80
--exclude-ports 80" do (since user specifically included it as well as
excluded it)?
I think that we should follow a "exclude has higher priority than
include" ideology and NOT scan 80 in this case. However, we could show a
warning to a user if he has included a port individually and then
excluded it (i.e. not using ranges). The warning thing could be added
later on, as a follow up.

What I meant to say can be explained by the following examples:

(a) "-p 80 --exclude-ports 80" causes an error and Nmap quits (because
no ports to scan)
(b) "-p 80,81 --exclude-ports 80" causes a warning and only port 81 is
scanned (because 80 is specified individually, not in a range)
(c) "-p 79-81 --exclude-ports 80" causes NO warning and only ports 79
and 81 are scanned (because 80 is in a range)
(d) "-p 80 --exclude-ports 1-100" causes an error and Nmap quits
(because no ports to scan)
(e) "-p 80,1000-1200 --exclude-ports 1-100" causes a warning and only
ports 1000-1200 are scanned (because 80 is specified individually and
then excluded)
(f) "-p 80-120 --exclude-ports 1-100" causes NO warning and only ports
101-120 are scanned

It is impossible for us to really know what the user wants when there is
an ambiguity (including a port and then excluding it); however, we can
use the above set of examples to make a model that captures any obvious
mistakes a user might have made (look at point (a) and (d)) or silly
mistakes (look at point (b) and (e)) and proper usage (look at point
(c)). Point (f) is almost like an extension of point (c) and should be
considered to be correct usage.

I am pretty sure that all use cases can be covered by the above ("-p-
--exclude-ports 80" is just an extension of point (c)) and that this
would probably be the best way to go about the option.

I welcome any further feedback.

@Jacek, after thinking about it a little more, I agree that the warnings
are critical to the feature. However, the panicking each time there is
an overlap kind of destroys the point of the feature (as you correctly
recognized).

@John, I think you will find that the above cases match with what you
were talking about not making it an error. The only case we show an
error and quit is when we have no ports to scan with.

Cheers,
Jay

[John]

Post by Jay Bosamiya
@John, I think you will find that the above cases match with what you
were talking about not making it an error. The only case we show an
error and quit is when we have no ports to scan with.

Yes you have explained things well and obviously thought about it a lot
more then me. All the usage patterns you have describe seem logical to me

Thanks John

[Daniel Miller]

Jay,

Post by Jay Bosamiya
I think that this point may require some clear explanation.

Post by Jay Bosamiya
1. How does it interact with -p? More specifically, what does "-p 80
--exclude-ports 80" do (since user specifically included it as well as
excluded it)?
I think that we should follow a "exclude has higher priority than
include" ideology and NOT scan 80 in this case.

I agree--exclude should take priority over include.

Post by Jay Bosamiya
(a) "-p 80 --exclude-ports 80" causes an error and Nmap quits (because
no ports to scan)

Yes.

Post by Jay Bosamiya
(b) "-p 80,81 --exclude-ports 80" causes a warning and only port 81 is
scanned (because 80 is specified individually, not in a range)

I don't think this is necessary, because it will involve retooling the
portspec parsing code. It can be added later; perhaps a TODO comment in the
code would be sufficient. On the other hand, I lean against warnings of any
kind involving port exclusions because the user should be able to figure it
out (except for 10

Post by Jay Bosamiya
(c) "-p 79-81 --exclude-ports 80" causes NO warning and only ports 79
and 81 are scanned (because 80 is in a range)

This now becomes the same as (b)

Post by Jay Bosamiya
(d) "-p 80 --exclude-ports 1-100" causes an error and Nmap quits
(because no ports to scan)

This is probably the exact same code path as (a)

Post by Jay Bosamiya
(e) "-p 80,1000-1200 --exclude-ports 1-100" causes a warning and only
ports 1000-1200 are scanned (because 80 is specified individually and
then excluded)

Same as (b)

Post by Jay Bosamiya
(f) "-p 80-120 --exclude-ports 1-100" causes NO warning and only ports
101-120 are scanned

Yes.

A couple more:

(g) "-p 20-30 --exclude-ports 80-90" (disjoint) should probably cause no
warning, since the ports will not be scanned, just as requested.

(h) "-sSU -p 80 --exclude-ports U:80" should cause an error, because there
are no UDP ports available to scan, but the user requested a UDP scan.
Obviously this means that --exclude-ports should take the same format as -p

Post by Jay Bosamiya
2. How does it interact with --top-ports? More specifically, how many
ports does "--top-ports 2 --exclude-ports 80" scan? 1 or 2?
I think that it should scan 2 ports excluding port 80 (i.e. it should
scan ports 23 and 443). This seems to be the logical way that anyone
would read the command too.

I tend to think the opposite: --top-ports 2 --exclude-ports 80 should only
scan port 23, not 23 and 443, since --top-ports 2 would not normally scan
443. But I am willing to be persuaded by use cases for the other way
around. You will have to be careful of edge cases: --top-ports 65535
--exclude-ports 1-10 will not scan 65535 ports.

Post by Jay Bosamiya
3. How should the arguments for it be taken? Similar to -p, or only
individual ports should be taken?
I think that we should allow for ranges of ports to also be
specified, very similar to the way we take input for -p (kind of like
"--exclude-ports 80-90,666"). This would make it very easy for users to
use this option.

Yes, additionally it should take T: and U: and the other modifiers. This
probably means just reusing the existing parser (getpts in nmap.cc).

Post by Jay Bosamiya
4. Should there be a one-letter flag for it? If so, what should it be?
I don't think that it needs a one-letter flag but this depends on how
much it will be used.

I would not advocate for a 1-letter flag, but perhaps we should consider
what long option name gets used. The functions that handle this kind of
parsing and choosing are also used for the IP Protocol scan (-sO), so it
could fairly be called --exclude-protos. We don't have this problem with -p
simply because it is a 1-letter option. Suggestions, anyone?

Dan

[Fyodor]

Post by Jay Bosamiya
Hi All!
The --exclude-ports option would be a major boon to many Nmap users and
it has been in demand for quite a while now. However, there are some
things that need to be thought about before we add it in.
1. How does it interact with -p? More specifically, what does "-p 80
--exclude-ports 80" do (since user specifically included it as well as
excluded it)?

I think exclude ports should definitely overrule. I think the main thing
people will use this for is to remove a port from the default set, or from
-p-, etc.

"-p 80 --exclude-ports 80" is a special case because there are no ports
left to scan. So that might warrant an error suggesting that they use -sn
if they don't want to port scan.

I think that we should follow a "exclude has higher priority than

Post by Jay Bosamiya
include" ideology and NOT scan 80 in this case. However, we could show a
warning to a user if he has included a port individually and then
excluded it (i.e. not using ranges). The warning thing could be added
later on, as a follow up.

I'm not sure I'd even show a warning, as excluding ports which would
otherwise be scanned is a main purpose of this option. I also wouldn't
give a warning if people try to exclude a port which wasn't going to be
scanned, as they might just be doing that for safety/paranoia/policy
reasons.

Post by Jay Bosamiya
2. How does it interact with --top-ports? More specifically, how many
ports does "--top-ports 2 --exclude-ports 80" scan? 1 or 2?
I think that it should scan 2 ports excluding port 80 (i.e. it should
scan ports 23 and 443). This seems to be the logical way that anyone
would read the command too.

Yes, that sounds like the way to go. As you go through the top port lists,
you'd call the function to check if the port is excluded first.

3. How should the arguments for it be taken? Similar to -p, or only

Post by Jay Bosamiya
individual ports should be taken?
I think that we should allow for ranges of ports to also be
specified, very similar to the way we take input for -p (kind of like
"--exclude-ports 80-90,666"). This would make it very easy for users to
use this option.

Yeah, taking options in the same format as -p sounds reasonable, assuming
you can do it efficiently.

There is also the question of how this relates to NSE. One option is to
just use --exclude-ports to pare down the scanned port list at the
beginning of runtime, and not even need to keep the information around
after that. Alternatively, we could save the info and make a way for NSE
to query whether a port was specifically excluded. It might be best to
start with the former and then expand to the latter only if we find a
pressing need for it. Most NSE scripts aren't touching unscanned ports
anyway.

4. Should there be a one-letter flag for it? If so, what should it be?

Post by Jay Bosamiya
I don't think that it needs a one-letter flag but this depends on how
much it will be used.

I agree that it does not warrant a one-letter flag.

Cheers,
Fyodor

[Jay Bosamiya]

List,

[...] I don't think this is necessary, because it will involve
retooling the portspec parsing code. It can be added later; perhaps a
TODO comment in the code would be sufficient. On the other hand, I
lean against warnings of any kind involving port exclusions because
the user should be able to figure it out
[...] I'm not sure I'd even show a warning, as excluding ports which
would otherwise be scanned is a main purpose of this option. I also
wouldn't give a warning if people try to exclude a port which wasn't
going to be scanned, as they might just be doing that for
safety/paranoia/policy reasons.

I guess I completely scrap the idea of warnings then. We could just make
the documentation very clear, and this should be enough.


There is some confusion with respect to point number 2 in my original mail.

2. How does it interact with --top-ports? More specifically, how many
ports does "--top-ports 2 --exclude-ports 80" scan? 1 or 2?
I think that it should scan 2 ports excluding port 80 (i.e. it
should scan ports 23 and 443). This seems to be the logical way that
anyone would read the command too.
Yes, that sounds like the way to go. As you go through the top port
lists, you'd call the function to check if the port is excluded first.
I tend to think the opposite: --top-ports 2 --exclude-ports 80 should
only scan port 23, not 23 and 443, since --top-ports 2 would not
normally scan 443. But I am willing to be persuaded by use cases for
--top-ports 65535 --exclude-ports 1-10 will not scan 65535 ports.

Initially, the biggest reason I thought that we should scan exactly N
ports for "--top-ports N --exclude-ports A,B,C" in all cases (except for
edge cases) is that it reads more naturally (at least to me).
However, as I thought about it, I came up with a use case [1] that
clearly illustrates the reason we should keep it with exactly N ports.
I also decided that I could ask some folks as to what the command really
reads as [2].

[...] perhaps we should consider what long option name gets used. The
functions that handle this kind of parsing and choosing are also used
for the IP Protocol scan (-sO), so it could fairly be called
--exclude-protos. We don't have this problem with -p simply because it
is a 1-letter option. Suggestions, anyone?

I'm not so sure, but how about --exclude-p ? This basically comes from
the reasoning that we use --exclude for hosts since hosts have no option
name before them, so we use --exclude-p for the things that have -p
before them.
This could be quite debatable actually, but it is a triviality in
relation to the rest of the code. I will code in the option as
--exclude-ports for now (only because of the subject of the thread), but
we could decide the exact name anytime later before we commit it.


Cheers,
Jay

References/Links:
[1] Hypothetical Scenario: If I am a network admin who scans my network
daily (thoroughly) and hourly (just for top ports) to see if any
services/ports have opened up inadvertently/maliciously. I would have a
limited time span for which I can run the hourly scan and my top-ports
number is limited by this. I would have written a script that runs
"--top-ports N" and would look for any new additions that pop up. Now
that an exclude-ports option is there in Nmap, I'd just add all the
ports that I know to be open (since I know which services I've
installed) to the exclude-ports list and make the script run with a
"--exclude-ports A,B,C" tagged along (to be able to scan more ports that
should not be open in the same amount of time). Now, due to some new
software that I have installed, there is a new port opened. Then I'd add
a ",D" to the list and be done with it. This would basically scan the
same number of ports but would actually be a more informative scan. On
the other hand, if I had to make it "--top-ports N+4 --exclude-ports
A,B,C,D" then I had to either waste time checking that D is actually in
the top ports list (and then change to N+4) or I would forget about
changing it to N+4 which basically means that the time taken by the scan
is lesser (and time that could've been used to check for one more rogue
service goes to waste).
[2]
http://english.stackexchange.com/questions/177252/how-to-interpret-top-n-except-a-b-c

[Daniel Miller]

Post by Jay Bosamiya
I would have a
limited time span for which I can run the hourly scan and my top-ports
number is limited by this.

This sentence was enough to change my opinion on the topic. --top-ports is
semantically an "optimize my time" option, which is why the "fast scan"
option (-F) is an alias for --top-ports 100. If --exclude-ports reduces the
total number of ports scanned, then it changes the tradeoff calculus, which
we should probably avoid.

I think you have enough to start coding now.

Dan