ncat's nsock vs. nmap nsock

[[e-mail]]

Hi,

Currently, I'm working for Fyodor and GSoC on improving and
eventually merging ncat (the nmap netcat implementation) with
the main distribution. It currently resides in nmap-ext/ncat.

The main issue that prevents it from merging is that the nsock
version of ncat is a slightly modified version of the original nsock.
By diff'ing the 2006 nsock version against it (attached), I found
there's just one major difference, which is in do_actual_read():

- buflen = read(iod->sd, buf, sizeof(buf));
+ buflen = recv(iod->sd, buf, sizeof(buf), 0);

Ncat does a read() which is fully reliable, but nsock originally does a
recv() (which just differently, e.g. it is slightly lower level and more
errors have to be catched). The Bad News, Ncat, as designed, does
NOT work with a recv() in this place (doesn't receive any input), but
Good News: it DOES fully work with latest nsock from nmap CVS with
only a read() in place of the recv() (all features tested & working).

Without touching the original nsock implementation, which I
certainly don't want to, the straightforward solution I see would
be to clone a few functions from nsock_core.c in ncat which use
this low-level read: do_actual_read() handle_read_result()
iterate_through_event_lists() and nsock_loop(). That should be all.

Unless anyone comes up with a more beautiful solution, I would do
this (only) to the Ncat code, probably on the weekend, resulting in a
nmap-exp/ncat directory that produces a working ncat with externals
set to current nsock/nbase implementations. Other suggestions welcome
(and ideally, that should not be a full Ncat core rewrite, as I just have 2-3
months for adding a bunch of features within the GSoC project ;)

bye,
Mixter

[Fyodor]

Post by ***@gmail.com
By diff'ing the 2006 nsock version against it (attached), I found
- buflen = read(iod->sd, buf, sizeof(buf));
+ buflen = recv(iod->sd, buf, sizeof(buf), 0);

That is good news that the difference is so little! According to the
comment above that line in Nsock:

/* Traditional read() - no SSL - using recv() because that works
better on Windows */

So we did use read() there some years ago, but it didn't work properly
on Windows. Have you tested Ncat on Win?

Is Ncat not working at all with recv()? That seems strange, since
Nmap uses Nsock in similar ways with the same code path. For example,
version detection makes TCP connections on which it sends/receives
data, and also does UDP. NSE does the same. So I think Ncat *should*
be able to work with Nsock as is. But of course that doesn't explain
why it isn't :).

Can you research more what is causing Ncat to fail with our nsock?
I'd like to have Nmap and Ncat shre the same library code. Plus, even
if you import Nsock code into Ncat in order to use read(), you may
just encouter the Windows problems which caused us to abandon read()
years ago.

Cheers,
-F

[d***@hcsw.org]

Post by Fyodor
/* Traditional read() - no SSL - using recv() because that works
better on Windows */
So we did use read() there some years ago, but it didn't work properly
on Windows.

IIRC, unlike unix, win32 distinguishes between POSIX file descriptors
(int) and sockets (SOCKET). This goes all the way back to win3.1 winsock
and the consequence is that you can't use recv() to read from files (you
will get a WSAENOTSOCK error) and you can't use read() to read from
sockets (you will get an EBADF error). Also, you can't fdopen() sockets
for the same reason (sockets aren't file descriptors in win32).

http://msdn.microsoft.com/en-us/library/ms740121(VS.85).aspx
http://msdn.microsoft.com/en-us/library/wyssk1bs(VS.80).aspx

This is for the win32 API--I have no idea with .NET.

Doug

[[e-mail]]

Ok, I will research more this weekend into why this is failing. but
at the moment it is still a mystery to me. It ends up in trying
to recv() from FD -1... But it calls nsock_connect_tcp() and
nsock_read() just like nmap (though I haven't checked the
whole event handlers, there has got to be some differences):

connect(3, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("209.85.129.147")}, 16) = -1 EINPROGRESS
select(4, [3], [3], [3], {10, 0}) = 1 (out [3], left {9, 984000})
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
send(3, "", 0, 0) = 0
write(2, "Connected to 209.85.129.147:80\n", 31Connected to
209.85.129.147:80) = 31
dup(0) = 4
fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
gettimeofday({1210337816, 1524}, NULL) = 0
select(5, [3 4], [], [], NULL) = 1 (in [4])
gettimeofday({1210337820, 293207}, NULL) = 0
recv(4, 0xbfa3a0b8, 8192, 0) = -1 ENOTSOCK (Socket
operation on non-socket)

Post by Fyodor

Post by ***@gmail.com
By diff'ing the 2006 nsock version against it (attached), I found
- buflen = read(iod->sd, buf, sizeof(buf));
+ buflen = recv(iod->sd, buf, sizeof(buf), 0);

That is good news that the difference is so little! According to the
/* Traditional read() - no SSL - using recv() because that works
better on Windows */
So we did use read() there some years ago, but it didn't work properly
on Windows. Have you tested Ncat on Win?
Is Ncat not working at all with recv()? That seems strange, since
Nmap uses Nsock in similar ways with the same code path. For example,
version detection makes TCP connections on which it sends/receives
data, and also does UDP. NSE does the same. So I think Ncat *should*
be able to work with Nsock as is. But of course that doesn't explain
why it isn't :).
Can you research more what is causing Ncat to fail with our nsock?
I'd like to have Nmap and Ncat shre the same library code. Plus, even
if you import Nsock code into Ncat in order to use read(), you may
just encouter the Windows problems which caused us to abandon read()
years ago.
Cheers,
-F

[[e-mail]]

Ok... taking a second look at the strace I sent there... :)
The problem is actually clear, nsock has no problems with
network input, but with stdin input! dup(0) = 4, then
recv(4, ...) = -1 errno ENOTSOCK means that recv()
refuses to read from stdin as not being a real socket.

ncat.h:
struct conn_state {
nsock_iod tcp_nsi;
nsock_iod stdin_nsi;
[...]

So we have a minor design issue there, if we do not want to copy any
nsock code for nmap. My proposal is that we could have an additional
"real_socket"
field in "struct mspool" that takes care of making a difference
between non-network
(stdin) and network socket. read() only for stdin should even work on Win32. ;)
And only ncat would need to set that flag when initializing their
conn_state.stdin_nsi.

$.02

[Fyodor]

Post by ***@gmail.com
Ok... taking a second look at the strace I sent there... :)
The problem is actually clear, nsock has no problems with
network input, but with stdin input! dup(0) = 4, then
recv(4, ...) = -1 errno ENOTSOCK means that recv()
refuses to read from stdin as not being a real socket.

Good find. If nsock is not currently able to handle these different
types of descriptors transparently, I think it should be improved to
do so. The application calling Nsock shouldn't have to set a special
flag. I'm not sure of the best way to do this though. One option
would be that if recv() gives an ENOTSOCK error on a particular
socket, try again with read() and set a flag to do read in the future.

Nsock used to work with stdin and network sockets transparently. For
example, see nsock/examples/nsock_telnet.c. That small (~250 line)
sample Nsock app hasn't been updated for the latest Nsock API and so
it doesn't compile. But if you find a little spare time, you might
find it useful to do so. It is great for simple testing. If you do
update it, please also update the Makefile to compile it and check it
in. The change can go directly in /nmap/nsock/examples branch because
it is already broken so it would be hard to make things worse.

Cheers,
-F

[[e-mail]]

Done, nsock_telnet fixed. Should be a good testcase for fixing
nsock's stdin/stdout handling. Compile nmap before and you can
compile it with a 'make' from within nsock/examples now.

Your approach looks great, Fyodor (transparently handling ENOTSOCK).
Someone committing this eventually would be nice, to enable an official ncat
merge, till then I'm working with a read() in my local version...

./nsock_telnet 127.0.0.1 22
The event id is 8 -- initiating l00p
telnet_event_handler: Received callback of type CONNECT with status SUCCESS
Successfully connected to 127.0.0.1:5632 -- start typing lines
telnet_event_handler: Received callback of type READ with status SUCCESS
SSH-2.x-noneofyourbusiness
telnet_event_handler: Received callback of type READ with status ERROR
READ failed: Socket operation on non-socket

[Fyodor]

Post by ***@gmail.com
Done, nsock_telnet fixed. Should be a good testcase for fixing
nsock's stdin/stdout handling. Compile nmap before and you can
compile it with a 'make' from within nsock/examples now.
Your approach looks great, Fyodor (transparently handling ENOTSOCK).
Someone committing this eventually would be nice, to enable an official ncat
merge, till then I'm working with a read() in my local version...

Great. How about if you either send a patch to the list, or you could
make a branch of /nsock in /nmap-exp/mixter/nsock and then apply your
changes there.

Have you been able to test this on Windows yet?

Cheers,
-F