[1707] in WWW Security List Archive
Re: Site Scaning & IP graps
daemon@ATHENA.MIT.EDU (Dennis Boone)
Fri Mar 22 19:12:24 1996
To: www-security@ns2.rutgers.edu
From: Dennis Boone <drb@gopher.cl.msu.edu>
In-Reply-To: (Your message of Thu, 21 Mar 96 19:33:49 EST.)
<Pine.SOL.3.91.960321193127.9027Y-100000@thebrain.aa.ans.net>
Date: Fri, 22 Mar 96 15:45:09 -0500
Errors-To: owner-www-security@ns2.rutgers.edu
> Good spiders will ask for /robots.txt and find out what to do with
> themselves if they find it.
>
> Generally grepping for /robots.txt will give you a list of spiders that
> have found you.
The access log typically doesn't have the agent name, just the host name
that called for the file. Some are obvious (fourteen.srv.lycos.com seems
to be visiting me today), others not (204.162.98.47 ??).
While it is possible to have all of the information in one log file with
the NCSA server, I'd bet most folks aren't using the feature. I don't
know about other servers.
Dennis Boone
MSU CWIS Team
