[1708] in WWW Security List Archive
Re: Site Scaning & IP graps
daemon@ATHENA.MIT.EDU (James Ellison)
Sat Mar 23 01:34:18 1996
Date: Sat, 23 Mar 1996 12:55:13 +0900 (JST)
From: James Ellison <jellison@shrine.cyber.ad.jp>
To: KGANNON@dit.ie
cc: www-security@ns2.rutgers.edu
In-Reply-To: <01I2LP4HFQVMFKMH94@dit.ie>
Errors-To: owner-www-security@ns2.rutgers.edu
On Thu, 21 Mar 1996 KGANNON@dit.ie wrote:
> Can anyone tell me what the best way of detecting if a spider has had a look
> at your server i.e is there a list of common spiders.
>
> The story is there is a server here on site and while looking at the logs for
> my own page I saw a number of sites that I have never heard off. The thing
> is that there is no DNS entry for the server and it has not been advertised
> only 10 or so people on site even know it exists .
>
> I am just a little curious as to how people know about the server .
If you're running NSCA 1.5 or later, the Spiders etc. should log in the
browser-type field of your access logs, or in a separate agent logfile.
Other server types may log browsers in different places.
Do a keyword search on "Spider" (with grep or whatever). If you're being
touched by the web search bots you should see Lycos Spider,
ArchitextSpider, MOM Spider and few others show up. Keyword search on
"Robot" will probably turn up the Infoseek Robot.
If you're not seeing any of those logged, your site address may have been
put into someone's web page. Start checking the referrer link fields of
your web log (if the server logs them of course) to see where.
=======================================================================
James Ellison | Cyber Technologies International K.K.
=============================| General Manager, Tech Systems Dept.
jellison@cyber.ad.jp | Toranomon Suzuki Bldg, 3-20-4 Toranomon
tech@cyber.ad.jp | Minato-ku, Tokyo 150, Japan
http://www.cyber.ad.jp | Tel:+81-3-3578-4888 Fax:+81-3-3578-4889
=======================================================================
