[1699] in WWW Security List Archive
Re: Site Scaning & IP graps
daemon@ATHENA.MIT.EDU (Brian W. Spolarich)
Thu Mar 21 22:38:49 1996
Date: Thu, 21 Mar 1996 19:33:49 -0500 (EST)
From: "Brian W. Spolarich" <briansp@ans.net>
To: KGANNON@dit.ie
cc: www-security@ns2.rutgers.edu
In-Reply-To: <01I2LP4HFQVMFKMH94@dit.ie>
Errors-To: owner-www-security@ns2.rutgers.edu
Good spiders will ask for /robots.txt and find out what to do with
themselves if they find it.
Generally grepping for /robots.txt will give you a list of spiders that
have found you.
-brian
On Thu, 21 Mar 1996 KGANNON@dit.ie wrote:
> Date: Thu, 21 Mar 1996 13:44:27 +0000 (GMT)
> From: KGANNON@dit.ie
> To: www-security@ns2.rutgers.edu
> Subject: Site Scaning & IP graps
>
> Can anyone tell me what the best way of detecting if a spider has had a look
> at your server i.e is there a list of common spiders.
>
> The story is there is a server here on site and while looking at the logs for
> my own page I saw a number of sites that I have never heard off. The thing
> is that there is no DNS entry for the server and it has not been advertised
> only 10 or so people on site even know it exists .
>
> I am just a little curious as to how people know about the server .
>
--
Brian W. Spolarich - ANS CO+RE Systems - briansp@ans.net - (313)677-7311
Her vocabulary was as bad as, like, whatever.
