[4970] in WWW Security List Archive
RE: ROBOTS
daemon@ATHENA.MIT.EDU (DeepSummer-HomeofWebSiteDesignsExt)
Tue Apr 1 22:25:17 1997
From: Deep Summer - Home of Web Site Designs Extraordinare
<frank@deepsummer.com>
To: "'www-security@ns2.rutgers.edu'" <www-security@ns2.rutgers.edu>
Cc: "'frank'" <frank@deepsummer.com>
Date: Tue, 1 Apr 1997 17:23:58 -0700
Errors-To: owner-www-security@ns2.rutgers.edu
I can't speak for everyone, certainly, but I've had good luck
(as far as I can tell *wink*) with my robots.txt file. I get
an incredible number of robot hits, and the majority do seem
to honor robots.txt, or at least they seem to be paying heed
to mine.
If you want a copy it's at http://www.deepsummer.com/robots.txt
(should be able to do a shift-click on it to retrieve). If
not, let me know and I'll mail you a copy.
Also, if anyone wants to take a peek at it and let me know if
you see anything I might have done better then by all means
do so.
Anyway, if my HTTPd (Apache) logs are any indication, most do
seem to be respectable.
-frank
----------
From: Jeff Middleton[SMTP:jeffm@sgiserv3.aws.waii.com]
Sent: Monday, March 31, 1997 8:50 AM
To: www-security@ns2.rutgers.edu
Cc: jeffm@sgiserv3.aws.waii.com
Subject: ROBOTS
Is there any threat caused by allowing web indexing robots to enter your site?
I know there is a file "robots.txt" that can be setup, but is it necessary?
i.e. Will all robots respond to the file anyway?
Thanks for your help.
You can email me directly at jeff.middleton@waii.com
