[4953] in WWW Security List Archive
ROBOTS
daemon@ATHENA.MIT.EDU (Jeff Middleton)
Mon Mar 31 13:30:36 1997
From: "Jeff Middleton" <jeffm@sgiserv3.aws.waii.com>
Date: Mon, 31 Mar 1997 07:50:06 -0800
To: www-security@ns2.rutgers.edu
Cc: jeffm@sgiserv3.aws.waii.com
Errors-To: owner-www-security@ns2.rutgers.edu
Is there any threat caused by allowing web indexing robots to enter your site?
I know there is a file "robots.txt" that can be setup, but is it necessary?
i.e. Will all robots respond to the file anyway?
Thanks for your help.
You can email me directly at jeff.middleton@waii.com
