[4966] in WWW Security List Archive
Re: ROBOTS
daemon@ATHENA.MIT.EDU (Lars Eilebrecht)
Tue Apr 1 16:37:03 1997
From: Lars Eilebrecht <Lars.Eilebrecht@unix-ag.org>
To: www-security@ns2.rutgers.edu
Date: Tue, 1 Apr 1997 18:35:07 +0200 (CEST)
In-Reply-To: <9703310750.ZM15252@sgiserv3.aws.waii.com> from "Jeff Middleton" at Mar 31, 97 07:49:46 am
Errors-To: owner-www-security@ns2.rutgers.edu
According to Jeff Middleton:
> Is there any threat caused by allowing web indexing robots to enter your site?
> I know there is a file "robots.txt" that can be setup, but is it necessary?
> i.e. Will all robots respond to the file anyway?
http://info.webcrawler.com/mak/projects/robots/robots.html
Regards...
--
Lars Eilebrecht
sfx@unix-ag.org
