[47330] in Cypherpunks
Re: Spiderspace
daemon@ATHENA.MIT.EDU (John F. Fricker)
Tue Jan 16 19:15:40 1996
Date: Tue, 16 Jan 1996 15:10:23 -0800
To: cypherpunks@toad.com
From: jfricker@vertexgroup.com (John F. Fricker)
At 11:38 AM 1/16/96 -0800, you wrote:
>
>I've been thinking a lot about the problems and opportunities that are
>coming up as more and more "spiders" (Web searchers, crawlers) are indexing
>directories and files on systems they can find.
<a snip of agreement>
I haven't checked my logs for Alta Vista but many of the spiders do follow the robot exclusion standard. A simple text file that should be retrieved first by any spider that explains where one may not go.
The standard is fully explained at:
http://info.webcrawler.com/mak/projects/robots/norobots.html
But this does nothing for those items sitting in more traditional public space. Obscurity is no longer security.
