[47364] in Cypherpunks
Re: Spiderspace
daemon@ATHENA.MIT.EDU (Bill Stewart)
Wed Jan 17 01:27:20 1996
Date: Tue, 16 Jan 1996 22:13:24 -0800
To: cypherpunks@toad.com
From: Bill Stewart <stewarts@ix.netcom.com>
>... I was under the impression that the only documents that most web crawlers
>will search are documents that are link-accessible. Are you saying that this
>isn't true? Are you saying that Alta-Vista will search EVERYTHING that's
>publicly accessible, whether by anonymous FTP or web?
Don't archie servers already pick up the anonymous ftp fairly well?
Also, aside from no-robots conventions, you can build a cgi program for
access to files that might be more effective at blocking searches
while still preserving access.
Also, it wouldn't be hard for a web-crawler to follow ftp links,
as long as the root of an anon-ftp site is pointed to by a URL somewhere.
#--
# Thanks; Bill
# Bill Stewart, stewarts@ix.netcom.com, Pager/Voicemail 1-408-787-1281
#
# "Eternal vigilance is the price of liberty" used to mean us watching
# the government, not the other way around....
