[8262] in www-talk@info.cern.ch
Re: Byte ranges (actually robots)
daemon@ATHENA.MIT.EDU (Gavin Nicol)
Fri May 26 04:21:22 1995
Date: Thu, 25 May 1995 10:15:37 +0500
Errors-To: procmaster@www19.w3.org
Reply-To: gtn@ebt.com
From: Gavin Nicol <gtn@ebt.com>
To: Multiple recipients of list <www-talk@www10.w3.org>
>So what to do? If we recognize the user-agent as a known robot, we give it
>a flat presentation of the structure. The catch of course, is that there
>is no standard way to recognize a robot. Some convention in the UserAgent
>field would be sufficient, but....
Isn't there a standard permissions file that robots can fetch?
