[8274] in www-talk@info.cern.ch
Re: Byte ranges (actually robots)
daemon@ATHENA.MIT.EDU (Gavin Nicol)
Fri May 26 17:44:46 1995
Date: Thu, 25 May 1995 16:11:55 +0500
Errors-To: procmaster@www19.w3.org
Reply-To: gtn@ebt.com
From: Gavin Nicol <gtn@ebt.com>
To: Multiple recipients of list <www-talk@www10.w3.org>
>>Isn't there a standard permissions file that robots can fetch?
>
>Yes, but that just let's me tell them what to stay away from. The issue
>here is not that they should stay away, but that I should recognize them.
Fair enough. I'd also like to recognise them, and some kind of
convention in User-Agent or somesuch seems desirable.
Putting a robot on our server is a great way of stress-testing it ;-)
