[5014] in WWW Security List Archive
RE: ROBOTS
daemon@ATHENA.MIT.EDU (Matthew Studley)
Thu Apr 10 11:36:15 1997
Date: 9 Apr 1997 09:52:44 U
From: Matthew Studley <matthew.studley@bt-sys.bt.co.uk>
To: Deep Summer - Home of Web Site Designs Extraordinare <frank@deepsummer.com>
Cc: www-security@ns2.rutgers.edu
Errors-To: owner-www-security@ns2.rutgers.edu
Hi,
you (Frank - 'Deep Summer') wrote;
>However, I think the main issue (it's fading) was to
>do with how well robots.txt files work. For benevolent
>bots, my logs indicate that robots.txt works wonderfully.
>For evil bots (remember Arnold? Okay, so he was a
>cybernetic organism...) there are other ways of dealing
>with security that have nothing at all to do with
>robots.txt.
I'm working in a field connected with mobile agents. Obviously, they (will)
present a whole load of security problems; hopping from machine to machine,
making decisions about their next destination, changing their internal state
(could be goal-related learning, could be viral infection).
As a starting point, can I ask you what methods you'd use to deal with today's
naughty bots? I admit, there's little similarity between repeated requests
from a distant site and tomorrow's malicious agent, but.....
Cheers, Matt.
BT Labs, UK.
