[5018] in WWW Security List Archive
RE: ROBOTS
daemon@ATHENA.MIT.EDU (DeepSummer-HomeofWebSiteDesignsExt)
Thu Apr 10 13:58:35 1997
From: Deep Summer - Home of Web Site Designs Extraordinare
<frank@deepsummer.com>
To: Deep Summer - Home of Web Site Designs Extraordinare
<frank@deepsummer.com>,
"'Matthew Studley'"
<matthew.studley@bt-sys.bt.co.uk>
Cc: "www-security@ns2.rutgers.edu" <www-security@ns2.rutgers.edu>
Date: Wed, 9 Apr 1997 03:15:40 -0600
Errors-To: owner-www-security@ns2.rutgers.edu
Well, personally I make my number one defence a goal
of never leaving anything susceptable to security
violation in any place that gives it that susceptablity
state unless absolutely neccesary.
My next step would be putting up enough of a defense to
make it difficult for any evil agents. Sort of like the
front door philosophy - if someone really wants to
break in bad enough, the standard front door is not going
to stop them - even if you have no windows.
Finally, for extremely sensitive data that absolutely
has to be accessible via the net, I'd post questions
to this newsgroup asking for suggestions. The reason?
Well, quite frankly, I've not yet had the need to have
anything that sensitive online. You could wipe out my
entire site right now and I'd have it back online in
a matter of about an hour or two, and nobody would have
been hurt in the process because no sensitive data is
there. I guess I just insinuated the notion that backing
up one's data is a good thing. I mirror, 100%, everything
I do for myself and others, on a local non-net-accessible
storage device. That's obviously not a solution for
everyone, however.
That's one reason I joined this list, though - I'm wanting
to try and become more 'security knowledgable'. As it
stands, I now use only basic authentication for areas I
prefer keeping away from all eyes, but, my understanding
of basic auth (even the HTTP/1.0 RFC mentions this) is
that it's not a security solution. Perhaps said, it's more
of a detterant and a way to have a little bit of control
over your data.
I'd be curious to see responses to your query from others
on this list whom I know are a hundrefold more security
knowledgeable than myself. That would be good reading for
me as well.
Sincerely,
-frank
----------
From: Matthew Studley[SMTP:matthew.studley@bt-sys.bt.co.uk]
Sent: Wednesday, April 09, 1997 3:52 AM
To: Deep Summer - Home of Web Site Designs Extraordinare
Cc: www-security@ns2.rutgers.edu
Subject: RE: ROBOTS
Hi,
you (Frank - 'Deep Summer') wrote;
>However, I think the main issue (it's fading) was to
>do with how well robots.txt files work. For benevolent
>bots, my logs indicate that robots.txt works wonderfully.
>For evil bots (remember Arnold? Okay, so he was a
>cybernetic organism...) there are other ways of dealing
>with security that have nothing at all to do with
>robots.txt.
I'm working in a field connected with mobile agents. Obviously, they (will)
present a whole load of security problems; hopping from machine to machine,
making decisions about their next destination, changing their internal state
(could be goal-related learning, could be viral infection).
As a starting point, can I ask you what methods you'd use to deal with today's
naughty bots? I admit, there's little similarity between repeated requests
from a distant site and tomorrow's malicious agent, but.....
Cheers, Matt.
BT Labs, UK.
