[92507] in North American Network Operators' Group
Re: Potentially on-Topic: is MSNBot for real?
daemon@ATHENA.MIT.EDU (Mark Foster)
Fri Sep 22 20:18:12 2006
Date: Sat, 23 Sep 2006 12:09:59 +1200 (NZST)
From: Mark Foster <blakjak@blakjak.net>
To: Simon Waters <simonw@zynet.net>
Cc: nanog@merit.edu
In-Reply-To: <200609221310.22668.simonw@zynet.net>
Errors-To: owner-nanog@merit.edu
On Fri, 22 Sep 2006, Simon Waters wrote:
> On Friday 22 Sep 2006 11:39, you wrote:
>>
>> Is this unusual, or what? Are search engines supposed to be amongst the
>> biggest user agents recorded on a typical website? How much trolling and
>> indexing is considered 'too much' ?
>
> Whenever it becomes a problem.
>
> If you don't have enough genuine traffic, and you don't have much, then the
> search engines will look like they are dominating it, as they are pretty
> thorough.
I spose its all about scale. In a country of 4 million odd people, a
website with a domestic focus in a niche area - 40,000 hits in 21 days is
'fair' IMHO.
> I've seen issues arise with some search bots, where they have discovered loops
> in a websites structure and downloaded multiple copies, or found novels links
> to dynamic content and indexed your entire database. So worth checking what
> pages they have been to, to see if those could be an issue.
Good point. Thanks for the pointer.
>
>> Off-list thoughts on this welcome if the operational relevance of this
>> issue is questioned...
>
> Trust me, anything involving 40,000 hits is off-topic in Nanog, unless you
> have reason to believe the same 40,000 are happening to everyone on the net,
> or they took down 40,000 important websites.
Seeing stats on sites much bigger than my own helps put perspective on it,
so i'm already grateful for those who've responded.
>
> Most of the regular are just getting in, so expect to be flamed mercilessly.
Anythings gotta be better tham beating on Gadi, right?
=)
Mark.
