[129493] in North American Network Operators' Group
Re: yahoo crawlers hammering us
daemon@ATHENA.MIT.EDU (Ken Chase)
Wed Sep 8 12:20:51 2010
Date: Wed, 8 Sep 2010 12:20:07 -0400
From: Ken Chase <ken@sizone.org>
To: nanog@nanog.org
In-Reply-To: <AANLkTi=AHmvFAP9ZcYHG0ZHpw3oM3f33w6jSibc5=zFA@mail.gmail.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
On Wed, Sep 08, 2010 at 12:04:07AM -0700, Matthew Petach said:
>I *am* curious--what makes it any worse for a search engine like Google
>to fetch the file than any other random user on the Internet? In either case,
>the machine doing the fetch isn't going to rate-limit the fetch, so
>you're likely
>to see the same impact on the machine, and on the bandwidth.
I think that the difference is that there's a way to get to Yahoo and ask them
WTF. Whereas the guy who mass downloads your site with a script in 2 hrs you
have no recourse to (modulo well funded banks dispatching squads with baseball
bats to resolve hacking incidents). I also expect that Yahoo's behaviour is
driven by policy, not random assholishness (I hope :), and therefore I should
expect such incidents often. I also expect whinging on nanog might get me some
visiblity into said policy and leverage to change it! </dream>
/kc
--
Ken Chase - ken@heavycomputing.ca - +1 416 897 6284 - Toronto CANADA
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.
