[161404] in North American Network Operators' Group
Re: Bing/MSN/Microsoft contact
daemon@ATHENA.MIT.EDU (Jay Farrell)
Tue Mar 12 13:30:13 2013
In-Reply-To: <CAA5Ek4fRO06W4g+RWk=qvfR=UyXesf0HU1HC5ECtfGPaEeTvgg@mail.gmail.com>
From: Jay Farrell <jayfar@jayfar.com>
Date: Tue, 12 Mar 2013 13:29:40 -0400
To: Blair Trosper <blair.trosper@gmail.com>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
This might help:
http://www.bing.com/webmaster/help/how-to-report-an-issue-with-bingbot-25c1=
9802
How do I Report an Issue With Bingbot?
Bingbot is the name of the crawler used by Bing to crawl or =93spider=94 th=
e
web. It is Bingbot's job to find new and updated pages on websites across
the Internet, so that they can be processed for indexation. When crawling a
website, Bingbot looks at robots.txt for special instructions from the
website owner. Bingbot honors robots.txt directives, including the *
crawl-delay:* setting, and, in the absence of a crawl-delay, respects the
input from Webmasters in the Crawl Control Feature.
Generally, Bingbot does a good job at determining how frequent it should
visit pages on your site, taking robots.txt and Crawl Control rules and
hints into consideration. We call this =93Crawl Politeness=94. However, the=
re
still may be cases were you feel Bingbot is not polite enough and is
visiting your pages more than works for you (overcrawling).
REPORTING OVERCRAWLING
Here=92s the steps you can follow if you think Bingbot is overcrawling your
site or not observing robots.txt rules:
1. Verify that the bot traffic you are seeing is in fact from a valid
Bingbot server. You can do this by not only looking at the User-Agent
string (which can be easily spoofed by anyone) but also at the IP addres=
s
and using the *Verify Bingbot
tool<http://www.bing.com/toolbox/verify-bingbot>
* to get a verdict
2. Once you have verified that this involves genuine Bingbot traffic,
you can reduce crawler traffic as follows (if you haven=92t done so alre=
ady):
3. Lower the speed of crawl during busy hours using the Crawl Control
feature
4. If that=92s not sufficient, add a crawl-delay: directive to your
robots.txt: Bing supports whole number values ranging from 1 =96 30. Eac=
h
number maps to the number of seconds we need to wait between requests, s=
o 1
means a maximum of 1 request per 1 second =96 which is slow, but still
adequate for smaller sites. 30 is extremely slow and means we are allowe=
d
only 1 request per 30 seconds.
5. If you have followed step 1 and 2 and the issue is still present, you
can contact *Bing Webmaster
Support<http://go.microsoft.com/fwlink/p/?linkid=3D261881>
*. Fill out the required fields and in the =93*What type of problem do y=
ou
have?=94 *dropdown, select =93*Under-Crawling or Over-Crawling inquiry*=
=94 and
describe the problem you are seeing. You can expect a reply within 24-48
hours. When you report over-crawling issues, the support team will ask y=
ou
to provide server log samples that show the Bingbot activity over a cert=
ain
period of time in a next step, so make sure to have those ready.
On Tue, Mar 12, 2013 at 11:57 AM, Blair Trosper <blair.trosper@gmail.com>wr=
ote:
> If possible, I need someone from Microsoft/Bing (a la the MSN and Bing
> crawler bots) to contact me off list.
>
> Several IPs going back to AS8075 (with the user agent MSN and "bingbot")
> are basically attacking several IPs on my network with hundreds of reques=
ts
> per second.
>
> Thanks,
> Blair
>
