[88002] in North American Network Operators' Group
Re: GoDaddy.com shuts down entire data center?
daemon@ATHENA.MIT.EDU (Jay Hennigan)
Tue Jan 17 04:47:13 2006
Date: Tue, 17 Jan 2006 01:42:24 -0800
From: Jay Hennigan <jay@west.net>
To: "Patrick W. Gilmore" <patrick@ianai.net>
Cc: nanog@nanog.org
In-Reply-To: <F72A0A9C-7901-4DFB-92FF-F974B037F02A@ianai.net>
Errors-To: owner-nanog@merit.edu
Patrick W. Gilmore wrote:
>
> On Jan 17, 2006, at 1:32 AM, Jim Popovitch wrote:
>
>> I want to say, from an outsider's perspective, that I whole heartily
>> applaud GoDaddy on the actions they took [...]
>
>
> There seems to be a wide split on this topic. I was wondering if
> people would privately tell me yes or no on a few questions so I can
> understand the issue better.
>
> 1) Do you think it is acceptable to cause any collateral damage to
> innocent bystanders if it will stop network abuse?
In some cases. Our policy is to minimize such. Example: Customer has
a NATted network with multiple machines sharing one global address. One
of the machines at customer's premise is causing abuse (virus, etc.)
Null-routing one specific IP address will cause collateral damage to the
non-infected machines at that customer, but I think most of here would
agree that such is justified. Obviously, if the impact of the abuse is
minimal, having the customer fix the problem before shutting anything
down is preferred. Another example would be a customer's webserver
which has many name-based virtual hosts, one of which is abusive, and
you are providing IP connectivity. By null-routing one IP you are
causing collateral damage to the non-abusive virtual host customers of
your customer, but I think most would think that justified.
> 2) If yes, do you still think it is acceptable to take down 100s of
> innocent bystanders because one customer of a provider is misbehaving?
I assume here that you mean "Customer of a customer". Again, it
depends. If the customer has continual problems controlling abuse from
his customers, or you suspect that your customer is playing
"whack-a-mole", or the abuse is ongoing and/or serious and you can't
identify which of customer's customers is the cause (spoofed source
addresses, etc.) in some cases yes.
> 3) If yes, do you still think it is acceptable if the "misbehaving"
> customer is not intentionally misbehaving - i.e. they've been hacked?
Again, it depends on the seriousness of the abuse and its affect on the
network, as well as the frequency thereof and the seriousness of the
customer in rectifying the problem. Also whether you can reasonably
isolate the abuse and disconnect only the customer's abusive customer.
> 3) If yes, do you still think it is acceptable if the collateral damage
> (taking out 100s of innocent businesses) doesn't actually stop the spam
> run / DoS attack / etc.?
If it doesn't stop it but stops your network from being a part of it,
yes. If it has no affect on it at all, then you're probably pulling the
wrong plug.
> These are important question to me, and I'm surprised at the number of
> people who seem to feel so very differently than I thought they would
> feel - than I personally feel. Would people mind sending me private
> e-mails with yes/no answers? Longer answers are welcome, but yes/no
> will do.
This is IMHO operational, so posting publicly. I don't think this is as
black-and-white as to warrant simple yes-no answers. There are policies
involved as well as your agreements with your peers/upstreams. If the
issue is serious enough that you risk losing your own connectivity
because you can't stem the abuse from a customer's customer, then you
may need to do so, or the end result will be that you become part of
greater collateral damage.
> Using the case under discussion as an example, I am wondering why
> anyone thinks taking down 100s of innocent domains is a good way to
> stop a single hacked machine from doing whatever it is doing? If you
> somehow think all that is worth it, take a close look at your cost /
> benefit analysis. At this rate, every business on the Internet will be
> out of business before we take out even a single moderately large botnet.
The present example seems to be a combination of poor communication, bad
attitude and sloppy network design from what I've seen here. It's
unclear to me exactly what GoDaddy shut down, and the only data points
we have to go on are admittedly edited conversations that took place
after the plug was pulled. What went on beforehand? Did Nectar indeed
make a good faith effort to correct the original problem? Was their
attitude the same as shown on the phone calls? How long had the problem
existed, had it happened before, and did Nectar keep an open dialogue as
to the steps they were taking to fix it? Did GoDaddy have less
intrusive options to shut down just the abuser?
> I am also wondering why anyone thinks the miscreant will stop just
> because the legitimate owner's domain no longer resolves? Not only is
> the machine likely to continue sending spam as if nothing happened, we
> aren't even "catching" the guy. I guess you could say "well, it put
> pressure on his hosting provider to clean the infected machine", which
> is true. I just think that's a bit silly. But maybe I'm the one who's
> silly.
I think this was a case of a fake phishing website rather than outgoing
spam spew. If the domain was the target of a phish, then causing it not
to resolve would keep the phisher from reaping any benefit from the
abuse although the spam run would likely continue, at least for a while
until the phisher realizes it is in vain.
> Lastly, I wonder what "average" people - people who run businesses on
> hosting providers who really don't understand all this computer stuff -
> think about such actions. How many 100s of people have we just
> alienated for life to stop - er, NOT stop - a single zombie? And how
> many of their friends are going to hear over an over how the Internet
> is not a real business and no one should put any faith in it?
Well, "average" people who run businesses on hosting providers" probably
should hire someone who does understand all this computer stuff to do
some due diligence on the providers they are considering. If their
prospective providers netblocks are repeatedly mentioned in SPEWS,
Spamhaus, Spamcop, and NANAE, they may want to look elsewhere.
Googling "Nectartech abuse" is interesting. As far back as July of last
year they were battling GoDaddy over spam and abuse issues. It doesn't
look like this should have been all that big of a surprise. In fact,
Nectartech's predictions in post 23 of the following thread are eerily
accurate.
http://www.webhostingtalk.com/showthread.php?s=&threadid=422612
> Is this really a good thing?
If steps are taken to minimize collateral damage, yes. Allowing the
abuse to continue causes collateral damage to the rest of the Internet
for as long as it continues. The choice often boils down to severe
collateral damage to a few or raising the noise level and collateral
damage to the Internet as a whole. Is cutting off ten customers of an
infected customer better than allowing this customer's virus to infect
tens of thousands of random hosts on the net worth it? If you're one of
the tens of thousands, yes. If you're one of the ten customers, no.
--
Jay Hennigan - CCIE #7880 - Network Administration - jay@west.net
NetLojix Communications, Inc. - http://www.netlojix.com/
WestNet: Connecting you to the planet. 805 884-6323
