[159620] in North American Network Operators' Group
Re: Intermittent incorrect DNS resolution?
daemon@ATHENA.MIT.EDU (George Herbert)
Wed Jan 16 17:13:14 2013
In-Reply-To: <50F722FD.7010201@uberflip.com>
Date: Wed, 16 Jan 2013 14:13:02 -0800
From: George Herbert <george.herbert@gmail.com>
To: Erik Levinson <erik.levinson@uberflip.com>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
On Wed, Jan 16, 2013 at 2:00 PM, Erik Levinson
<erik.levinson@uberflip.com> wrote:
> Hi everyone,
>
> I'm having an unusual DNS problem and would appreciate feedback.
>
> For the zones in question, primary DNS is provided by GoDaddy and
> secondary DNS by DNS Made Easy. Over a week ago we made changes to
> several A records (including wildcards on two different zones), all
> already having a TTL no greater than one hour.
>
> The new IPs on those A records have taken many millions of requests
> since the changes. Occasionally, a small amount of traffic appears at
> the old IPs that those A records had. This is HTTP traffic. Packet
> captures of this traffic show various Host headers.
>
> Attempting to resolve those various Host headers from various networks
> in Canada against various random private and public resolvers and
> against the authoritative NSs all yield correct results (i.e. new IPs).
>
> However, both GoDaddy and DNS Made Easy use anycast, which makes it less
> likely that I can see the entire picture of what's happening.
>
> I suspect that somewhere, one of their servers has the wrong data, or
> some resolver is misbehaving, but based on the
> pattern/traffic/volume/randomization of hostnames, the resolver theory is
> less likely. I haven't analyzed the source IPs yet to see if they're in a
> particular set of countries.
>
> I've opened a ticket with DNS Made Easy and they replied very quickly
> suggesting the problem is not with them. I've opened a ticket with
> GoDaddy and...well, it's GoDaddy, so I don't expect much (no response yet).
>
> Any ideas? Can folks try resolving eriktest.uberflip.com and post
> here with details only if it resolves to an IP starting with 76.9 (old IPs)?
>
>
> Thanks
>
> Erik
The other likely cause of this is local cacheing nameservers somewhere
at some ISP or major site, that do not respect TTL values for some
reason.
This is sadly a common problem - not statistically, most nameservers
do the right thing, but if you run big sites and flip things, there's
always a long tail of people whose nameservers just didn't get it.
--
-george william herbert
george.herbert@gmail.com
