[8944] in athena10
Re: Comcast's DNS servers + Kerberos + dialup.mit.edu = fail
daemon@ATHENA.MIT.EDU (Geoffrey Thomas)
Thu Apr 19 03:39:58 2012
Date: Thu, 19 Apr 2012 03:39:53 -0400 (EDT)
From: Geoffrey Thomas <geofft@MIT.EDU>
To: John Hawkinson <jhawk@MIT.EDU>
cc: debathena@MIT.EDU
In-Reply-To: <20120418001741.GT11594@multics.mit.edu>
Message-ID: <alpine.DEB.2.00.1204190311010.11232@dr-wily.mit.edu>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
On Tue, 17 Apr 2012, John Hawkinson wrote:
> Geoffrey Thomas <geofft@MIT.EDU> wrote on Tue, 17 Apr 2012
> at 02:37:30 -0400 in <alpine.DEB.2.00.1204170214400.8722@dr-wily.mit.edu>:
>
>> Comcast's residential DNS IPs (75.75.75.75 and 75.75.76.76) are
>> anycast addresses for muliple machines, which means that when
>> looking up *.dialup.mit.edu, which is load-balanced at the DNS
>> level, there's no guarantee that two successive client lookups
>> (from a stub resolver) will hit the same server, and thus are more
>
> This seems a bit weird to me. Anycast addressing shouldn't mean you
> get a different anycast machine for two sucessive queries; is
> that what you're saying is happening?
Yes. From residential Comcast in SF, a couple of `host
$RANDOM.ns.ldpreload.com`s give the following tcpdump output:
geofft@water-buffalo:~$ sudo tcpdump -n port 53
[...]
03:16:08.041698 IP 69.252.96.25.41559 > 18.181.0.115.53: 45875 [1au] A? 2083.ns.ldpreload.com. (50)
03:16:08.042058 IP 18.181.0.115.53 > 69.252.96.25.41559: 45875 NXDomain*- 0/1/1 (114)
03:16:08.661893 IP 68.87.76.189.59278 > 18.181.0.115.53: 30465 [1au] A? 2083.ns.ldpreload.com. (50)
03:16:08.662267 IP 18.181.0.115.53 > 68.87.76.189.59278: 30465 NXDomain*- 0/1/1 (114)
03:17:39.023066 IP 68.87.76.181.17646 > 18.181.0.115.53: 56086 [1au] A? 14090.ns.ldpreload.com. (51)
03:17:39.023921 IP 18.181.0.115.53 > 68.87.76.181.17646: 56086 NXDomain*- 0/1/1 (115)
03:17:39.557971 IP 69.252.96.23.54191 > 18.181.0.115.53: 33966 [1au] A? 14090.ns.ldpreload.com. (51)
03:17:39.558553 IP 18.181.0.115.53 > 69.252.96.23.54191: 33966 NXDomain*- 0/1/1 (115)
03:17:41.679114 IP 69.252.96.24.62110 > 18.181.0.115.53: 41761 [1au] A? 1724.ns.ldpreload.com. (50)
03:17:41.679486 IP 18.181.0.115.53 > 69.252.96.24.62110: 41761 NXDomain*- 0/1/1 (114)
03:17:41.897269 IP 69.252.96.25.47875 > 18.181.0.115.53: 15742 [1au] A? 1724.ns.ldpreload.com. (50)
03:17:41.897601 IP 18.181.0.115.53 > 69.252.96.25.47875: 15742 NXDomain*- 0/1/1 (114)
03:17:50.998964 IP 69.252.96.22.42864 > 18.181.0.115.53: 64361 [1au] A? 8704.ns.ldpreload.com. (50)
03:17:50.999280 IP 18.181.0.115.53 > 69.252.96.22.42864: 64361 NXDomain*- 0/1/1 (114)
03:17:51.536824 IP 69.252.96.23.20948 > 18.181.0.115.53: 14908 [1au] A? 8704.ns.ldpreload.com. (50)
03:17:51.537168 IP 18.181.0.115.53 > 69.252.96.23.20948: 14908 NXDomain*- 0/1/1 (114)
(The fact that it attempts two queries in short succession is somewhat
interesting. With zero evidence, I wildly assert that this is the fault of
this router trying the primary DNS server, getting an NXDOMAIN, and then
trying the secondary DNS server to see if it has better luck there. It's
a Netgear N600, if anyone knows what they do.)
> Somehow I thought the proper plan was to query once, resolve, and then
> reverse-resolve that IP address. How did that fall by the wayside?
I assume what's going on is that, having opened a connection and resolved
the IP, SSH nonetheless passes the hostname, not the IP address, to the
Kerberos libraries. See how gss_host is set/used in the GSSAPI keyex
patch:
http://patch-tracker.debian.org/patch/series/view/openssh/1:5.5p1-6+squeeze1/gssapi.patch
That said, intriguingly, get_canonical_hostname() (in the OpenSSH source
proper, canohost.c) claims to be memoized, so I'm a little unsure why that
cache isn't working. I guess we leave GSSAPITrustDNS as no in sshd_config,
but the Kerberos libraries do that internally? (I thought there was a
krb5.conf setting for this that defaults to on, but I can't find it, and
RFC 4120 says you MUST NOT do that. But this entire setup wouldn't work
unless someone canonicalized athena.dialup, since no host/athena.dialup
keytab exists or is used.) Given that Kerberos apparently does this
anyway, is it fine to switch to GSSAPITrustDNS yes if that enables
memoization of the canonicalization?
--
Geoffrey Thomas
geofft@mit.edu
