[89049] in North American Network Operators' Group
Re: shim6 @ NANOG (forwarded note from John Payne)
daemon@ATHENA.MIT.EDU (Kevin Day)
Wed Mar 1 13:33:28 2006
In-Reply-To: <BDCC3DD4-B902-4B43-B461-D416FF4888EC@isc.org>
Cc: NANOG list <nanog@nanog.org>
From: Kevin Day <toasty@dragondata.com>
Date: Wed, 1 Mar 2006 12:32:33 -0600
To: Joe Abley <jabley@isc.org>
Errors-To: owner-nanog@merit.edu
On Mar 1, 2006, at 9:07 AM, Joe Abley wrote:
>
> On 1-Mar-2006, at 02:56, Kevin Day wrote:
>>
>> If you include "Web hosting company" in your definition of ISP,
>> that's not true.
>
> Right. I wasn't; I listed them separately.
>
> It's important to note that even if you are a hosting company who
> *does* qualify for PI v6 space, you still need shim6-capable
> servers, if you want to make them optimally available to multi-
> homed, shim6-capable hosts. The difference PI makes is in the
> distribution of addresses to servers (the servers only need a
> single set).
>
Which isn't a point to be glossed over. PA v.s. PI is a big deal to
people for many reasons. (Portability between providers being the
biggest.)
>
>> I'm just one guy, one ASN, and one content/hosting network. But I
>> can tell you that to switch to using shim6 instead of BGP speaking
>> would be a complete overhaul of how we do things.
>
> You are not alone in fearing change.
>
It's not so much fearing change, as the bar of effort/difficulty in
transitioning from 4 to 6 being really high.
If IPv6 is made as painless as possible to people to transition to it
NOW, people will. If you can tell every ISP, NSP, hosting company and
end site that they can continue doing what they're doing now in 4,
but with vastly more address space, you'd have a lot of convertors.
Every thing that you make people do differently (even if it is for a
greater good, and even if it benefits them directly) is one more
reason people will give NOT to do it until they have to.
Telling a hosting company "Here, you can get a /32 or /44 of your own
which is virtually unlimited space for your needs, continue using BGP
and TE as you do now, just deploy IPv6 on top of IPv4 and you're
live" is an easy sell.
Telling a hosting company "You have to lose the independence of PI
space. You need to completely start over with your traffic
engineering and do it in a way totally orthogonal to how you have to
continue doing it in IPv4 space(adding workload instead of replacing
what you're doing in IPv4). You now need to trust your customers/end
users to do the right thing with routing. Routing now involves
routers, servers and DNS - instead of a handful of devices making
routing decisions now ALL of your devices need to make routing
decisions. Even if you do get PI space somehow, you can't deaggregate
it even if you truly run multiple isolated networks.(Don't say 'Get
additional allocations then!' because that still results in the same
number of routes added to the global table, while wasting even more
space)" is a really hard sell. The only carrot you can offer someone
is "You can have lots more space", which I personally don't think is
worth even half of those negatives.
If significant percentages of networks are going to heavily push back/
delay deployment of IPv6, IPv4 will be exhausted before a critical
mass has switched to IPv6, making the whole "how do we protect the
long term future of IPv6" rational for these policies a little less
important.
It reminds me of a story from engineer who worked for AT&T/Bell when
touch tone dialing was first being tested in a few markets. AT&T
wanted to offer touch tone dialing as a convenience for users, but
they also had a desire to standardize the dialing procedure
everywhere. In some markets you could make local calls by just
dialing the last 4 or 5 digits of their number, others required the
full 7. Some required a 1+XXX-XXXX to make calls outside their city,
some only required the 1 if it was a toll call. AT&T really wanted to
change their network where everyone in every city used the same
procedure to make outbound calls. They decided that they'd make the
new dialing plan mandatory with touch tone service. The carrot was
"Look how much easier it is to dial compared to a rotary phone!", and
they got the benefits of forcing a standardized system on everyone
everywhere at the same time. The customer gets something that makes
their life easier, and the operator of the network gets to make their
job easier by standardization and reduced overhead of supporting
hundreds of incompatible dialing plans in each exchange that people
were used to.
They tested it in a few cities, with a few customers(business and
residential). A large number of people perceived touch tone dialing
negatively, so much so that they asked for their rotary phones back.
It had nothing to do with the push-button interface, it was asking
people to take a perceived negative along with a positive they
weren't sure they needed in the first place. Asking people to make
too many changes at once outweighed the convenience of faster
dialing. Users found it too confusing to have the new system (touch
tone dialing, a.k.a. IPv6) work so much differently than what they
were used to (rotary dialing a.k.a. IPv4) that they couldn't see past
the change into what the advantage was. In the end, they gave the
customers touch tone dialing, then gradually deployed the new dialing
plan, using permissive grace periods where both dialing plans worked
for as long as possible.
I'm worried the same will happen with IPv4/IPv6. The temptation of
virtually unlimited addressing is really nice. But I think the
negatives of the allocation policy and the direction of how
multihoming is going will scare off the willing participants, and
we'll be stuck with only getting people to switch when they're forced
to due to IPv4 exhaustion.
My advice: Get people to switch now.
If you have IPv4 PI space, you can get IPv6 PI space. If you have a /
21 now, you get a /48. If you have a /20 now, you get a /47. If you
have a /19 now you get a /46. Etc.
If you have anything bigger than a /48, you can rely on people
accepting deaggregated prefixes of /48 or shorter.
Push shim6 for people who don't need fancy traffic engineering, as a
tool for small/medium business. Heck, even residential if you want to
go that far.
Get a critical mass of people using IPv6 as soon as possible. Once
it's there, once people are comfortable with it, and once IPv6-acting-
as-much-like-IPv4-as-possible has proven itself, let people WILLINGLY
deploy shim6 if it truly would be advantageous to them. I'd have no
problem with raising the initial/yearly fee per ASN if it made
everyone comfortable that they were only being used where it was
truly needed.
On top of all of that, I'm still not convinced that IPv6-acting-as-
much-like-IPv4-as-possible isn't going to have significantly less
routes than IPv4, even if everyone moved today. Aren't a large number
of the routes being advertised due to people having to go back and
ask for more/bigger allocations again and again? If everyone out
there right now had to announce only 1 route per POP, and 1 route per
multihomed customer with PI space, wouldn't your outbound routes
shrink pretty significantly? That would be a huge step in the right
direction, and would buy loads of time to allow for a much more
gradual transition.
>> Putting routing decisions in the control of servers we don't
>> operate scares me. I wouldn't rely on 90% of our customers to get
>> this right unless it was completely idiot proof. Even if it was, I
>> don't see how we can trust that users aren't messing with things
>> to "game the system" somehow.
>
> This is the kind of feedback that the shim6 architects need. There
> is talk at present of whether the protocol needs to be able to
> accommodate a site-policy middlebox function to enforce site policy
> in the event that host behaviour needs to be controlled. The scope
> of that policy mediation function depends strongly on people like
> you saying "at a high level, this is the kind of decision I am not
> happy with the hosts making".
>
While I'm happy to give that feedback anywhere it's needed/welcome,
I'm kind of surprised alarm bells didn't go off already about this. :)
>> We deal with long lived TCP sessions (hours/days). I don't see how
>> routing updates can happen that won't result in a disconnect/
>> reconnect, which isn't acceptable.
>
> One of the primary objectives of shim6 is to provide session
> survivability over re-homing events. Since routing protocols are
> not used to manage re-homing, the speed at which a session can
> recover from a topological event depends on the operation of the
> shim6 protocol between client and server.
>
I'd... be really curious to see how that works, without having to add
intelligence to the application layer and stateful firewall layer to
handle this.
I don't mean to take an "I'll believe it when I see it" stance, but I
think a lot of layers(that may exist on other hosts) would have to be
changed to support doing that.
>
>> We have peering arrangements with about 120 ASNs. How do we mix
>> BGP IPv6 peering and Shim6 for transit?
>
> You advertise all your PA netblocks to all your peers.
>
Ok, I was a bit too vague there...
How do we ensure that peering connections are always used instead of
transit connections?
Currently for outbound, we can localpref peer-learned routes over
everything else. How do all of our servers on our end know that
routes learned from a BGP session on our own routers are desirable?
For inbound, we can either rely on our peers to do the same, prepend
what we send out to transit to make peer routes look even better to
our peers, or if we want to force the issue we can send peers more
specific routes than we send to transit (operating on the principle
of "most specific wins", no matter what else is done).
I'm not seeing how BGP routing information can enter into Shim6's
decision making, in any scalable fashion, and again.. something that
updates near instantly.
>> So far it looks like Shim6 is going to rely on DNS. The DNS
>> caching issue is a real problem. We need changes to happen faster
>> than DNS caching will allow.
>
> Well, not quite.
>
> If you change a transit provider, then you need to remove a set of
> AAAA records from the servers you operate, and substitute a new
> set. The time taken for this change to propagate in the DNS is non-
> zero, assuming you use reasonable TTLs. This is your point above, I
> think.
Reasonable TTLs on our end or not, lots and lots of people don't
respect TTLs.
Seriously, try this sometime. Set the TTL for a very busy site to 5
minutes. Wait 2 weeks, to make SURE everyone is seeing the new TTL.
Change the IP in the A record. Watch how long it takes for traffic to
stop coming in the old IP. If you see 90% of your traffic moved in
less than 4 hours, I'd be surprised. If you saw 99% of your traffic
moved in less than a day, I'd be even more surprised.
I don't know if this is intentional misbehavior of DNS caches, broken
software, broken OSes, or where the issue is. But, I can attest that
having to make sudden changes from one provider's PA space to another
with no grace period will result in support issues for at least a day
of "Why can't I reach your site?".
>
> Renumbering for hosting providers can be a monstrous pain in the
> neck, especially for hosting providers who rely on third parties
> (or, horrors, their customers) to maintain the zone files within
> which services are named.
>
Yep. We don't control the DNS for quite a number of the sites we're
hosting. Making our customer get involved every time we add/remove a
transit connection isn't going to be fun. Especially when "big"
providers who can qualify for PI space of their own don't have to do
this.
> Some hosting providers of my acquaintance insist on customer zones
> being redelegated to the hosting providers' nameservers, so that
> any renumbering that needs to happen can be coordinated by the
> hosting provider directly. Hosting providers who don't do this, and
> who use PA addresses with shim6 to multi-home, are definitely going
> to face some challenges.
>
That's possible for some hosting providers, not for others. It's not
uncommon for a customer to use one provider for some services(web
hosting, for example) and another provider for others(secure/e-
commerce, for example). Trying to make two competing providers play
together nicely when managing DNS for one domain is a recipe for
disaster, even with sub-delegation.
>
>> Some of our applications are extremely sensitive to jitter/
>> latency. We've spent ages tweaking route-maps manually (and
>> through automated continual tweaking) to make sure we avoid any
>> congested links. [...]
>
> The site-policy middleware that I alluded to earlier seems like the
> analogous place to specify this policy. Such a facility might
> actually give you more control than you have now -- tweaking BGP
> attributes to accomplish this kind of thing is often like a game of
> whack-a-mole; if you were able to control the route taken by
> traffic in both directions by influencing the locator selection for
> each and every session, you'd have far greater, and more fine-
> grained, control over your external traffic than BGP/swamp-abuse
> gives you currently.
While I don't doubt that there are advantages and disadvantages to
each way of doing things, I'd much prefer to be given the choice of
selecting the one that works best for us, possibly a mix of both.
>> We'd still be relying on PA space. No matter how great dhcp6 is,
>> there will be significant renumbering pain when providers are
>> changed. Static ACLs, firewall rules, etc. If you're including
>> customer machines in the renumbering, many simply won't do it.
>
> Agreed, renumbering is a pain. Dhcp6 sounds like a scary thing to
> use with servers. Customers suck. Change in operational practices
> will be required.
>
> Lest I sound too much like a foam-at-the-mouth shim6 advocate, I
> think it would be perfectly fine if, in the final analysis, the
> conclusion was that shim6 and PA/renumbering was not an option for
> hosting providers. A reasoned technical argument which came to that
> conclusion would provide a solid basis for the RIRs to modify their
> allocation policies such that hosting providers could use PI space
> instead. As perhaps the recent attempt to change the v6 PI policy
> indicates, the chances of making changes without such a reasoned
> argument are slim.
And that's kind of my overall point I've been not-so-successfully
trying to make.
IPv4 is running out. We need people to switch to IPv6, sooner rather
than later. Instead of trying to make the process as painless as
possible, and with using tools that are available now, a large swath
of what probably would be the FIRST people(content/hosting companies)
who would want to move to IPv6 are being told to "hang tight until we
figure out this multihoming thing, just don't expect it to work at
all the same as how you do it now." The additional bite of "You can't
do what you used to do, because if everyone did it the internet would
break. Oh, and your bigger competitors can still do things like that,
just not you." isn't helping matters.
Stopping the routing table from exploding in the future is obviously
a goal everyone needs to have. Everyone needs to think about this NOW
rather than when it's too late, I agree.
But, policies shouldn't be written depending on tools that don't
exist yet, which is basically what's happening now. If IPv6 were
permitted to be used in the same manner that IPv4 is now (PI space is
accessible to just about everyone, everyone with an ASN can run BGP,
you're trusted that if you deaggregate your space it's for a good
reason, etc), people could actually begin moving things now. Taking
an existing IPv4 network and overlaying an IPv6 network over the top
of it is relatively easy, we went from planning to full deployment in
a week.
Even if 100% of the IPv4 networks moved to IPv6 today, we'd still
have a smaller table size in 6 than 4. Growth would be slower (ISPs
and NSPs wouldn't continually be adding more networks as they grew,
initial allocations should be enough for just about everyone).
Then when Shim6 is developed, you can rely on the current release of
every major OS supporting it, router/middleware/etc vendors
supporting it, and all the kinks are worked out, you can let the
people who find Shim6 appropriate for their needs to use it. Make one
of the requirements to get an ASN a justification for why non-ASN/non-
public-routing doesn't work for your organization. Then let each
network operator choose the right tools for the job.
Without totally upending my network, I need:
1) PI space
2) The ability to deaggregate that PI space where truly needed. (or
the ability to request multiple PI blocks, but I don't see how that
helps matters)
3) BGP announcements to the world
If IPv6 can't give me those, and the only thing it's offering is more
space... That's just not worth the serious amount of labor and
reengineering of our network. If that's the value proposition, we'd
hold out on IPv4 as long as possible... And I'm *FOR* IPv6. :)
-- Kevin
(wondering how many people are muttering 'kook' right now)