[142594] in North American Network Operators' Group
Re: Why is IPv6 broken?
daemon@ATHENA.MIT.EDU (Jason Hellenthal)
Sat Jul 9 20:22:39 2011
Date: Sat, 9 Jul 2011 20:21:45 -0400
From: Jason Hellenthal <jhell@DataIX.net>
To: Bob Network <networkjoe@hotmail.com>
In-Reply-To: <BLU153-w35B189A65550C808EB4537D1430@phx.gbl>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
<deBunk>
Where did you get all this from ?
There is not even one single reference to a URL, not to be rude but how
long did it take you to write this theory ?
As for "It's broken, first and foremost..." They may be a Tier 1
provider of other services and also happen to offer IPv6 at which they
are only a Tier 2 or 3 but using the marketing gimics of theyre original
Tier 1 status to get acknowledgement.
I stopped reading shortly after 'I think' the second paragraph and scanned
the rest for URLs that might have made this clear and to the point but
did not find any.
Heresay.
</deBunk>
On Sat, Jul 09, 2011 at 03:25:27PM -0600, Bob Network wrote:
>=20
> Why is IPv6 broken?
>=20
> It's broken, first and foremost, because not all network providers who cl=
aim to be tier 1 are tier 1.
>=20
> Even worse, some of these providers run 6to4 relays or providers to home =
users. A user has no choice which provider is running their 6to4 relay...s=
o, they might end up using a relay that is run by a provider who doesn't pe=
er with their intended destination. I don't think the IETF saw that one co=
ming. But the result is to make 6to4 even more broken. Now, I know some p=
eople want 6to4 to die, but while it still exists in some form, user experi=
ence is worse than it could be. The temporary fix is for any provider to r=
un their own 6to4 relay for their own customers (assuming that they themsel=
ves have full connectivity).
>=20
> Right now, unless you buy transit from multiple tier 1s, and do so with c=
arefully chosen tier ones, you have only part of the IPv6 internet. Many t=
ier 1s are unsuitable even as backup connections, since you still want your=
backup connection to have access to the whole internet! Good tier 2 provi=
ders might be an excellent choice, sine good providers have already done th=
is leg work and can monitor their providers for compliance.
>=20
> A few myths...
>=20
> Routing table size has nothing to do with completeness of routes. Google=
may be one route, through aggregation. And SmallCo may advertise a large =
route through one provider, and, due to traffic engineering, a smaller rout=
e through a second one - in many cases, anyone that had the large route wou=
ld be able to contact SmallCo, even without the smaller route being present=
=2E So routing table size doesn't work. In addition, some providers aggre=
gate their routing tables to reduce routing load and such. Others intentio=
nally don't or deaggregate it intentionally so that they can brag about hav=
ing bigger routing tables. What you need to ask is: "How many /64s can you=
get to from your network, and how many of these /64s are reachable from at=
least one other major provider (you don't care about internal-only network=
s, after all)?" They can give you that information, but many won't want to.
>=20
> It's also not about technical people not getting along. It's about busin=
ess players trying to make money, but not just that either. It's also abou=
t ensuring that providers don't end up assuming more than their share of co=
sts for a link. Just because you have a common peering point doesn't mean =
that turning peering on would reduce your costs. In some cases it may incr=
ease costs tremendously, particularly on your long haul backbone links, bec=
ause the other party would like to take advantage of an attitude of trust o=
n the internet. That's why we end up with peering policies and contracts.
>=20
> What is the issue?
>=20
> Let's take Hurricane. This is no different than other providers...basica=
lly, they want to say, "We shouldn't need to pay for IPv6 transit from anyo=
ne." This is what Cogent said on IPv4 a few years ago. Google used to say=
this too for IPv6, not sure if they are still saying it. Basically, "We k=
now we're big enough that you won't want to screw your users by not peering=
with us."
>=20
> A small network couldn't do this tactic - a 100 node network who said to =
the IPv4 tier 1s: "Hey, I'm in the Podunk Internet Exchange, so are you, so=
I'm going to peer from you so I don't have to buy any bandwidth for my web=
server (placed in the Podunk exchange). Sure, they would like to - it wou=
ld save a ton of money if their site got lots of hits. I mean, who wouldn'=
t want free connectivity?
>=20
> In IPv6, we're going through what we settled years ago in IPv4 - who has =
to pay who to connect. After all, even free peering connections have a cos=
t in manpower, debugging, traffic engineering, documentation, etc.
>=20
> Some players who aren't getting free interconnection to tier 1s in IPv4 w=
ant to get it in IPv6. So they've worked to attract lots of users, and don=
e so under the guise of "We like IPv6 and want to promote it." Others have=
not bothered with trying to attract the users, but have said, "We're too b=
ig for you to not want to give us connectivity for free, since it would pis=
s off your users if you don't" (Google did this at one point in the past, m=
ay still be doing it). The Google example is basically trying to use a mon=
opoly position to force business decisions.
>=20
> Now, HE, Google, and others would want you to think, "Hey, IPv6 is all ne=
w, and these $#@! other providers just want to make a buck on something the=
y have no right to." Well, perhaps. But what they aren't saying is, "We c=
an turn on BGP for IPv6 on our existing connections to other providers, wit=
h no cost to us, and actually have full connectivity." The issue isn't abo=
ut cost today - nobody is charging extra for IPv6 in addition to IPv4 on a =
pipe where you already buy IPv4 bandwidth. And Google and HE already buy I=
Pv4 bandwidth. What they are thinking of is the future, 15 years from now,=
when there is no IPv4 - in that future, IPv6 isn't insignificant bandwidth=
, it's everything. Wouldn't it be nice to be a tier 1 and not pay for that=
? Of course! And certainly one can argue for or against the current tier =
1 club's exclusivity. But it's the way the internet works right now, for b=
etter or worse. In the meantime, in pursuit of this future, today's custom=
ers are screwed by these providers trying to position themselves to make mo=
re profit margin down the road.
>=20
> Which is better for the customer? A system where they are screwed today =
so that their provider can have a better negotiating position in business d=
iscussions OR a position where they do whatever they have to take to provid=
e the customer with full connectivity? (To HE's credit, they are giving aw=
ay transit today on IPv6, so it's not like you are losing anything of value=
by not having the full internet routing tables, but it's a huge reason to =
not pay HE anything in other services, such as data center colocation - go =
with a provider that you pay and which gives you what you pay for - full tr=
ansit).
>=20
> A bit about peering...
>=20
> Lots of people who aren't running big networks don't understand peering. =
They think, "Doesn't this benefit everyone if everyone exchanges traffic?"=
Maybe, on a pure level, but the business doesn't work that way.
>=20
> I'll give you an example. Let's say you are a little ISP, and located in=
Virginia, near a major peering point. You say, "All the tier 1s are there=
, I can pull fiber to that peering point, which is only a block away, and h=
ave free internet, other than the cost of the line." So, let's say you run=
the line, and, let's say that all the tier 1s agree to let you peer for fr=
ee, since they want your traffic too. Now, let's say your user downloads 1=
,000 TB from a server in California, on Qwest's network.
>=20
> You paid, let's say, $15,000 for your piece of fiber going a block. You =
needed to hire contractors and buy permits and such, after all. So you sha=
red in the costs of letting the user get to the server. What did Qwest pay=
? Well, they dug trenches, pulled fiber, negotiated with cities, counties,=
and states, paid taxes on their work, lit this fiber, etc. It cost a lot =
because they went a lot further than your one block. And a lot more than $=
15,000.
>=20
> You say, "So what! Their customer benefits too!" That's true, but let's =
go a bit further. Let's say you have a network that extends to California =
- you by DS3s from Sprint to do it. There's some cost in that, but your us=
er in Virginia would need more bandwidth than your DS3s. So you decide NOT=
to peer in California, just in Virginia. That way you don't have to upgra=
de your lines for your Virginia user. Maybe you even legally break your co=
mpany into two entities, so that you can peer in California and Virginia bo=
th, but you can say with a straight face, "We only have Virginia offices fo=
r this user - the other company is a separate entity, and not the entity th=
at owns either the server or the end user."
>=20
> In other words, you found a way to shift most of the traffic burden and i=
nfrastructure costs to Qwest, away from your user.
>=20
> This is why Qwest has some sort of peering policy. Among other things, i=
t will require multiple exchange points, and Qwest will probably say they w=
ill send traffic to the closest peering point, to minimize their costs. Yo=
u get to do the same (more on that later).
>=20
> Let's say that you currently buy bandwidth from NTT - you're not big enou=
gh to get free peering from everyone, but Qwest agrees to peer with you. O=
f course Qwest and NTT also have a business relationship, to give each othe=
r free peering. If Qwest gives you and many other customers free peering, =
however, you'll send less traffic across NTT's network. That might be good=
from a technical standpoint, but NTT now is selling you a smaller pipe - a=
nd making less money. In effect, Qwest undercut NTT's business and lowered=
NTT's profits on the connection. How will NTT respond to that, when they =
were also giving free peering to and from Qwest? Well, they might decide t=
hat Qwest isn't a very nice partner and tell Qwest, "Pay us for transit or =
get lost." That could be ugly - both NTT and Qwest could lose, but Qwest, =
if they actually care about stable service, won't want to risk it. So gene=
rally you don't give peering to anyone who is a customer of one of your fre=
e peers. You don't hurt their business. In fact, it's often a requirement=
in the peering connection, legally. (that said, you could argue whether o=
r not there is an abuse of monopoly here...that's a different issue)
>=20
> Going one further, let's say you have the server, and Qwest has the end-u=
ser. That doesn't change anything - the economics are still such that Qwes=
t has the cost, you don't. That said, it's convention that the person rece=
iving the traffic pays for most of the backhaul.
>=20
> Asymmetry in the Internet:
>=20
> What's the path between your host and a remote server? How do you find i=
t? If you said "traceroute", you might be right, but are probably wrong. =
You need to trace route both sides.
>=20
> Every provider on the internet is trying to minimize costs. This means t=
hat you want traffic to leave your network and go to the destination networ=
k with as little distance traveled as possible, because costs go up with di=
stance. It's cheaper to increase the size of pipes within a city to get to=
a peering point than to increase your backbone pipe size. So, peering con=
tracts typically specify that you dump traffic to the peer as soon as possi=
ble. That means the person receiving the traffic generally pays more. It =
also means that any traffic that crosses an AS boundary almost certainly tr=
avels a completely different path each way. In many cases, one third party=
provider may be used in one direction, another in the other direction. So=
seeing packet loss on your traceroute at some random tinet router doesn't =
mean that this router is the cause of any problem, since the return path fo=
r that packet from that provider's router might actually cross yet another =
network that is never transited in either direction for your network connec=
tion. (I'm ignoring that most large providers also don't always send ICMP =
reliably BECAUSE they limit this intentionally to spare the router CPU from=
overload - it takes router CPU to generate an ICMP TTL exceeded, but it do=
esn't take router CPU to forward a packet - so traceroute or ping indicatin=
g loss at a router doesn't mean anything in itself - the path itself likely=
has zero percent loss).
>=20
> So, here's the scenerio.
>=20
> Let's say a user and a server are on two seperate networks, U (user) and =
S (server).
>=20
> Let's say they both utilize transit provider T. So the path could be: U=
-- T -- S. S buys an OC12 from T, while U buys a T1.
>=20
> But let's say that the user has a second transit provider, BIG, who is a =
free peer of T. He bought an OC3 from BIG. So there's another path betwee=
n U and S: U -- BIG -- T -- S. Likely this path is much faster than U -- T=
-- S.
>=20
> So, the path for the traffic to S goes U -- T -- S.
>=20
> Now, what path does the traffic from T's router go, when T's router gener=
ates an ICMP TTL exceeded in response from a traceroute from a user? Does =
it go straight over the T1 line, or does it go over the peering connection =
to BIG and then to the customer? The answer, it turns out, depends on netw=
ork configuration and policy. Let's say it goes out over the T1, but the T=
1 is congested. It will look like the congestion is at the connection betw=
een BIG and T, because this is the first hop that will show packet loss. B=
UT...the congestion is actually at the U's connection to T, which is irrele=
vant to the actual traffic path between U and S. So the user, at this poin=
t, calls up BIG and T and bitches about "Your peering congestion is congest=
ed" when the real problem is that traffic completely unrelated to the user'=
s problem is going via a congested path that is never used for connectivity=
between U and S.
>=20
> If you add several providers into this loop, you can end up with a situat=
ion where traffic uses Sprint in one direction, but never hits a Sprint rou=
ter in the other. This is actually very common. A user with slow download=
s might be experiencing packet loss on the path from server to user, but no=
t the other way around. In other words, the problem is a provider that nev=
er shows up on the user's traceroute!
>=20
> Remember that the providers hand off the traffic as soon as possible to=
=20
> their peer. So, whoever receives the larger amount of traffic needs the
> bigger cross-country (or trans-oceanic) links. If one side transmits a
> T1's worth of data, the other side transmits an OC48's worth of data,=20
> only one needs the OC48s across the country - the one receiving the=20
> traffic. That's why you hear about "traffic ratios". If the traffic is =
even both ways, both sides have to pay for the same amount of cross-country=
infrastructure to carry that traffic. So most providers won't peer with s=
omeone for free that sends, say, 10 times the amount of traffic that they w=
ill receive. It would end up costing a lot of money
>=20
> Back to IPv6...that's interesting, but what does it have to do with IPv6?
>=20
> Some providers want to do away with traffic ratio policies, mutliple loca=
tion peering, not providing free services to the other's customers, etc.
>=20
> THAT is why you can't ping some sites from your HE tunnel. It's not just=
that providers won't peer. It's also that providers have rules to keep th=
emselves from getting screwed.
>=20
> Certainly, there's ways around some of this (for example, traffic ratios =
- if I make sure my network is used for the cross-country traffic I send, n=
ot yours, then I've addressed that concern at a bit of increased expense fo=
r myself). But it's generally not worth doing until the size of the provid=
ers is sufficiently large. Other things don't have a good technical fix, l=
ike not peering with your peer's customer - that's a business rule.
>=20
> =20
