[8660] in Commercialization & Privatization of the Internet
NAP philosophy & IP technology
daemon@ATHENA.MIT.EDU (Erik E. Fair" (Your Friendly Postm)
Tue Nov 30 07:09:03 1993
From: "Erik E. Fair" (Your Friendly Postmaster) <fair@apple.com>
To: com-priv@psi.com
Date: Tue, 30 Nov 93 03:43:41 -0800
There appear to be people worried about the NAP portion of the new
NSF solicitation. Some of those same people are enamoured of the CIX
approach to things. I'm going to set down some thoughts I've been
having on the subject lately for your consideration.
I hope we can all agree on some goals:
1. Internet routing should be complete: everyone can send IP
packets everywhere (forget security firewalls for the purpose
of this discussion). Among other things, this means that I
should be able to buy one IP network service, and get
everywhere - I don't have to have a connection to each service
provider, like the X.25 PDN world in the U.S.
2. Internet routing should be robust - where multiple paths
exist, the routing system should recognize and re-route around
failed links, preferably quickly enough that TCP connections
are uninterrupted.
3. Internet routing should be symmetric (the path your packet
took to get to me is the path my packet should take to get to
you. TCP's retransmission timer algorithms assume this. TCP
still works with asymmetric routing, but not as well).
Now throw into this mix the concept of a "default route" - that is,
absent explicit routing information, the "default route" tells you
where to send a packet next. Primarily, you use this to cut down on
the amount of routing information you need to have in your routers.
However, the implication of having a default route is that somewhere,
there really must be at least one router that has complete information,
and *no* default route. The buck (packet) has to stop some place.
As an aside, you can construct a network without default routes; you
just have to schlep all your network's routing information everywhere.
Unfortunately, this can leave you with sub-optimal cases, like a single
ethernet site on the end of a small leased line. Do they need to know
everything about the rest of the world? Heck no! They've got their
local ethernet, and everything else goes out the leased line. Giving
them a complete picture of the network routing could (depending upon
the routing protocol being used) be expensive in bandwidth, and force
them to buy a bigger router (fast CPU, more RAM) than they might need
if they could use a default route. This is a problem that most
proprietary network protocols (e.g. AppleTalk) suffer from: no support
at all for a "default route".
So far as I know, almost everyone who is on the Internet today, except
the backbones (where the buck is supposed to stop) has a default route
in their tables. This includes the regional networks (e.g. NEARNET,
BARRNET, CERFNET, etc), sites like Apple Computer, and so on. We all do
this so that we can keep the amount of RAM and the CPU load on our
routers down. Also, for those networks who are still using an interior
routing protocol that blares everything it knows every N seconds (e.g.
RIP, IGRP), the less routing information blared, the more bandwidth we
have left over for real applications.
I'm stepping out beyond my direct experience hereafter, into the realm
of speculation and extrapolated behavior based on hearsay and my own
small research efforts - expert correction on the technical points
where I err would be appreciated.
There are two models for a NAP that we can examine today: the CIX, and
the FIXes (I think, for the purpose of discussion, that MAE-EAST can be
considered to be like the FIXes, though I'm not clear on the policies that
guide implementation at MAE-EAST - there may be something novel going on).
The CIX is a single cisco router, in San Jose, California. As I
understand it, all members are supposed to connect to it at T1 rate
with synchronous PPP encapsulation, and speak Border Gateway Protocol
(BGP) with it to exchange routing information (or arrange with some
existing directly connected CIX member for transit, but such agreements
are privately arrived at between the connected CIX members and the CIX
members desiring transit to the CIX). All information from direct
customers of CIX members is supposed to be exchanged, to satisfy goal
#1 that I stated above.
There are two FIXes (Federal Internet eXchanges) that I know of: one
east near D.C. that is run by SURANET, and one west at NASA/Ames
Research Center in Mountain View, California (I live in that city). My
understanding of FIX-WEST is that it's a LAN (ethernet or FDDI, I
haven't asked Milo in a long time), and routers that belong to several
different networks are on the LAN, exchanging routing information in
some semblance of a mesh. I know there's an E-PSP (a splinter of an
NSFNET E-NSS) there, and routers from BARRNET, SPRINTNET, MILNET,
ESNET, the NASA Science Network (NSN), PACCOM (Pacific Basin) and
probably some others. I've never seen it, so I can't produce a complete
list. The point is one of architecture: the FIX is a LAN that everyone
connects a router to, but who they exchange routing information with
(and how) is a matter of negotiation.
The interesting thing about these two architectures is one of default
behavior - by default the CIX is going to hand you everything from
everyone else, with no manual intervention required. A network service
provider connected to the CIX would have to go to trouble to filter out
routes from someone else on the CIX that they didn't like.
To get full routing on the FIX, by contrast, a network service
provider must be informed of all additional routers added to the FIX
LAN, and then set up the exterior routing protocols to "peer" with the
new arrivals to exchange routing information. It's easy to not talk to
some other network - just don't exchange routing information with them.
Now let's play blue sky: how do you scale these models into a backbone?
The CIX backbone is well understood: more than one router owned by the
consortium, interconnected by some leased line mesh, or other switching
fabric. Call it a "cloud" if you like. This sounds like ANS CO+RE.
The FIX model backbone would have to be a shared medium, like SMDS, or
maybe an ATM fabric. MCI has a nationwide SMDS offering called
HyperStream, and Metropolitan Fiber Systems (MFS) is doing ATM now.
Then each of BARRNET, SURANET, NEARNET, NWNET, WESTNET, etc would attach
their routers to this fabric, and set up whatever peering arrangements
they agree upon between each other through the WAN, just like on the
FIX LAN. Hmmm. This sounds like CoREN.
The key difference is in who owns the routers on the edges of the
cloud, be it a marvelous TPC switching fabric, or a lumpen leased line
network: do the "customers" own the routers on the edge and negotiate
lots of peering arrangements with everyone else on the fabric, or is
there an operating organization that owns the edge routers that can
therefore set a coherent routing policy for all edge routers?
There's fear & loathing to be found in both models of the universe: if
you have a single organization owning the edge routers, it can turn
into an overbearing, overcharging, competitor to the attached regional
network service providers (after all, it has routers everywhere),
unless some agreement with or ownership of the backbone organization on
the part of the regional networks exists. On the other hand, you can
end up with network balkanization if the network operators who own the
routers on the edge of the WAN cloud get either sloppy or petty.
The NSF NAP solicitation, as I understand it, seems to be from outer
space: edges with no middle? It only works for the degenerate case of
one and only one NAP. Unfortunately, that leaves a nest of nasty
backhaul problems for the network service providers who are
geographically far away from the NAP, unless you take the FIX WAN type
approach. I bet there's not nearly enough money in the solicitation for
that, because that would put the NSF into the backbone business again -
the very trap they're trying to escape. So moneys for a FIX WAN to
interconnect multiple NAPs would have to come from elsewhere. CoREN,
again? Or maybe a modified CIX?
If NSF really wants to play the game of multiple NAPs with no explicit
single backbone, there must be a rule: anyone who wants to play
inter-NAP carrier *must* connect to all NAPs and provide all their
routing information at every one - this is the only way to get the
routing system to work right, with multiple inter-NAP carriers who each
have less than the total customer base. And no matter what, this scheme
will play hob with goal #3 above, unless each of the inter-NAP
carriers' networks have *exactly* the same inter-NAP packet delays.
If there are multiple NAPs, and the inter-NAP carriers are not
required to connect to every NAP, then where do you point your default
route? And how do you get to networks using an inter-NAP carrier that
is not connected to the NAP that you are?
The technology really wants one and only one backbone for the Internet.
It can be as small as one router, and it can be as big as a worldwide
SMDS or ATM fabric, but there really ought to be just one.
Erik E. Fair apple!fair fair@apple.com
