[128539] in North American Network Operators' Group
Re: off-topic: summary on Internet traffic growth History
daemon@ATHENA.MIT.EDU (Jeffrey S. Young)
Wed Aug 11 19:26:57 2010
From: "Jeffrey S. Young" <young@jsyoung.net>
To: John Lee <john@internetassociatesllc.com>
In-Reply-To: <53A6C7E936ED8544B1A2BC990D254F946FC6040B0B@MEMEXG1.HOST.local>
Date: Thu, 12 Aug 2010 09:26:29 +1000
Cc: "nanog@nanog.org" <nanog@nanog.org>, Andrew Odlyzko <odlyzko@umn.edu>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Worldcom bought MFS.
Worldcom bought MCI.
Worldcom bought UUnet.
In your statement s/MCI/Worldcom/g
I don't know if UUnet was part of Worldcom when MO first made statements =
about backbone growth, but I do know that internetMCI was still part of =
MCI and therefore, MCI was not a part of Worldcom. May seem like =
splitting hairs to some, but it is important to a few of us to point out =
that we never worked under Ebbers. Not that we had a choice :-). =20
Growth of the NAPs during this period is a poor indicator of growth. =
Because of the glitch you mention in carrying capacity the tier 1's all =
but abandoned the NAPs for peering between themselves and from that =
point forward (mid '97) preferred direct peering arrangements.
jy
On 12/08/2010, at 4:13 AM, John Lee <john@internetassociatesllc.com> =
wrote:
> Andrew,
>=20
> Earlier this week I had a meeting with the ex-Director of the Network =
Operations Center for MFS-Datanet/MCI whose tenure was through 1999. =
=46rom 1994 to 1998 they were re-architeching the Frame Relay and ATM =
networks to handle the growth in traffic including these new facilities =
called peering points of MAE-East and MAE-West. =46rom roughly 1990 to =
then end of 1996 they saw traffic on their switches grow at 50-70% =
growth every 6 months. By the last half of 1996 there was a head of line =
blocking problem on the DEC FDDI switches that was "regularly" bringing =
down the Internet. The architecture had lower traffic circuits were =
going through concentrators while higher traffic circuits were directly =
attached to ports on the switchs.
>=20
>=20
>=20
> MFS-Datanet was not going to take the hit for the interruptions to the =
Internet and was going to inform the trade press there was a problem =
with DEC FDDI switches so Digital "gave" six switches for the =
re-architecture of the MAEs to solve the problem. Once this problem was =
solved the first quarter of 1997 saw a 70% jump in traffic that quarter =
alone. This "historical event" would in my memory be the genesis of the =
100% traffic growth in 100 days legend. (So it was only 70% in 90 days =
which for the marketing folks does not cut it so 100% in 100 days sounds =
much better?? :) )
>=20
>=20
>=20
> MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet =
had all of the fiber running to key locations at the time and could =
drastically cut MCI's costs. UUNET "merged" with MCI and their traffic =
was put on this same network. MCI went belly up and Verizon bought the =
network.
>=20
>=20
>=20
> Personal Note: from 1983 to 90 I worked for Hayes the modem folks and =
became the Godfather to Ascend communications with Jeanette, Rob, Jay =
and Steve whose team produced the TNT line of modem/ISDN to Ethernet =
central site concentrators (in the early ninties) that drove a large =
portion of the user traffic to the Internet at the time, generating the =
"bubble".
>=20
>=20
>=20
> John (ISDN) Lee
> ________________________________________
> From: Andrew Odlyzko [odlyzko@umn.edu]
> Sent: Wednesday, August 11, 2010 12:55 PM
> To: nanog@nanog.org
> Subject: off-topic: summary on Internet traffic growth myths
>=20
> Since several members of this list requested it, here is a summary
> of the responses to my request for information about Internet growth
> during the telecom bubble, in particular the perceptions of the
> O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days"
> myth.
>=20
> First of all, many thanks to all those who responded, on and off-list.
> This involved extensive correspondence and some long phone =
conversations,
> and helped fill out the picture of those very confusing times (and
> also made it even clearer than before that there were many different
> perspectives on what was happening).
>=20
> The entire message is rather long, but it is written in sections,
> to make it easy to get the gist quickly and neglect the rest.
>=20
> Andrew
>=20
>=20
> -------------------------------------------------------------------
>=20
> 1. Short summary: People who got into the game late, or had been
> working at small ISPs or other enterprises, were generally willing
> to give serious credence to the "Internet doubling every 100 days"
> tale. The old-timers, especially those who worked for large ISPs
> or other large corporate establishment or research networks, were
> convinced by the late 1990s that this tale was false, but did not
> talk about it publicly, even inside the NANOG community.
>=20
> -------------------------------------------------------------------
>=20
> 2. Longer version: The range of views was very wide, and hard to
> give justice to in full. But there seemed to be two distinct
> groups, and the consensus views (which obviously exclude quite
> a few people) appear to have been:
>=20
> 2A: Those who entered the field in the late 1990s, especially
> if they worked for small ISPs or other small enterprises, tended
> to regard the claim seriously. (But it should be remarked that
> hardly anybody devoted too much effort or thought to the claim,
> they were too busy putting out fires in their own backyards to
> worry about global issues.) They remembered periods of desperate
> efforts to keep up with exploding demand in their businesses.
> We saw just a few hours ago a post about LINX growing 5.5x in
> one year. Somebody else wrote about growing their business's
> traffic 1,000x in 2 years, or about 30x per year. People involved
> in such incidents often tended to think that their experience
> during such times might not have been untypical.
>=20
> 2B: Those who worked at places with large traffic, and especially
> those who got into the field in the early 1990s, were quite sure
> by the late 1990s that the UUNet fable was just that. Comments
> regarding everything emanating from UUNet during that period
> included phrases like "blowing smoke," "rolling our eyes," "taking
> it with a rock of salt." They had no direct knowledge of what
> went on inside UUNet, but from watching peering traffic, talking
> to salespeople about customer losses and wins, and to suppliers
> about deliveries of equipment, they could be pretty certain that
> neither the traffic nor the capacity of UUNet could be exploding
> at the mythical rates. They could see occasional spikes in
> traffic growth at some customers, or in some parts of their
> networks, but overall could see traffic growth settling down
> to a fairly regular doubling or a bit more than doubling each
> year.
>=20
> However, they did not discuss this in public, and I discuss that
> below, in point #4.
>=20
> -------------------------------------------------------------------
>=20
> 3. Growth spurt in mid-1990s: The old-timers also provided very
> informative feedback about the global Internet traffic growth
> spurt in the 1995-96 time frame. It did hit suddenly and
> unexpectedly. (I am still trying to get confirmation on this,
> but I believe one of the informants said that before this
> period, the engineers at that person's Tier-1 ISP would routinely
> double the forecasts provided by the marketing team. At the peak
> of the bubble, the marketers would demand that the engineers plan
> for double the capacity that the engineers thought was going
> to be necessary.) Moreover, the dramatic slowdown in traffic
> growth that took place in 1997 was, at least in a number of
> cases, due substantially to a capacity crunch. Router and
> photonic equipment manufacturers did not have the technology
> needed for the traffic, and ILECs were slow in supplying
> access as well as backbone links. Hence the traffic growth
> spurt in 1996-96 was followed by a capacity growth spurt
> in 1997-98, which helped provide more credibility for the
> myth.
>=20
> -------------------------------------------------------------------
>=20
> 4. Information viscosity: This incident provides far more
> information confirming the concept of "information viscosity"
> that I wrote about in my paper, that important and relevant
> information was available, but was not widely dispersed.
>=20
> Why did the information that Internet traffic was not doubling
> every 100 days get out to the public? It was not a closely
> guarded national security secret, after all.
>=20
> There seemed to be many reasons operating. In the case of
> Genuity (as the quote from Scott Marcus in my paper explains)
> it was a high-level decision, that going public would hurt
> the company, as it might be suspected of losing market share.
> But that seemed to be unusual, in that Genuity, while a
> major Tier-1 ISP, was small and independent, and so had
> people like Scott who were engineers, yet involved in
> policy making. At other places, other dynamics operated.
>=20
> At AT&T, for example, I was called in to a meeting with
> the management of WorldNet, the AT&T ISP unit, at the
> end of 2000. They had been telling their customers that
> Internet traffic was growing 10x per year, and some of
> those customers asked them about the discrepancy between
> that claim and the estimates of some folks from AT&T
> Labs - Research (Kerry Coffman and myself) that growth
> was just 2x per year. Now it is an interesting perspective
> on "information viscosity" and AT&T (and other large
> bureaucratic organizations) that those folks had not
> heard of Kerry's and my work, even though we had publicized
> it at the company. But in any event, at that meeting,
> I did succeed in convincing them that Internet traffic
> was growing only 2x per year (using evidence in my papers
> with Kerry, as well as extensive additional data from
> within AT&T itself), and they agreed they would not
> propagate the myth among customers. But the interesting
> thing was that many of the attendees (who included quite
> a few engineering types, not just the management) seemed
> disappointed. That really surprised me. After all,
> AT&T's Internet traffic was growing just about 4x per
> year, which meant our market share was doubling, instead
> of being cut in half. But it seems that many of them
> had really bought into the Internet dream. And, furthermore,
> the story that we were losing market share was a good one
> to pry additional resources from the corporation.
>=20
> Several of the NANOG old-timers said that they felt constrained
> from speaking about the falsity of the myth of astronomical
> growth rates by group solidarity. After all, it was a
> small, select group, and bad-mouthing one of their own
> in front of outsiders or even NANOG newbies did not seem
> the right thing to do. Then there was the additional
> factor that one person discussed very explicitly, and
> that I infer also applied in other cases. The myth was
> useful. Back in the mid-1990s, when it was not a myth,
> it did spur equipment suppliers and ILECs to greater
> effort. And afterwards, it was handy in internal fights
> over power and resources. If the Internet was exploding,
> one should not worry about closely examining expansion
> plans.
>=20
> In some ways, the persistence of false perceptions about
> Internet traffic growth mirrors that of utilization rates
> of data networks. When I wrote the paper "Data networks
> are lightly utilized, and will stay that way," back in 1998,
>=20
> http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf
>=20
> the more clueful data network engineers all knew that
> utilization rates were generally low, and usually had
> a good understanding as to why. But top management, as
> well as the research community, were almost uniformly
> convinced that congestion was the rule. It took me a
> while to understand the dynamics of the situation, in
> which network engineers and managers found it easier
> to say that they were experiencing 80% utilization and
> 20% packet losses and needed to upgrade, without having
> to explain to CEOs, CIOs, and especially to clueless
> CFOs, that this referred just to the peak hour on one crucial
> corporate WAN link, and yet justified a big new investment.
> Few people were lying, but many had incentives to maintain
> delusions among the top levels of the hierarchy.
>=20
> These demonstrations of information viscosity of course
> undermine the foundations of much of modern economics,
> especially of the efficient markets hypothesis. But that
> last myth is even more durable than "Internet traffic
> doubling every 100 days," especially since it does not
> lead directly to lots of dark fiber lying around unused,
> and lots of companies bankrupt. Information viscosity
> in facts and ideas in economics is very high, so we
> should not expect any changes on that scene.
>=20
>=20
> -------------------------------------------------------------------
>=20
> 5. As a reminder, the paper that led to this discussion is
> "Bubbles, gullibility, and other challenges for economics,
> psychology, sociology, and information sciences,"
>=20
> http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf
>=20
> The page with source materials from the bubbles times,
>=20
> http://www.dtc.umn.edu/~odlyzko/isources/
>=20
> now has, in addition to a transcript of the O'Dell lecture
> at Stanford in May 2000, a copy of the Sidgmore paper from
> the Vortex98 Conference of May 1998.
>=20
