[80] in 6.033 discussion
Denver International Airport
jerhy@ATHENA.MIT.EDU (jerhy@ATHENA.MIT.EDU)
Wed May 4 18:07:14 1994
I've grabbed this post from comp.software-eng. The person who posted
this message followed it by a long article called The Art of
Troubleshooting, which you can seek out on Usenet if you're
interested.
jeremy
===
Newsgroups: comp.software-eng
From: tmp@netcom.com
Subject: Denver International Airport DOA, Again / The Art of Troubleshooting
Message-ID: <tmpCp8EA1.H5z@netcom.com>
Organization: NETCOM On-line Communication Services (408 241-9760 guest)
Date: Tue, 3 May 1994 14:44:25 GMT
Lines: 237
====
The Apr 30 Denver, Colorado Rocky Mountain News front page story
`Debacle at DIA-- Project out of control; airport won't open for
months' describes how the faulty baggage system has now caused the
fourth delay in the multi-billion-dollar project. Tests on Friday
resulted in jammed baggage carts on elevated tracks, 100-cart
pileups, bags falling onto tracks, and slow delivery. ``At times,
tempers wore short as workers below the surface of the $3.7 billion
airport toiled to clear jams...''. Further large-scale tests of the
baggage system were cancelled.
``It's not ready. I'm not going to do anything to embarrass the
city,'' said Mayor Wellington Webb. The newspaper reports that
``problems with the one-of-a-kind system are becoming so complex that
Webb is searching for a NASA level consultant to assist BAE Automated
Systems'' (the baggage system contractor). ``We're dealing in areas
we don't even understand'' said the mayor.
On Friday Continental tested transit of 429 bags from its ticket
counters to its concourse, of which 38 were lost in the system. An
additional 15% of the bags that arrived had to have their bar-coded
tags scanned twice. An afternoon test by Continental went worse, with
a 61% rate of 425 bags sent (113 rescanned, 53 lost). United ran a
low-volume 300 bag test (less than 10 minutes of business) in the
morning that had a 95% success rate. Their afternoon test involving
outbound, inbound, and transfer baggage of 2,100 items was cancelled
because of `a significant number of jams', according to Public Works
spokeswoman Amy Lingg.
The Rocky Mountain News reports that the baggage system is suffering
problems in two basic areas: software glitches and mechanical
failures. In the former case, the system probably employs real-time
software of around 1 million lines of code, with `two to three errors
per 1,000 lines of code typical'. ``Industry average for debugging
errors takes 25% of the project's time or as many as 24 man-hours per
error.''
In hardware areas, the 30,000 feet of conveyors, 19,000
bearings, 4,600 syncrhonous drives, 3,000 conveyor drives, 2,750
photo cells, 2,100 linear induction motors, 1,300 conveyor power
turns, 350 motor control panels, 300 radio frequency readers, and 112
programmable logic controllers add up to a fiendish complexity
proving intensely difficult to finetune to success. Currently a major
problem is in vibrations that cause connections between rails to
separate from each over time, with cart wheels jamming as they pass
over the warped section. ``The Frankfurt Airport in Germany is the
only other airport in the world to employ such a large system, and it
took AEG Inc. six years to install the system and two years to
correct all the software and hardware bugs.''
The newspaper devoted an entire column, albeit speculative in tone, to
the software problems that plague the system, interviewing various
industry experts on the difficulties of real-time software
(apparently BAE engineers were unavailable or unwilling to comment).
``The BAE system employs laser scanners that read bar-coded labels
placed on baggage. Experts say that means the BAE system probably
employs real-time, numerical-control software. If running
off-the-shelf software is the equivalent of riding a bicycle with
training wheels, running real-time software is like running a
unicycle.'' John Sloan, software engineer for the National Center for
Atmospheric research in Boulder, commented: ``You run a test and
something won't work. What you'd like to do is say `We'll fix the
problem and run exactly the same test again.' But you can never
produce the same circumstance two times in a row.''
The column compared the DIA baggage software problems to the
now-infamous incident of the AT&T switching failure: ``Even the
smallest error can cause a ripple effect that turns into a tidal
wave of the kind that swamped AT&T's main switching system several
years ago and shut down nearly 90% of the phone company's domestic
long-distance operations for hours.'' Well-known Boulder software
programmer Phil Zimmerman, developer of Pretty Good Privacy
encryption software, commented: ``Software is the most complex of
human inventions. It's kind of like if airplanes were like software
systems, if someone forgot to tighten a screw under a seat, it would
cause the engines to explode.''
The airport is now costing about $15 million a month in interest
payments. Bond rating agencies Moody's and Standard & Poors currently
rate DIA airport bonds `BBB' or two notches above `junk', and have
notified the city that a downgrade may be imminent. This would
escalate interest rates on future borrowing by the city to finance
further delays in opening. Delays at this point cost the city about
$1 million a day to service the $3.1 billion in bonded debt and pay
operating expenses.
===
