[88685] in North American Network Operators' Group
NANOG36-NOTES 2006.02.15 talk 2 Katrina--telecom infrastructure impacts
daemon@ATHENA.MIT.EDU (Matthew Petach)
Wed Feb 15 12:02:51 2006
Date: Wed, 15 Feb 2006 08:59:31 -0800
From: Matthew Petach <mpetach@netflight.com>
To: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: owner-nanog@merit.edu
------=_Part_3885_23323591.1140022771176
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
2006.02.15 Hurricane Katrina: Telecom Infrastructure Impacts,
Solutions, and Opportunities, Paula Rhea, Verizon
A more interactive presentation from her, in the
aisles.
Verizon Business group--combined MCI/Verizon team.
Agenda
Hurricane Katrina Recap
Telecom infrastructure impacts
telecom provider successes
business continuity planning
conclusions
references
appendices: case examples.
Many of the people in this room would be considered
part of the critical infrastructure for the nation
by the department of homeland security
After world trade center 9/11 issue, there was
a lessons learned; hopefully there will be a
similar report post Katrina.
New Orleans is still very much like a war zone
right now; it's definitely a disaster recovery
training session for many industries.
Neighborhoods are wiped out; no capital
investments, infrastructure in holding pattern.
Many with no power, 20% of houses condemned.
Neighborhoods that are entirely silent--eerie.
Aim is not to diss anyone specifically, certainly
not in this room; aimed to be an assessment in a
neutral fashion.
Critical infrastructure:
food and water supply
energy
transportation
healthcare
banking/finance
telecommunications/infrastructure
Oddly enough, much of critical infrastructure is
privately owned, rather than government owned.
The domino model says that any one piece will
cause the rest to start to fall.
35th largest city in US
port of new orleans is #1 in US by tonnage
50% of total US grain exports shipped via gulf
10.8% of total US refining from new orleans
5th largest port
Key space shuttle facility in Michoud supported
fuel tanks for international space station
Storm recap
hirricane hit aug 29 2005
135MPH winds, 20foot storm surge sent inland
55foot surges logged in gul pior to landfall
levee failures create secondary crisis
2.3M homes without power
spawned 33 reported tornadoes in NA
1090 fatalities in LA recorded to date
people dancing about cat 5 dropping to cat 4,
thought they were spared, then levees broke;
had been predicted the year before. :(
Still 2500 people missing/unaccounted for.
Map of eastern LA parishes
st bernards/plaque mines parishes between the lake
and the gulf, hardest hit when levees broke as water
headed back towards gulf.
Lack of interoperability between parish govt systems.
New orleans telecom impact (multi-carrier)
1.75M lines down immediately following kat.
38 911 centers out (1/3) initially
1,000 cellular towers out
two class 4 toll switches initially out of service
no power/unable to secure extended diesel fuel
Traffic out of lata logjammed with toll switches out.
LECs had backup power systems, but no fuel.
Took 4 days to inspect causeway to allow emergency
crews into the city with main bridge out. Most
nurses and doctors were in suburbs, not in city.
Central offices post katrina
new orlenas lake co
CLLI NWORLALK
Venice LA CA
CLLI: VENCLAMA
Buras CO
CLLI: BURSLAMA
19 COs are totally destroyed, and will have to be
rebuilt.
These slides are public domain info, no inside info.
I2/Abilene link from Houston to Atlanta initially out,
restored on sept 8 2005
fiber optic path on lake pchatrain bridge
offline following hurricane katrina
wifi, wimax and voip play key role in area communications
public internet was actually very resilient
Telecom provider successes: alphabetic
1,000 amateur radio operators helped
bellsouth
cingular
cisco
cox,
iridium added 10,000+ new phones to first responders
MCI
Nortel
Sprint/Nextel donated up to 10M
Verizon donated 8M and 200 workers
Carriers have mutual aid agreements; Verizon sent 200
people who volunteered to spend 8 weeks living in a tent
to help rebuild--had to work with armed guards.
The CO rebuilds wasn't any type of upgrade, it was
bulldozing damaged/destroyed facilities, digging new
vaults, and starting over to restore just what was
in place before hand.
Bill Norton--COs underwater, can you imagine some type
of preventative design that could have been put in
place to help avoid impacts like this?
Most of the area is reclaimed land, 2 miles below
sea level (some dispute about that number).
Bill wonders if they could be built above sea level
somehow. Even if they were, Paula notes that they
wouldn't have power, wouldn't have 2 weeks of diesel
fuel to run them, etc. Really, it comes back to the
levees.
Randy Bush noted that early on, community based wifi
was one of the early-on means of communication to
daisy-chain packets along.
Roland, from Cisco; did some logistical work with relief;
Verizon donated eVDO boxes to make eVDO to wifi bridges,
did VoIP over wifi to eVDO boxes to juryrig connectivity.
But doesn't work so well with towers down, and no power.
With the cell phone infrastructure down, that really hurt
too.
Thanks to Todd Underwood/Renesys for their graphs; did
a pre-and-post analysis routingwise.
Top red is LA;
about 170 networks totally out during the bulk period.
teal/TX not impacted,
MS also hit, in tierms of percentage more so than LA
AL somewhat hit.
There's likely to be some significant number of businesses
that never recover from this.
6months later,
port of NO at 50% capacity
130,000 lines remain out in LO; awaiting local govt
repopulation lans
cellular service fully operational in occupied areas
outside of those with restritcted access
US congtressional budget office estimates damage
at $60B in insured and uninsured losses; insurance is
tagging it at $75B. It would have been far better to
have spent money helping fix the levees up front.
BCP
will your organization be able to survive a disaster?
1 what staf, materials, proceedures, and equipment is vital
to the firm
2 what suppliers and resources do I need daily
3 what will you do if your building is not accessible
4 have you planned for payroll continuity
5 who should yo invo
BCP compontents
NIST 800-34 document
develop contingency plan
intro/activation stage/recovery phase/reconstitution stage/
plan backup.
People are still in emergency mode six months later
disaster recovery journal (DRJ)
survey
IAEM, international association of emergency managers
GAP in business continuity
% stating bc plan is important 62%
60% have a plan
58% tested in last 12 months
18% never tested plan
55% implmented security measures
only 49% have bckup servers
Telecom conclusinos
interdepenedt infrsatrurctures incrase risk
business continuity management should include plans
for "worst case" scenrios
operate--interopearte
factor in data recovery and storage into plans
be flexible
June 1st, hurricane season starts again
DNA records lost from courts, many records that
might have been mandated also lost.
Vision of public internet realized
ARPANet launched predecessor in 1969
Goals: use disparate computers to communicate in order
to provide high reliability and survivablility
Katrina proved it is possible for Internet to meet that
mandate.
------=_Part_3885_23323591.1140022771176
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
<br>
2006.02.15 Hurricane Katrina: Telecom Infrastructure Impacts,<br>
Solutions, and Opportunities, Paula Rhea, Verizon<br>
<br>
A more interactive presentation from her, in the<br>
aisles.<br>
<br>
Verizon Business group--combined MCI/Verizon team.<br>
<br>
Agenda<br>
Hurricane Katrina Recap<br>
Telecom infrastructure impacts<br>
telecom provider successes<br>
business continuity planning<br>
conclusions<br>
references<br>
appendices: case examples.<br>
<br>
Many of the people in this room would be considered<br>
part of the critical infrastructure for the nation<br>
by the department of homeland security<br>
<br>
After world trade center 9/11 issue, there was<br>
a lessons learned; hopefully there will be a<br>
similar report post Katrina.<br>
<br>
New Orleans is still very much like a war zone<br>
right now; it's definitely a disaster recovery<br>
training session for many industries.<br>
Neighborhoods are wiped out; no capital<br>
investments, infrastructure in holding pattern.<br>
Many with no power, 20% of houses condemned.<br>
Neighborhoods that are entirely silent--eerie.<br>
<br>
Aim is not to diss anyone specifically, certainly<br>
not in this room; aimed to be an assessment in a<br>
neutral fashion.<br>
<br>
Critical infrastructure:<br>
food and water supply<br>
energy<br>
transportation<br>
healthcare<br>
banking/finance<br>
telecommunications/infrastructure<br>
<br>
Oddly enough, much of critical infrastructure is <br>
privately owned, rather than government owned.<br>
<br>
The domino model says that any one piece will<br>
cause the rest to start to fall.<br>
<br>
35th largest city in US<br>
port of new orleans is #1 in US by tonnage<br>
50% of total US grain exports shipped via gulf<br>
10.8% of total US refining from new orleans<br>
5th largest port<br>
Key space shuttle facility in Michoud supported<br>
fuel tanks for international space station<br>
<br>
Storm recap<br>
hirricane hit aug 29 2005<br>
135MPH winds, 20foot storm surge sent inland<br>
55foot surges logged in gul pior to landfall<br>
<br>
levee failures create secondary crisis<br>
2.3M homes without power<br>
spawned 33 reported tornadoes in NA<br>
1090 fatalities in LA recorded to date<br>
<br>
people dancing about cat 5 dropping to cat 4, <br>
thought they were spared, then levees broke;<br>
had been predicted the year before. :(<br>
<br>
Still 2500 people missing/unaccounted for.<br>
<br>
Map of eastern LA parishes<br>
st bernards/plaque mines parishes between the lake<br>
and the gulf, hardest hit when levees broke as water<br>
headed back towards gulf.<br>
Lack of interoperability between parish govt systems.<br>
<br>
New orleans telecom impact (multi-carrier)<br>
1.75M lines down immediately following kat.<br>
38 911 centers out (1/3) initially<br>
1,000 cellular towers out<br>
two class 4 toll switches initially out of service<br>
no power/unable to secure extended diesel fuel<br>
<br>
Traffic out of lata logjammed with toll switches out.<br>
LECs had backup power systems, but no fuel.<br>
<br>
Took 4 days to inspect causeway to allow emergency<br>
crews into the city with main bridge out. Most<br>
nurses and doctors were in suburbs, not in city.<br>
<br>
Central offices post katrina<br>
new orlenas lake co<br>
CLLI NWORLALK<br>
Venice LA CA<br>
CLLI: VENCLAMA<br>
Buras CO<br>
CLLI: BURSLAMA<br>
<br>
19 COs are totally destroyed, and will have to be<br>
rebuilt. <br>
<br>
These slides are public domain info, no inside info.<br>
<br>
I2/Abilene link from Houston to Atlanta initially out,<br>
restored on sept 8 2005<br>
<br>
fiber optic path on lake pchatrain bridge<br>
offline following hurricane katrina<br>
<br>
wifi, wimax and voip play key role in area communications<br>
<br>
public internet was actually very resilient<br>
<br>
Telecom provider successes: alphabetic<br>
1,000 amateur radio operators helped<br>
bellsouth<br>
cingular<br>
cisco<br>
cox,<br>
iridium added 10,000+ new phones to first responders<br>
MCI<br>
Nortel<br>
Sprint/Nextel donated up to 10M<br>
Verizon donated 8M and 200 workers<br>
<br>
Carriers have mutual aid agreements; Verizon sent 200<br>
people who volunteered to spend 8 weeks living in a tent<br>
to help rebuild--had to work with armed guards.<br>
<br>
The CO rebuilds wasn't any type of upgrade, it was<br>
bulldozing damaged/destroyed facilities, digging new<br>
vaults, and starting over to restore just what was<br>
in place before hand.<br>
<br>
Bill Norton--COs underwater, can you imagine some type<br>
of preventative design that could have been put in<br>
place to help avoid impacts like this?<br>
Most of the area is reclaimed land, 2 miles below<br>
sea level (some dispute about that number). <br>
Bill wonders if they could be built above sea level<br>
somehow. Even if they were, Paula notes that they<br>
wouldn't have power, wouldn't have 2 weeks of diesel<br>
fuel to run them, etc. Really, it comes back to the<br>
levees.<br>
Randy Bush noted that early on, community based wifi<br>
was one of the early-on means of communication to<br>
daisy-chain packets along. <br>
Roland, from Cisco; did some logistical work with relief;<br>
Verizon donated eVDO boxes to make eVDO to wifi bridges,<br>
did VoIP over wifi to eVDO boxes to juryrig connectivity.<br>
But doesn't work so well with towers down, and no power.<br>
With the cell phone infrastructure down, that really hurt<br>
too.<br>
<br>
Thanks to Todd Underwood/Renesys for their graphs; did<br>
a pre-and-post analysis routingwise.<br>
Top red is LA;<br>
about 170 networks totally out during the bulk period.<br>
teal/TX not impacted,<br>
MS also hit, in tierms of percentage more so than LA<br>
AL somewhat hit.<br>
<br>
There's likely to be some significant number of businesses<br>
that never recover from this.<br>
<br>
6months later, <br>
port of NO at 50% capacity<br>
130,000 lines remain out in LO; awaiting local govt<br>
repopulation lans<br>
cellular service fully operational in occupied areas<br>
outside of those with restritcted access<br>
US congtressional budget office estimates damage<br>
at $60B in insured and uninsured losses; insurance is<br>
tagging it at $75B. It would have been far better to<br>
have spent money helping fix the levees up front.<br>
<br>
BCP<br>
will your organization be able to survive a disaster?<br>
1 what staf, materials, proceedures, and equipment is vital<br>
to the firm<br>
2 what suppliers and resources do I need daily<br>
3 what will you do if your building is not accessible<br>
4 have you planned for payroll continuity<br>
5 who should yo invo<br>
<br>
BCP compontents<br>
NIST 800-34 document<br>
develop contingency plan<br>
intro/activation stage/recovery phase/reconstitution stage/<br>
plan backup.<br>
<br>
People are still in emergency mode six months later<br>
<br>
disaster recovery journal (DRJ)<br>
<br>
survey<br>
IAEM, international association of emergency managers<br>
GAP in business continuity<br>
% stating bc plan is important 62%<br>
60% have a plan<br>
58% tested in last 12 months<br>
18% never tested plan<br>
55% implmented security measures<br>
only 49% have bckup servers<br>
<br>
Telecom conclusinos<br>
interdepenedt infrsatrurctures incrase risk<br>
business continuity management should include plans<br>
for "worst case" scenrios<br>
operate--interopearte<br>
factor in data recovery and storage into plans<br>
be flexible<br>
<br>
June 1st, hurricane season starts again<br>
<br>
DNA records lost from courts, many records that<br>
might have been mandated also lost.<br>
<br>
Vision of public internet realized<br>
ARPANet launched predecessor in 1969<br>
Goals: use disparate computers to communicate in order<br>
to provide high reliability and survivablility<br>
Katrina proved it is possible for Internet to meet that<br>
mandate.<br>
<br>
------=_Part_3885_23323591.1140022771176--
