[157619] in North American Network Operators' Group
NJ impact
daemon@ATHENA.MIT.EDU (Alex Rubenstein)
Wed Oct 31 15:26:56 2012
From: Alex Rubenstein <alex@corp.nac.net>
To: "nanog@nanog.org" <nanog@nanog.org>
Date: Wed, 31 Oct 2012 15:24:50 -0400
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
I had to summarize this recently for a news article I was interviewed for, =
so I figured I forward:
--
Of our three datacenters, this is what we saw:
Parsippany 1 (OCT) - The worst we saw here was several sub-second power hit=
s. UPS's held without problem, and we did not transfer to generator at all =
yet.
Parsippany 2 (WBR) - Transferred to generator at about 7:55 PM EST Monday a=
s a precautionary measure due to ongoing utility power hits. However, short=
ly after transfer, utility voltage went to 0 on all phases; around 10p powe=
r returned, but abnormally high (seeing about 550 volts on 480 volt bus). W=
e retransferred last night as utility voltage settled down.
Cedar Knolls 1 (MMU) - Briefly transferred to generator around 7:10, then b=
ack to utility. We then force transferred to generator around 8pm and staye=
d until this morning. Returned to utility and all systems are normal.
We have accommodated many customers (and non-customers) by allowing them to=
use our conference rooms and some offices, mainly because they lost power =
at their buildings, lost access to their buildings, or generators ran out o=
f fuel. We literally have card tables setup in hallways and common areas wh=
ere people sit and use Wifi to conduct business.
There has been no service interruption of any sort to any of our datacenter=
customers, as of this writing.
We lost one span of fiber optics between MMU and 165 Halsey, Newark (NWR), =
network healed in less than 1 second. It was imperceptible. That was early =
though, Monday afternoon. This has since restored today.
We have sales people onsite who are actively taking sales calls, and there =
is a measurable amount of customers moving equipment into our facilities as=
I write this. Many are moving equipment from offices with no power and no =
ETR in sight. We have electricians onsite to help accommodate.
The facilities department has been manning all locations around the clock.
All sites remain accessible by road. There was never a situation where you =
could not over the last 48 hours.
We have no equipment in 111 8th, so I cannot offer any telemetry there. How=
ever, our gear in 60 Hudson St (Tel-X 9th floor) is still online.
We saw no interruption at 165 Halsey, except for some very brief utility hi=
ts. UPS's held.
We are seeing hundreds, if not thousands, of DSL / T1 / T3 / Ethernet custo=
mers down - presumably mostly power loss on the CPE. We lost our connection=
to Covad, presumably they have gear in a building which lost power. These =
are only just starting to come back.
We have seen substantial interruption to the cellular networks; in many cas=
es, ATT Wireless (who NAC uses) is unable to complete a call. In certain ar=
eas, there is no signal at all.
Where I live, in 07874 (Byram), many roads are impassable and most everyone=
does not have power. Trees are down everywhere. I heard that a Byram squad=
car had a tree land on it while responding to a call. Schools are closed, =
Halloween has been "rescheduled" according to our governor. My cable is dow=
n, cable modem is down, POTS service is out - yet I still have xDSL (loop b=
y Verizon, IP by NAC) and I run on that successfully. I am unable to get an=
y ATT cell service at my house (usually at least three bars). I am on gener=
ator at my house, consuming about 2 gal/hour (propane), with a little over =
500 gallons left.
Gasonline is becoming an issue. As I understand, having talked to our oil s=
upplier, there is 0 refinery capacity in NJ at this time. All gasoline is b=
eing imported from PA, DE, MD, etc. Most stations don't have power. Ones th=
at do are generally out of gas. If they have gas, they have mile long lines=
and are rationing.
Diesel is unavailable if you had not pre-arranged for it.
Widespread power outages remain. I am running on generator at my house (Gen=
erac QT048) on a 1000 gallon tank of propane, have been since Monday about =
7pm. It is estimated at least 7 to 10 days before power restoration. I have=
yet to see one JCPL truck anywhere near where I live.
That is all I have.
This is the road leading to the Byram Police Dept - I took this yesterday m=
orning:
https://sphotos-a.xx.fbcdn.net/hphotos-prn1/12696_4311181250657_900557870_n=
.jpg
This is typically what Northern NJ looks like:
https://sphotos-a.xx.fbcdn.net/hphotos-ash3/548544_4311185050752_1011136048=
_n.jpg
https://sphotos-b.xx.fbcdn.net/hphotos-ash3/61115_4317167440308_289655722_n=
.jpg
(not the top of the telephone pole hanging from the wires)
