[148887] in North American Network Operators' Group
Re: NANOG Digest, Vol 48, Issue 89
daemon@ATHENA.MIT.EDU (Rafael Cresci)
Wed Jan 25 18:39:50 2012
From: Rafael Cresci <rafael@cresci.org>
In-Reply-To: <mailman.1524.1327531898.46019.nanog@nanog.org>
Date: Wed, 25 Jan 2012 21:38:47 -0200
To: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
On 25/01/2012, at 20:51, nanog-request@nanog.org wrote:
> Message: 4
> Date: Wed, 25 Jan 2012 15:11:42 -0500 (EST)
> From: Jay Ashworth <jra@baylink.com>
> To: NANOG <nanog@nanog.org>
> Subject: Equinix Miami 1 condemnation
> Message-ID:
> <22786388.6606.1327522302701.JavaMail.root@benjamin.baylink.com>
> Content-Type: text/plain; charset=3Dutf-8
>=20
> Last week, we saw some traffic about the Lightfiber problems because =
EqM1=20
> is apparently in a building that's been condemned by the city or =
county
> of Miami.
>=20
> That struck me curious, so I wanted to look into it further.
>=20
> Amazingly, there doesn't seem to be any coverage of the incident, even =
in
> technical circles. Does anyone know anything they're permitted to =
tell
> about how a building which contained a datacenter managed to get =
itself=20
> condemned?
> This was the Metro Mall facility, operated by Switch and Data until
> Equinix's acquisition of S&D a few years ago. It is now known as
> Equinix-Miami 1. 1 NE 1st Street.
>=20
And...
> The issue here has nothing to do with any hurricanes in the past. My
> opinion is that they lacked proper documentation and planning. Having =
a
> bit of insider knowledge here and knowing where this fiber came from
> originally (ACSI) and how FiberLight came to acquire it, I'm fairly =
sure
> I'm on the money.
I have no NDA on this so I can talk better in details.
Equinix Miami 1 (formerly Switch & Data, formerly another company name I =
don't recall now) was the temporary building that the NAP was built on, =
before moving to the current location that Terremark is (if not wrong, =
they called it the "T-Rex project" and the NOTA exchange started there =
in that place). Thus, (nearly) every major fiber strands that come from =
the submarine cable landing stations in SFl come into EQX MIA1 first, =
physically, before going into Terremark at 9th Street.
The building itself (Metromall) is not a datacenter by design, it's a =
normal commercial building. Some handfuls of jewelry stores, coffee =
shop, travel agency, a hair dresser, in the lobby and ground and =
basement levels. The loading dock of the building is on the same alley =
as Equinix Miami 2 (36 NE 2nd Street/Telesource =
building/Telx/Colohouse/Fibermedia/Savvis) loading dock entrance. Global =
Crossing's datacenter also was in the Metromall at another floor before =
going into Terremark's 6th floor. Equinix/S&D was on the 5th floor =
(whole floor). They were the biggest lease there, but they had no word =
on the building administration or maintenance, only on their own floor, =
that was pretty empty (no more than two or three dozen of customers). =
S&D considered Miami more of a strategic network PoP than a real =
customer datacenter (as does Savvis, that has a full floor in the 36 NE =
2nd St building, but no customers, and doesn't announce it as a =
datacenter/colocation PoP even having all that square footage and huge =
generators), they had only a small number of customers and the two =
facilities (MIA1 and MIA2) were acquisitions of former small/broken =
telco companies. They were not built by S&D, less yet by Equinix.
The building failed the 40-year inspection by the city of Miami. The =
landlord/owner of the building decided not to fix the numerous issues =
that were presented by the inspectors, and then decided to just let it =
go, close everything, expel all stores (mostly were on monthly leases) =
with little to no notice, and maybe sell the building or the ruins to =
someone who wanted it to fix or to demolish.
Equinix (or better, Switch and Data) was in negotiations with a company =
I worked for as an outsourced partner, to build a pipe between both =
buildings, when they faced a long delay on approval by the landlord, =
When they pressed, they got the real reason why he was holding it. Last =
June Equinix started to evacuate the building preemptively, moving all =
customers to Miami 2 (it's in the same block, just two corners or a =
backdoor away) and paying their moving costs (materials, manpower, telco =
circuit movings) up to a ceiling of $10k (in service credits, not cash, =
power install was covered by Equinix). They did some preparation work on =
Miami 2 for receiving these customers - not many - and started that, =
with an ETA to move everybody out by October 2011.
The "secret" ingredient was more of a "let's not annouce it so no one =
tries to signup to Miami 1 and then take advantage of service credits", =
until the unavoidable was unavoidable and until Miami 2 was ready for =
deployment of these customers. Only the customers, partners and the =
Digital Realty Trust management (Miami 2 is a DRL building)/security =
were aware of that while the move was not completed or at least mass =
moved. Some competitors got to the knowledge after the customers were =
notified (in June 2011) and some customers leaked that information by =
accident or intentionally at that time.
AFAIK, Fiberlight spliced all Fiber on the manhole during 3 nights in =
July/August 2011, to reroute them out of MIA1. What happened these =
couple days ago was something completely different and probably the =
result of something they forgot to do on that manhole, or pure stupidity =
of doing maintenance/upgrades on both sides of the ring at the same =
time.
-- rc=
