[155341] in North American Network Operators' Group
RE: Verizon FiOS - is BGP an option?
daemon@ATHENA.MIT.EDU (Ralph E. Whitmore, III)
Mon Aug 6 00:54:40 2012
From: "Ralph E. Whitmore, III" <ralphw@interworld.net>
To: "nanog@nanog.org" <nanog@nanog.org>
Date: Mon, 6 Aug 2012 04:53:38 +0000
In-Reply-To: <00b201cd737c$f3431090$d9c931b0$@iname.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
What I think most people are objecting to is that most of the issue with ma=
intain service is not related to technical capabilities, but related to the=
cost of providing these support services impacting the profit margins of t=
he large monopolistic carriers. Verizon with their FIOS offerings, at leas=
t in my area is CO based and all optical, so all I have to provide is power=
to my own FIOS terminal which is easy to do, floated on batteries (not the=
POS that VZ provided but a real battery bank) and I stay online. We have b=
een through 10 outages at least 10 hours in duration and one 3 day outage a=
ll courtesy of SCE who can't figure out how to replace the 57 year old wire=
s that keep breaking from corrosion here in So. Calif. As a subscriber, I =
paid for the copper Wire that VZ installed, I paid for the Copper that Edis=
on installed, I pay to maintain both of these services on a 7x24x365 basis=
. And each and every month, SCE and VZ take a mandatory deduction out of m=
y bill specifically to replace the copper every 50 years (the design life =
of the infrastructure). Edison squanders that money and just patches the i=
nfrastructure with no regard for the customers, and VZ replaces the copper =
with FIOS so they don't have to allow any competition from anyone else and =
demos the copper when they are done so no one else can use it. All of thes=
e companies fail to understand that they were granted a license and handed=
the keys to provide a public service and we expect them to perform that jo=
b rain or shine 7x24x365. Hurricanes (while not a problem where I am) are =
a known problem in many parts of the company and it is their job to maintai=
n service despite the hurricanes. These fiber huts/RSU's were installed to=
minimize VZ's (insert your favorite carrier here) cost of maintenance for =
their network . This way, they can increase their profits by laying off mor=
e workers and hiring more subcontractors. So be it, that is their business=
model. What people/PUC/ Regulatory bodies fail to follow up on is that ju=
st because they are allowed to install Fiber Huts/RSU's the customers shoul=
d expect the same level of service and redundancy that is provided by a bri=
ck and mortar CO built to the ATT/Bellcore standards for stability and reli=
ability. I am all for the carriers pushing the edge closer to the customer=
s, but it should not be allowed to occur at a substandard level. They cert=
ainly aren't offering a discount for substandard service received by some. =
My customers get 99.999% reliability from my infrastructure, I expect the =
carriers to do at least as well, obviously that doesn't happen.
All my Roadside cabinets have a DC plant that is engineered to hold the fac=
ility for at least 18 hours based on the equipment in the box. All my faci=
lities have an external Transfer switch and a generator plug. A single 5kw=
generator, with one cable and padlock, with one tank of fuel (generally p=
ropane or diesel)(about 5 gallons) will run the hut for 8-10 hours, fully r=
echarge the DC plant inside buying you another 18 hours on battery (therefo=
re 28 additional hours before fuel is needed) and to boot have electronic m=
onitoring to tell me if there is another AC failure of the generator during=
that 8-10 hours. One contractor, god forbid we wouldn't do this with act=
ual employees of the company, with a pickup truck and a small trailer shoul=
d be able to support between 12-20 Fiber Huts /RSU's in a single shift. If =
your site is bigger than a 5KW site then you should have generator backup o=
nsite in the first place. If I can do this Verizon/ATT/Centurylink should =
have no trouble supporting their facilities in a similar fashion. I am a l=
ittle dinky company with 12 employees, but Verizon( your carrier name here)=
If you want to use Wisper Watt 25KW generators, more power to you, they wi=
ll just pass the costs on to us anyways in the next rate hike. The argumen=
t about lack of fuel availability is total bunk as well as I can't believe =
that any carrier doesn't already have 1000's of gallons onsite already to =
support their CO's, trucks, ect. if you need more you can order a 5000gallo=
n tank like everyone else in the real world does, you can probably have it =
delivered tomorrow afternoon if you place the order tomorrow morning before=
noon. Lack of power at an Fiber hut/RSU is negligence at best on a carrie=
rs part, not an unsolvable technical problem
My uncle who retired from Bell Atlantic/VZ after 30 years was in a rural ar=
ea of Virginia and now works as a contractor for emergency restoration aft=
er storms. He was called in to work on Storm restoration he along with 2=
0 other crews, were told report to the staging yard and wait for an assignm=
ent. It took three days to get to the staging area, They waited 4 days in t=
he yard before being assigned work and was then told that they couldn't wo=
rk more than 12 hours in a day, because VZ would only pay for a max of 4 h=
ours OT for any personnel in a 24 hour period. They worked for 4 days and=
were then sent home as they were no longer needed. To the best of my reco=
llection people were without service for weeks if not months. The safety w=
ackos in the world will say well we don't want to tax our employees and mor=
e than 12 hours isn't safe. Ok, I will not argue the validity of their argu=
ment, personally I disagree with it, but this is an emergency response not=
a shareholders meeting, but sending the crews home weeks before all servic=
e is restored doesn't resonate well. IMHO Restoration was not about servi=
ng the people who were out of service, but rather spending at little as the=
y could get away with without regard for the people who pay for the infrast=
ructure and pay their bills. While we are at it, my uncle installed the R=
SU he is serviced from at his house before he retired from Bell Atlantic/VZ=
, they engineered only 4 hours of battery backup for their site near his ho=
use, the fiber hut/RSU is approx. 8'x8'x5' and it has only 4 -105AH batteri=
es installed in it, there are three vacant shelves for more batteries and o=
ne totally vacant compartment left in the cabinet for battery expansion. G=
iven the rural nature of his service RSU, the nearest stocking yard for VZ =
is better than an hour's drive from his RSU. Given the Union contract give=
s the workers 2 hours to show up at the yard from the time they are contact=
ed ( which could take hours to do in the first place if his RSU is down) th=
ere is almost no way that you can callout a tech and make it to the site be=
fore the battery back is exhausted and Dominion Power is highly un-likely t=
o arrive onsite in less than 12 hours from the time of call. So this site i=
s doomed to go down from the start.
A single house, my house, your house, despite what we think are not critica=
l infrastructure to the carriers, but the fiber huts that service them cert=
ainly is and needs to be supported better than they are now.
Back to the original topic, BGP works great over FIOS if you have a busines=
s account with static IP's and a Cooperative Colo. I have IPv4 and IPv6 ru=
nning at my house and office (we microwave the Home's business FIOS to the =
office) through a GRE tunnel. It's the cheapest 35x35Mb connection around =
for about $200.00 with 32 static IP's around TWC wants about $2k for the sa=
me service. It took a little bit of fiddling to get the MTU right across =
the tunnel (ip tcp adjust-mss is your friend) I could upgrade the service =
for another $100.00 to 150Mb x 65Mb, but the wireless link won't support t=
hat without a large expenditure and for 12 people it's not needed.
Boy that was a lot, glad I had time on a Sunday night to write this after f=
ueling my generator.
Ralph
-----Original Message-----
From: Frank Bulk [mailto:frnkblk@iname.com]=20
Sent: Sunday, August 05, 2012 7:41 PM
To: 'William Herrin'; Andy Koch
Cc: nanog@nanog.org
Subject: RE: Verizon FiOS - is BGP an option?
I think the term "critical" is being used in different senses in this discu=
ssion. Are people's lives critical? Yes, but the regulations for wired an=
d wireless infrastructure don't require service providers to expend any and=
all costs to maintain connectivity. And we don't have ambulances and fire=
trucks at every corner and hospitals in every subdivision. Despite the al=
most incalculable value of life, there are still limitations on all the ser=
vices provided to residences. Would I like to have the same uptime at my h=
ome as we have in the CO? or data center? Sure, but collectively we aren't=
willing, nay, able, to pay that price.
Frank
-----Original Message-----
From: William Herrin [mailto:bill@herrin.us]
Sent: Saturday, August 04, 2012 11:15 PM
To: Andy Koch
Cc: nanog@nanog.org
Subject: Re: Verizon FiOS - is BGP an option?
<snip>
On Sat, Aug 4, 2012 at 10:26 PM, Nathan Eisenberg <nathan@atlasnetworks.us>=
wrote:
>> Residences aren't critical infrastructure, no matter how angry the=20
>> owners
get.
>
> 911 access isn't a critical service? Fire and security panels aren't
critical services?
> If basic life safety and property protection aren't critical services,=20
> I'm
not sure what is.
Whether each individual's residence contains critical infrastructure is a d=
ecision best left up to that individual. By necessity that makes the upstre=
am aggregation components critical infrastructure. No different than it was=
for POTS 20 years ago.
The Internet isn't just a toy any more. It's the primary communications cha=
nnel in to many folks homes and well on its way to becoming the primary cha=
nnel period.
Regards,
Bill Herrin
--
William D. Herrin ................ herrin@dirtside.com bill@herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls C=
hurch, VA 22042-3004
