[160685] in North American Network Operators' Group
Re: Muni fiber: L1 or L2?
daemon@ATHENA.MIT.EDU (Stephen Sprunk)
Mon Feb 11 13:56:22 2013
Date: Mon, 11 Feb 2013 12:54:08 -0600
From: Stephen Sprunk <stephen@sprunk.org>
To: nanog@nanog.org
In-Reply-To: <CAMrdfRySSA7CaLxu2FURcZVYqnrLvHTYCUhMetjwwQbwdTg0QA@mail.gmail.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
This is a cryptographically signed message in MIME format.
--------------ms090503060300040609030307
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On 05-Feb-13 11:37, Scott Helms wrote:
> On Tue, Feb 5, 2013 at 11:30 AM, Jay Ashworth <jra@baylink.com> wrote:
>> ----- Original Message -----
>>> From: "Scott Helms" <khelms@zcorum.com>
>>>> Yes it does... It locks you into whatever is supported on the ring.
>>> I don't know how I can explain this more plainly, I can (more accurat=
ely have) taken a fiber build that was created as a ring & spoke SONET sy=
stem and with the same fiber plant overlaid that with GigE and ATM (furth=
er back in time) to backhaul for PON, DSL, VOIP, and direct Active Ethern=
et.
>> "Overlaid"? Could you clarify that?
> Sure, ring, hub & spoke, home run, star these are all descriptions of t=
he physical architecture and many layer 2 technologies will happily use t=
hem all including Ethernet. To use a specific example an existing SONET =
ring (OC-3 to be precise) had be in service with an ILEC for more than a =
decade. This physical topology was a common one with a physical ring of =
fiber (32 strands, yes this was built back in the day) connected to Add/D=
rop
> Multiplexers (Fujitsu IIRC) along the ring as needed to deliver 25,000 =
or shorter copper loops either directly from the same cabinet that ADM wa=
s in or from a subtended Digital Loop Carrier off of a spur (collapsed ri=
ng) of the ring. Now, SONET connections work off a pair of fibers, one f=
or transmit and one for receive. To run Ethernet (initially 100mbps but =
now 10G) we simply lit 2 of the remaining 30 strands to overlay an Ethern=
et ring on top of the SONET ring. We then placed switches in the same re=
mote cabinets we had the ADMs and DLCs and started trenching the fiber dr=
ops.
=2E.. but now you are dictating what technology is used, via the active
aggregation equipment (i.e. ADMs) you installed at your nodes on the
ring. Also, the fiber provider now has to maintain and upgrade that
active aggregation equipment, as opposed to just patching fiber from one
port to another.
The point of this exercise is to design and implement a fiber plant that
can support _any_ technology, including ones that haven't even been
invented yet.
>> Owen's assertion (and mine) is that a loop architecture *requires* act=
ive equipment, suited to the phy layer protocol, at each node. And while=
those loop fibers are running SONET, they can't be running anything else=
at the same time.
> You're confounding the physical layer topology with the layer 2 protoco=
l. You can't run SONET and Ethernet on the same physical fiber at the sa=
me time (unless you use WDM but that's confusing the discussion) but you'=
d never build a ring of fiber with only two strands.
If you're going to dictate SONET ADMs, with a fixed set of downstream
connection types, why _not_ build your ring with one pair of fiber?=20
Hint: the fiber itself is a tiny fraction of the total cost.
You're optimizing the wrong variable as a result of assuming you can
predict what technology will be used 50+ years in the future.
>>> This wasn't always true because we've only had 40G and 100G Ethernet =
for carrier networks for a few years. In the past we were limited by how =
big of an etherchannel network we could use for the ring. I'd also point =
out that the ring architecture is optimal for redundancy since you have f=
ewer fiber bundles to get cut in the field and any cut to your ring gets =
routed around the ring by ERPS (http://en.wikipedia.org/wiki/ERPS) in les=
s than 50 milliseconds.
>> I infer from that continuation of your thought that you mean the secon=
d: active optical muxes out in the plant.
>>
>> I'm sure I've made clear why that design limits me in ways I don't wan=
t to be limited when building a fiber plant for a 50 year lifetime, but l=
et's address your responses below.
> The only limitation you have is a limited supply of total fibers (hint,=
this is a big reason why its cheaper to build and run).
Exactly! Lay enough fiber that you don't _need_ aggregation at the
local level, i.e. enough that you can patch _every_ customer connection
directly to their destination of choice without any active aggregation
equipment at all. Every pair of fiber can be running whatever
technology the customer desires, whether that is SONET, Ethernet, or
something else that hasn't even been invented yet.
>> The vast majority of businesses don't want [dark fiber] at the price
>> they have to pay for it now -- or more to the point, the consultants
>> who do their IT don't. You have no real way, I should think, to
>> extrapolate whether that will continue as prices drop, especially if
>> sharply.
> The vast majority of businesses don't know and don't care about HOW
> their connectivity is delivered and wouldn't know the difference
> between Layer 1 and Layer 2 if it punched them in the face. Almost all
> businesses want INTERNET connectivity at the highest quality & speed
> at the lowest cost and that's it. There are a small percentage, mainly
> larger businesses, that do have special requirements, but those
> special requirements very seldom include a L1 anything.
Most customers will buy from a service provider, who lights the fiber.=20
The point of dark fiber is that the service provider gets to decide how
to light the fiber to said customers, allowing competition based on
innovation. If the fiber owner puts active aggregation equipment in the
path, though, that means the technologies available are dictated by that
equipment's capabilities--and you have introduced another point of
failure into the system.
Sure, almost nobody asks for dark fiber today because they know it costs
several orders of magnitude more than a T1 or whatever. However, if the
price for dark fiber were the same (or lower), latent demand would
materialize. Why would I pay through the nose for a T1 when I can light
the fiber myself with 10GE for $20/mo?
> If your customer base is primarily residential with a few businesses
> (hospitals and schools also count here) then you'll be lucky to sell a
> handful of L1 connections and some of the people who will be
> interested will want it for very low bit rate (means low price too)
> uses like RS-232 over fiber for managing SCADA nodes or other
> telemetry pieces.
Why should the fiber owner care what they use it for? It's just dark
fiber, patched from one place to another, so the rental price is the
same whether they light it at 10Mb/s or 10x100Gb/s.
What you're missing is that in this model, _every_ connection is L1 from
the fiber owner's perspective. Let service providers worry about L2 and
above.
> The problem I see is that you seem to think that by building the L1
> piece you're going to have ISPs that are eager to serve your
> customers. If your demographics are like most small towns in the US
> that just isn't very likely. Any ISP partner is going to have to build
> and maintain a lot of infrastructure before they can serve the first
> end user and your "no upfront engineering" is simply not true unless
> you're going to configure and run MetroE and/or GPON shelves for them.
> In any sharing scenario (L1, L2, or L3) the ISP is going to have to
> connect to you with enough bandwidth to serve those end users as well.
> How many service providers have expressed interest? Have you talked
> pricing for the loops and colo space yet?=20
Why would the ISP "have to build and maintain a lot of infrastructure"?=20
All they need is a fiber-capable Ethernet switch in a colo to turn up
their first customer. That's a lot simpler than trying to turn up their
first customer via an ILEC's DSLAM, for instance.
There's nothing wrong with the muni operating a L2 (or even L3) carrier
of last resort, just to ensure that _some_ useful service is available
to residents. However, it should (a) be priced high enough to attract
competitors and (b) be a distinct entity, treated by the fiber arm as no
different from any other L1 customer. None of the shenanigans like the
ILECs play, where the wholesale rate to competitors is higher than the
retail rate for the ILEC's own service.
> Its an admirable goal, but you're never going to have CCIEs (probably
> not even CCNAs) doing installs. Installation is, has been, and will in
> all likelihood continue to be done by people with limited skill sets.
> You building your own fiber plant and making it easier for ISPs to
> connect isn't going to change that.
You're missing the simplicity of dark fiber. The carrier orders a L1
circuit from a customer to their facility. The L1 provider just patches
one fiber pair to another fiber pair, which can be done by a trained
monkey. Then the carrier connects their own equipment to the fiber at
their own facility and at the customer site, everything lights up and
the spice^Wdata flows. Again, that can be done by a trained monkey.=20
You don't need a CCIE or even a CCNA to do this. Heck, it's even
simpler than what's required today for DSL, cable or satellite installers=
=2E
(Note that inside wiring is a completely separate issue, and carriers
_will_ have to train techs on how to do that since few are familiar with
fiber, but that is an optional service they can charge extra for. The
L1 provider's responsibility ends at the NIU on an outside wall, same as
an ILEC's, so it's not their problem in the first place.)
S
--=20
Stephen Sprunk "God does not play dice." --Albert Einstein
CCIE #3723 "God is an inveterate gambler, and He throws the
K5SSS dice at every possible opportunity." --Stephen Hawking
--------------ms090503060300040609030307
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFezCC
BXcwggNfoAMCAQICAwxEDzANBgkqhkiG9w0BAQUFADB5MRAwDgYDVQQKEwdSb290IENBMR4w
HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmlu
ZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xMjA5
MjMwMTA3NDlaFw0xMzAzMjIwMTA3NDlaMD0xGDAWBgNVBAMTD0NBY2VydCBXb1QgVXNlcjEh
MB8GCSqGSIb3DQEJARYSc3RlcGhlbkBzcHJ1bmsub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1qAUmWE5Nv/9HFl7f52VrEoGqhbYG1rPE7go13pSxgUBi02iDNKyDTJH
b/CswpO/FE9p34QbUlnOXY3LRrLh+/NaS496breDA8mRKCceNHIPaiK3Mwpl0q787WQalrp9
mBV2jKHobOMOWbtNiuYSq6k4qfqIvFJvGOsgizcqeMjjLyR8GimjGCQYuH2o970PhlNidpAj
mVMCBehc3RhjmevEn4ydIL7j0bmFj9YuLxaoISq6icdh7VFxqAzdEbHhQHKp1N8aNq5yhcWD
M//Bq8MpMd1+zNse/pVc7/42G5iArYkImUbu/P48PgR8ZX9D59hJxP7kRyKVT9dbTNkh4wID
AQABo4IBQjCCAT4wDAYDVR0TAQH/BAIwADBWBglghkgBhvhCAQ0ESRZHVG8gZ2V0IHlvdXIg
b3duIGNlcnRpZmljYXRlIGZvciBGUkVFIGhlYWQgb3ZlciB0byBodHRwOi8vd3d3LkNBY2Vy
dC5vcmcwDgYDVR0PAQH/BAQDAgOoMEAGA1UdJQQ5MDcGCCsGAQUFBwMEBggrBgEFBQcDAgYK
KwYBBAGCNwoDBAYKKwYBBAGCNwoDAwYJYIZIAYb4QgQBMDIGCCsGAQUFBwEBBCYwJDAiBggr
BgEFBQcwAYYWaHR0cDovL29jc3AuY2FjZXJ0Lm9yZzAxBgNVHR8EKjAoMCagJKAihiBodHRw
Oi8vY3JsLmNhY2VydC5vcmcvcmV2b2tlLmNybDAdBgNVHREEFjAUgRJzdGVwaGVuQHNwcnVu
ay5vcmcwDQYJKoZIhvcNAQEFBQADggIBABcg4VFbPyYyZnEHpU1j7VJ3rXpNleEdxSACqVGk
zC8ViFJE4ka0RLaHTZjWBYiVfCmEd2aJeJlXe73picAE2g9pJWWmmQ/5s3b2C4ARZODr95GM
CFyxso3/KX2Iud9soUR1YDhM4j9Zi/dIjkaTRnGTwh0tvrtpEXRpcMMLJWniBayYfK2K99FW
YiYHW6wQz/xHDtmZtY2xYaEgT2aFNx2PiJq0n8NAchfpuODEN+Pit9aeNd3UjmPHTYm9JE5Y
MmoS/1ReSrrvz3zkFxf4Zz2FCHYRT6tzcX5UXJVC1q0B/xEDkUztw5Xrb8Atwg7wdxeoD0S2
U3hcXmr7+DutZ+ciYKGcJQt2iJ+SOYOUzxZ8oMpmasvkeAVafT6O406r3NEeevtFZgatmUHy
5eD/o/cQ9jHFN3pgVdl9dY2lM6quVTeFMBNJEnF+nmkelfLagoixwKQlRAW/hrK2xBLv7/ds
3mdYhIsoXgScjw1CYVjbz7jpjjxwlHFC33e60x8zZNYN0ZEnGsyflreBsc6D5O2fNeOCh43M
mTPsPpMDFXIMjyyBV0jHJ8/g3UEwKP3Vgqf5PaDHjdrxNfUkIcz8Na4A1+sUS0X9AVmKSwQB
hCS0Fjo+AQA0ezLLVHprls0iCwAH7Dmrd6umN27xzDMzyc8ZTxMYbZknqJuyejmXF38dMYID
lDCCA5ACAQEwgYAweTEQMA4GA1UEChMHUm9vdCBDQTEeMBwGA1UECxMVaHR0cDovL3d3dy5j
YWNlcnQub3JnMSIwIAYDVQQDExlDQSBDZXJ0IFNpZ25pbmcgQXV0aG9yaXR5MSEwHwYJKoZI
hvcNAQkBFhJzdXBwb3J0QGNhY2VydC5vcmcCAwxEDzAJBgUrDgMCGgUAoIIB6DAYBgkqhkiG
9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xMzAyMTExODU0MDhaMCMGCSqG
SIb3DQEJBDEWBBQ4PBs8diSnNUdyyJJlaC3onslCYDBfBgkqhkiG9w0BCQ8xUjBQMAsGCWCG
SAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYF
Kw4DAgcwDQYIKoZIhvcNAwICASgwgZEGCSsGAQQBgjcQBDGBgzCBgDB5MRAwDgYDVQQKEwdS
b290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENl
cnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9y
ZwIDDEQPMIGTBgsqhkiG9w0BCRACCzGBg6CBgDB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYD
VQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBB
dXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZwIDDEQPMA0GCSqG
SIb3DQEBAQUABIIBAJf0Q775T78QByv7/X7/Cj6uC258fXYWQjaB0uAKMv+UHwxgldeJmH3B
xhvXlk1+KcTv4jNxt/F2oI74Q2+6BBeIS3dbZGfVhdv6UFF6i0dPa0/DT0I9iyCos3zs3cj4
xdNz9fbdaY1MivHoFUfykXW1/Ck7LI1D0EBejMiXq+0jlUTWVLxoY7Tctuebj4nvXMxavxdA
dTZKN8JJZpdsCO2dae0uig/0yKBtpHzQH+aH4jPA1xUf5o371F3TQRoGuKbptN3fbQzBI+oJ
2ON9hHLTbQMXnl6XOaD36S4g6iHpnXLLZ9lW1jfg1ESKlBbZxdYoTJYFBPtOboyuRn6+WZMA
AAAAAAA=
--------------ms090503060300040609030307--
