[132744] in North American Network Operators' Group
Re: Level 3 Communications Issues Statement Concerning Comcast's
daemon@ATHENA.MIT.EDU (Peter Bruno)
Tue Nov 30 15:57:59 2010
In-Reply-To: <20101130180256.GA20426@ussenterprise.ufp.org>
Date: Tue, 30 Nov 2010 15:57:47 -0500
From: Peter Bruno <brunopeter@gmail.com>
To: NANOG list <nanog@nanog.org>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
GigaOm has begin tracking this story:
http://gigaom.com/2010/11/30/a-play-by-play-on-the-comcast-and-level-3-spat
On Tue, Nov 30, 2010 at 1:02 PM, Leo Bicknell <bicknell@ufp.org> wrote:
>
> Having been involved with a few peering spats in the past I know
> what is said publically rarely matches the reality behind the scenes.
> In this particular case my spidy sense tells me there is absolutely
> something interesting behind the scenes, but the question is what.
>
> I'd never really paid attention to how Netflix delivers its content.
> It's obviously a lot of bandwidth, and likely part of the issue
> here so I thought I would investigate.
>
> Apparently Akamai has been the primary Netflix streaming source
> since March. =A0LimeLight Networks has been a secondary provider, and
> it would appear those two make up the vast majority of Netflix's
> actual streaming traffic. =A0I can't tell if Netflix does any streaming
> out of their own ASN, but if they do it appears to be minor.
>
> Here's a reference from the business side of things:
> http://www.fool.com/investing/general/2010/11/11/netflix-takes-streaming-=
to-a-new-level.aspx
>
> This is also part of the reason I went back to the very first message in
> this thread to reply:
>
> In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W.=
Gilmore wrote:
>> <http://www.marketwatch.com/story/level-3-communications-issues-statemen=
t-concerning-comcasts-actions-2010-11-29?reflink=3DMW_news_stmp>
>>
>> I understand that politics is off-topic, but this policy affects operati=
onal aspects of the 'Net.
>
> Patrick works for Akamai, it seems likely he might know more about
> what is going on. =A0Likely he can't discuss the details, but wanted
> to seed a discussion. =A0I'd say that worked well.
>
> I happen to be a Comcast cable modem customer. =A0Gooling for people
> who had issues getting to Netflix streaming turned up plenty of
> forum posts with traceroutes to Netflix servers on Akamai and
> Limelight. =A0I did traceroutes to about 20 of them from my cable
> modem, and it's clear Comcast and Akamai and Comcast and Limelight
> are interconnected quite well. =A0Akamai does not sell IP Transit,
> and I'm thinking it is extremely unlikely that Comcast is buying
> transit from Limelight. =A0I will thus conclude that these are either
> peering relationships, or that they have cut some sort of special
> "CDN Interconnect" deal with Comcast.
>
> But what about Level 3? =A0One of my friends I was chatting with on AIM
> said they thought Comcast was a Level 3 customer, at least at one time.
> Google to the rescue again.
>
> Level 3 provides fiber to Comcast (20 year deal in 2004):
> http://blog.tmcnet.com/blog/tom-keating/voip/level-3-and-comcast.asp
>
> Level 3 provides voice services/support to Comcast:
> http://cable.tmcnet.com/news/2005/jul/1168088.htm
>
> Perhaps the most interesting though is looking up an IP on Comcast's
> local network here in my city in L3's looking glass:
> http://lg.level3.net/bgp/bgp.cgi?site=3Dsjo1&target=3D68.86.240.141
>
> Slightly reformatting for your viewing pleasure, along with my comments:
>
> =A0 =A0 =A0Community: North_America
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Lclprf_100
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Level3_Customer =A0 =A0 =A0 # Level 3 thi=
nks they are a customer
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 United_States
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 San_Jose
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 EU_Suppress_to_Peers
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS174 =A0 =A0 # Cogent
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS1239 =A0 =A0# Sprint
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS1280 =A0 =A0# ISC
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS1299 =A0 =A0# Telia
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS1668 =A0 =A0# AOL
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS2828 =A0 =A0# XO
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS2914 =A0 =A0# NTT
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS3257 =A0 =A0# TiNet
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS3320 =A0 =A0# DTAG
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS3549 =A0 =A0# GBLX
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS3561 =A0 =A0# Savvis
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS3786 =A0 =A0# LG DACOM
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS4637 =A0 =A0# Reach
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS5511 =A0 =A0# OpenTransit
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS6453 =A0 =A0# Tata
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS6461 =A0 =A0# AboveNet
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS6762 =A0 =A0# Seabone
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS7018 =A0 =A0# AT&T
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Suppress_to_AS7132 =A0 =A0# AT&T (ex SBC)
>
> So it would appear Comcast is a transit customer of Level 3 (along with
> buying a lot of other services from them). =A0I'm going to speculate that
> the list of supressed ASN's are peers of both Level 3 and Comcast, and
> Comcast is going that so those peers can't send some traffic through
> Level 3 in attempt to game the ratios on their direct connections to
> Comcast.
>
> Now a more interesting picture emerges. =A0Let me emphasize that this is
> AN EDUCATED GUESS on my part, and I can't prove any of it.
>
> Level 3 starts talking to Netflix, and offers them a sweetheart deal to
> move traffic from Akamai to Level 3. =A0Part of the reason they are
> willing to go so low on the price to Netflix is they will get to double
> dip by charging Netflix for the bits and charging Comcast for the bits,
> since Comcast is a customer! =A0But wait, they also get to triple dip,
> they provide the long haul fiber to Comcast, so when Comcast needs more
> capacity to get to the peering points to move the traffic that money
> also goes back to Level 3! =A0Patrick, from Akamai, is unhappy at losing
> all the business, and/or bemused at the ruckus this will cause and
> chooses to kick the hornets nest on NANOG.
>
> One last thing, before we do some careful word parsing. =A0CDN's like
> Akamai and LimeLight want to be close to the end user, and the
> networks with end users want them to be close to the end user. =A0It
> doesn't make sense to haul the bits across the country for any party
> involved. =A0Akamai does this by locating clusters inside providers
> networks, LimeLight does it by provisioning bandwidth from their
> data centers directly to distribution points on eyeball networks.
>
> So let's go back and look at the public statements now:
>
> Level 3 said:
> http://www.marketwatch.com/story/level-3-communications-issues-statement-=
concerning-comcasts-actions
> -2010-11-29?reflink=3DMW_news_stmp
>
> =A0"On November 19, 2010, Comcast informed Level 3 that, for the first
> =A0 time, it will demand a recurring fee from Level 3 to transmit Interne=
t
> =A0 online movies and other content to Comcast's customers who request su=
ch
> =A0 content."
>
> Comcast said:
> http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html
>
> =A0"Comcast has long established and mutually acceptable commercial
> =A0 arrangements with Level 3's Content Delivery Network (CDN) competitor=
s
> =A0 in delivering the same types of traffic to our customers. Comcast
> =A0 offered Level 3 the same terms it offers to Level 3's CDN competitors
> =A0 for the same traffic."
>
> You can make both of these statements make sense if the real situation
> is that Comcast told Level 3 they needed to act like a CDN if they were
> going to host Netflix. =A0Rather than having Comcast pay as a customer,
> they needed to show up in various Comcast distribution centers around
> the country where they could drop traffic off "locally". =A0To Comcast
> this is the same deal other CDN's get, it matches their statement. =A0To
> Level 3, this means paying a fee for bandwidth to these locations, and
> being that they are Comcast locations it may even mean colocation fees
> or other charges directly to Comcast. =A0Comcast said if you're not going
> to show up and do things like the CDN players then we're going to hold
> you to a reasonable peering policy, like we would anyone else making us
> run the bits the old way.
>
> The most interesting thing to me about all of this is these companies
> clearly had a close relationship, fiber, voice, and IP transit all on
> long term deals. =A0If my speculation is right I'm a bit surprised Level =
3
> would choose to piss off such a long term large customer by bringing
> Netflix to the party like this, which is one of the reasons I doubt my
> speculation a bit.
>
> But, to bring things full circle, neither of the public statements tell
> the whole story here. =A0They each tell one small nugget, the nugget that
> side wants the press to run with so they can score political points.
>
> Business is messy, and as I've said throughout this thread this isn't
> about peering policies or ratios, there are deeper business interests
> on both sides. =A0This article:
> http://www.marketwatch.com/story/level-3-outlines-network-expansion-on-ne=
tflix-pact-lvlt-nflx-llnw-akam-2010-11-11
>
> Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix.
> I'm sure a lot of that is going to Comcast users, since they are the
> largest residential broadband ISP.
>
> Messy. =A0Very messy.
>
> --
> =A0 =A0 =A0 Leo Bicknell - bicknell@ufp.org - CCIE 3440
> =A0 =A0 =A0 =A0PGP keys at http://www.ufp.org/~bicknell/
>
