[171879] in North American Network Operators' Group
Re: Observations of an Internet Middleman (Level3)
daemon@ATHENA.MIT.EDU (Scott Berkman)
Thu May 15 16:16:34 2014
X-Original-To: nanog@nanog.org
Date: Thu, 15 May 2014 16:16:26 -0400
From: Scott Berkman <scott@sberkman.net>
To: "nanog@nanog.org" <nanog@nanog.org>
In-Reply-To: <8AFC1EC8-5F2E-4645-B57C-A147FC56B390@cable.comcast.com>
Errors-To: nanog-bounces@nanog.org
I guess I should have said this another way.
Everyone knows Comcast uses (or used) Sandvine for shaping (unless=20
they've finished building a new probably internal solution, I'm sure=20
this is another secret we'll only have rumors to work with, ). By=20
shaping other traffic (IPSEC VPNs or P2P traffic for example) into BE or=20
limited queues, and then not shaping or prioritizing traffic to test=20
sites, the customer gets invalid data and expectations.
I'm no longer in a position to test this for reporting to the FCC as=20
suggested, but in a previous life we were able to prove it enough for=20
the Comcast customer getting the short end of the stick to stop yelling=20
at us and get a new provider, which of course made everyone involved=20
happier.
If Comcast has since actually completely torn down that infrastructure=20
to openly comply with the FCC's rules that came out of the legal battle=20
regarding P2P shaping, again congrats to the customers that hopefully=20
get to see some benefit. I'd love to see a case study published by=20
Comcast on how that project went and what the impacts to the network and=20
bottom line were.
-Scott
On 05/15/2014 11:50 AM, McElearney, Kevin wrote:
> There is no gaming on measurements and disputes are isolated and tempor=
ary with issues not unique over the history of the internet. I think all=
the same rhetorical quotes continue to be reused
>
> - Kevin
>
>> On May 15, 2014, at 11:43 AM, "Scott Berkman" <scott@sberkman.net> wro=
te:
>>
>> Unfortunately these build-outs are primarily in subscriber facing band=
width and number of headend locations (to add more customers to the netwo=
rk). These peering point/transit connection issues have been going on fo=
r a long time, evidenced by Level 3 coming out with this post. Comcast i=
s also suspiciously absent from public exchanges (TelX's TIE would be one=
example) while many of their competitors participate for the benefit of =
the Internet as a whole and their customers.
>>
>> Measured broadband is also a game, because its very easy for large pro=
viders to give priority to (or otherwise "help") known speed test and sim=
ilar sites, giving customers a false impression of their available capaci=
ty or performance. We've all seen cases where customers have some amazin=
g result on their favorite test site, and then real world performance can=
't even come close.
>>
>> That said, if Comcast does or is making efforts to finally resolve thi=
s, more power to them and congratulations to their customers. Unfortunate=
ly trying to brute-force the industry and external content providers tell=
s a very different story. Where is Comcast's official blog post showing =
evidence as to where they do ensure their peering and or transit to the l=
argest Tier 1 providers are not congested? Instead all we see are policy=
arguments about who should pay for what, while users continue to suffer.
>>
>> This is really similar to when TV providers have spats with content ow=
ners, and the result is the end users missing out on something they are p=
aying for. It is good for related industries and the large players in e=
ach to keep working with each other in open ways to keep pricing reasonab=
le (as opposed to working together in hiding to price fix), but it is not=
OK to do so by throwing tantrums and making everyone involved suffer.
>>
>> -Scott
>>
>>
>>> On 05/15/2014 10:57 AM, McElearney, Kevin wrote:
>>> Upgrades/buildout are happening every day. They are continuous to ke=
ep ahead of demand and publicly measured by SamKnows (FCC measuring broad=
band), Akamai, Ookla, etc
>>>
>>> What is not well known is that Comcast has been an existing commercia=
l transit business for 15+ years (with over 8000 commercial fiber custome=
rs). Comcast also has over 40 balanced peers with plenty of capacity, an=
d some of the largest Internet companies as customers.
>>>
>>> - Kevin
>>>
>>> 215-313-1083
>>>
>>>> On May 15, 2014, at 10:19 AM, "Owen DeLong" <owen@delong.com> wrote:
>>>>
>>>> Oh, please do explicate on how this is inaccurate=85
>>>>
>>>> Owen
>>>>
>>>>> On May 14, 2014, at 2:14 PM, McElearney, Kevin <Kevin_McElearney@ca=
ble.comcast.com> wrote:
>>>>>
>>>>> Respectfully, this is a highly inaccurate "sound bite"
>>>>>
>>>>> - Kevin
>>>>>
>>>>> 215-313-1083
>>>>>
>>>>>> On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com> wrote=
:
>>>>>>
>>>>>> Yes, the more accurate statement would be aggressively seeking new
>>>>>> ways to monetize the existing infrastructure without investing in =
upgrades
>>>>>> or additional buildout any more than absolutely necessary.
>>>>>>
>>>>>> Owen
>>>>>>
>>>>>> On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com> wrot=
e:
>>>>>>
>>>>>>>> So they seek new sources of revenues, and/or attempt to thwart
>>>>>>>>> competition any way they can.
>>>>>>> No to the first. Yes to the second. If they were seeking new sour=
ces of
>>>>>>>> revenue, they'd be massively expanding into un/der served market=
s and
>>>>>>>> aggressively growing over the top services (which are fat margin=
).
>>>>>>> Sure they are (seeking new sources of revenue). They're not nece=
ssarily
>>>>>>> creating new products or services, i.e. actually adding any value=
, but they
>>>>>>> are finding ways to extract additional revenue from the same pipe=
s, e.g.
>>>>>>> through paid peering with content providers.
>>>>>>>
>>>>>>> I'm not endorsing this; just pointing out that you two are actual=
ly in
>>>>>>> agreement here.
>>>>>>>
>>>>>>> --
>>>>>>> Hugo
>>>>>>>
>>>>>>>
>>>>>>>>> On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote:
>>>>>>>>>
>>>>>>>>> On 2014-05-14 02:04, Jean-Francois Mezei wrote:
>>>>>>>>>
>>>>>>>>> On 14-05-13 22:50, Daniel Staal wrote:
>>>>>>>>>
>>>>>>>>> They have the money. They have the ability to get more money. =
*They see
>>>>>>>>>> no reason to spend money making customers happy.* They can ma=
ke more
>>>>>>>>>> profit without it.
>>>>>>>>> There is the issue of control over the market. But also the pre=
ssure
>>>>>>>>> from shareholders for continued growth.
>>>>>>>> Yes. That is true. Except that it's not.
>>>>>>>>
>>>>>>>> How do service providers grow? Let's explore that:
>>>>>>>>
>>>>>>>> What is growth for a transit provider?
>>>>>>>>
>>>>>>>> More (new) access network(s) (connections).
>>>>>>>> More bandwidth across backbone pipes.
>>>>>>>>
>>>>>>>>
>>>>>>>> What is growth for access network?
>>>>>>>> More subscribers.
>>>>>>>>
>>>>>>>> Except that the incumbent carriers have shown they have no inter=
est in
>>>>>>>> providing decent bandwidth to anywhere but the most profitable r=
ate
>>>>>>>> centers. I'd say about 2/3 of the USA is served with quite terri=
ble access.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> The problem with the internet is that while it had promises of =
wild
>>>>>>>>> growth in the 90s and 00s, once penetration reaches a certain l=
evel,
>>>>>>>>> growth stabilizes.
>>>>>>>> Penetration is ABYSMAL sir. Huge swaths of underserved americans=
exist.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> When you combine this with threath to large incumbents's media =
and media
>>>>>>>>> distribution endeavours by the likes of Netflix (and cat videos=
on
>>>>>>>>> Youtube), large incumbents start thinking about how they will b=
e able to
>>>>>>>>> continue to grow revenus/profits when customers will shift spen=
ding to
>>>>>>>>> vspecialty channels/cableTV to Netflix and customer growth will=
not
>>>>>>>>> compensate.
>>>>>>>> Except they aren't. Even in the most profitable rate centers, th=
ey've
>>>>>>>> declined to really invest in the networks. They aren't a real bu=
siness. You
>>>>>>>> have to remember that. They have regulatory capture, natural/def=
acto
>>>>>>>> monopoly etc etc. They don't operate in the real world of
>>>>>>>> risk/reward/profit/loss/uncertainty like any other real business=
has to.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> So they seek new sources of revenues, and/or attempt to thwart
>>>>>>>>> competition any way they can.
>>>>>>>> No to the first. Yes to the second. If they were seeking new sou=
rces of
>>>>>>>> revenue, they'd be massively expanding into un/der served market=
s and
>>>>>>>> aggressively growing over the top services (which are fat margin=
). They did
>>>>>>>> a bit of an advertising campaign of "smart home" offerings, but =
that seems
>>>>>>>> to have never grown beyond a pilot.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> The current trend is to "if you can't fight them, jon them" whe=
re
>>>>>>>>> cablecos start to include the Netflix app into their proprietar=
y set-top
>>>>>>>>> boxes. The idea is that you at least make the customer continue=
to use
>>>>>>>>> your box and your remote control which makes it easier for them=
to
>>>>>>>>> switch between netflix and legacy TV.
>>>>>>>> True. I don't know why one of the cablecos hasn't licensed roku,=
added
>>>>>>>> cable card and made that available as a "hip/cool" set top box o=
ffering and
>>>>>>>> charge another 10.00 a month on top of the standard dvr rental.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Would be interesting to see if those cable companies that are ag=
reeing
>>>>>>>>> to add the Netflix app onto their proprietary STBs also play p=
eering
>>>>>>>>> capacity games to degrade the service or not.
>>>>>>>> So how is the content delivered? Is it over the internet? Or is =
it over
>>>>>>>> the cable plant, from cable headends?
