[119269] in North American Network Operators' Group
Re: Resilience - How many BGP providers
daemon@ATHENA.MIT.EDU (David Hiers)
Wed Nov 11 19:06:35 2009
In-Reply-To: <63076.1257982897@baklawasecrets.com>
Date: Wed, 11 Nov 2009 16:05:55 -0800
From: David Hiers <hiersd@gmail.com>
To: adel@baklawasecrets.com
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
It is wise to stack the deck in your favor, but you'll never really
know how much real redundancy you've purchased:
http://www.atis.org/ndai/ATIS_NDAI_Final_Report_2006.pdf
David
On Wed, Nov 11, 2009 at 3:41 PM, <adel@baklawasecrets.com> wrote:
> I suppose I could take the whole resilience thing further and further and=
further. =A0One of the replies used a phrase which I thing captured the pr=
oblem quite nicely: "diminishing returns".
> Basically I could spend lots and lots of money to try and eliminate all s=
ingle points of failure. =A0Clearly I don't have the money to do this and w=
hat I'm really trying to establish is at what
> point do the returns start to diminish with regards to obtaining multiple=
transit providers. =A0The answer appears to be "it depends". =A0So if gett=
ing a third BGP peering with divergent paths,
> separate last mile, separate facility and separate router will increase c=
osts by 5x but only increase resilience by 0.001% is it really worth it? =
=A0I'm trying to quantify the resilience of my
> Internet connectivity and quantify the effects of adding more providers. =
=A0Now to run through my case:
>
> - I have one facility to locate BGP routers at. =A0Thats not changing for=
the moment.
> - I can afford two BGP routers.
> - The facility I'm located at tell me they have divergent fibre paths and=
multiple entries into the facility. (Still need to verify this by getting =
them to walk the routes with me)
> - I am going to take transit from two upstreams.
> - I could ask the question as to whether I can peer with separate routers=
on each of the upstreams. =A0i.e. to protect against router failures on th=
eir side.
> - I will make sure that neither upstream peers with the other directly. (=
Does this give me some AS path redundancy?)
>
> So from the above:
>
> - I have no resilience with regards to datacentre location. =A0i.e. if a =
plane fell out of the sky etc., I'm done.
> - I can afford some BGP router resilience on my side. =A0So I should be a=
ble to continue working if a router failure which only affects one of my ro=
uters occurs.
> - I have some resilience in terms of actual fibre paths to the facilites =
where I will be picking up the BGP feeds from. (to be verified)
> - I have some "AS resilience" if this is the right term. =A0So if the AS =
of one of my upstreams drops off the face of the Internet, I can still get =
to the Internet through the AS of my other
> provider
> - Peering with separate routers may give me some resilience for router fa=
ilure on the side of my upstreams? (not totally sure on this)
>
> In this situation, if I add another peering with another upstream, am I r=
eally getting much return in terms of resilience? =A0Or should I spend this=
money examining the many other SPOFs in
> my architecture? =A0I'm perfectly sure there is absolutely no point me pe=
ering with 6 providers, but maybe some gains in peering with 3? =A0I'm tryi=
ng to figure out at what point is adding
> another peering in my case a waste of money.
>
> I haven't gone into switch and power redundancy, because I "think" I unde=
rstand it. =A0I wanted to concentrate on the multiple upstreams question. =
=A0Heads starting to whirl right about now.
>
> Adel
>
>
> On Wed =A0 5:27 PM , "Dylan Ebner" dylan.ebner@crlmed.com sent:
>>
>> You question has many caveats. Just having two providers does not
>> necessarily get you more resiliency. If you have two providers and they =
are
>> terminating on the same router, then you still have a SPOF problem. You
>> also need to look at pysical paths as well. If you have two (or three)
>> providers and they are using a common carrier, then you have a problem a=
s
>> well. For example, GLBX has a small prescence in the Minneapolis metro. =
If
>> I were to use them as a provider, they would use Qwest as a last mile. I=
f
>> my other provider is Qwest (which it is), I may not have path
>> divergence.Facilities are important too. We have three upstreams; Qwest,=
MCI and ATT.
>> The facility only has two entrances, so that means two of these are in t=
he
>> same conduit. IF you only have one entrance, all you connections are goi=
ng
>> to run through that conduit, and that makes you susceptable to a rouge
>> backhoe.
>> You are on the right track to question your resilancy. Some upstreams ca=
n
>> offer good resilancy with multiple feeds. Others cannot. I would start w=
ith
>> your provider and see what you are getting. Maybe you already have path
>> divergence, sperate last miles, and multiple paths in the isp core. =A0I=
f you
>> go with multiple providers, you want to make sure you don't risk losing
>> something you already have.
>>
>>
>>
>> -----Original Message-----
>> From: adel@baklawasecrets.com [adel@
>> baklawasecrets.com] Sent: Wednesday, November 11, 2009 11:14 AM
>> To: nanog@nanog.o
>> rgSubject: Resilience - How many BGP providers
>>
>>
>>
>> Hi,
>>
>> After recent discussions on the list, I've been thinking about the
>> affectsof multiple BGP feeds to the overall resilience of Internet
>> connectivityfor my organisation.=A0 So originally when I looked at the d=
esign
>> proposals, there was a provision in there for four connections with the
>> same Internet provider.=A0 Thinking about it and with the valuable input
>> ofmembers on this list, it was obvious that multiple connections from th=
e
>> same provider defeated the aim of providing resilience.
>>
>> So having come to the decision to use two providers and BGP peer with
>> both, I'm wondering how much more resilience I would get by peering
>> with=A0more than two=A0providers.=A0 So will it significantly
>> increase myresilience by peering with three providers for example, as bo=
th of the
>> upstreams I choose will be multihomed to other providers.=A0 Especially
>> asI am only looking at peering out of the UK.
>>
>> Hope the above makes sense.
>>
>> Adel
>>
>>
>>
>>
>>
>
>
>
