[119268] in North American Network Operators' Group
RE: Resilience - How many BGP providers
daemon@ATHENA.MIT.EDU (adel@baklawasecrets.com)
Wed Nov 11 18:41:14 2009
To: "nanog@nanog.org" <nanog@nanog.org>, "Dylan Ebner" <dylan.ebner@crlmed.com>
Date: Wed, 11 Nov 2009 23:41:37 +0000
From: adel@baklawasecrets.com
Reply-To: adel@baklawasecrets.com
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
I suppose I could take the whole resilience thing further and further and f=
urther. One of the replies used a phrase which I thing captured the proble=
m quite nicely: "diminishing returns". =20
Basically I could spend lots and lots of money to try and eliminate all sin=
gle points of failure. Clearly I don't have the money to do this and what =
I'm really trying to establish is at what=20
point do the returns start to diminish with regards to obtaining multiple t=
ransit providers. The answer appears to be "it depends". So if getting a =
third BGP peering with divergent paths,=20
separate last mile, separate facility and separate router will increase cos=
ts by 5x but only increase resilience by 0.001% is it really worth it? I'm=
trying to quantify the resilience of my=20
Internet connectivity and quantify the effects of adding more providers. N=
ow to run through my case:
- I have one facility to locate BGP routers at. Thats not changing for the=
moment.
- I can afford two BGP routers.
- The facility I'm located at tell me they have divergent fibre paths and m=
ultiple entries into the facility. (Still need to verify this by getting th=
em to walk the routes with me)
- I am going to take transit from two upstreams.
- I could ask the question as to whether I can peer with separate routers o=
n each of the upstreams. i.e. to protect against router failures on their =
side.
- I will make sure that neither upstream peers with the other directly. (Do=
es this give me some AS path redundancy?)
So from the above:
- I have no resilience with regards to datacentre location. i.e. if a plan=
e fell out of the sky etc., I'm done.
- I can afford some BGP router resilience on my side. So I should be able =
to continue working if a router failure which only affects one of my router=
s occurs.
- I have some resilience in terms of actual fibre paths to the facilites wh=
ere I will be picking up the BGP feeds from. (to be verified)
- I have some "AS resilience" if this is the right term. So if the AS of o=
ne of my upstreams drops off the face of the Internet, I can still get to t=
he Internet through the AS of my other=20
provider
- Peering with separate routers may give me some resilience for router fail=
ure on the side of my upstreams? (not totally sure on this)
In this situation, if I add another peering with another upstream, am I rea=
lly getting much return in terms of resilience? Or should I spend this mon=
ey examining the many other SPOFs in=20
my architecture? I'm perfectly sure there is absolutely no point me peerin=
g with 6 providers, but maybe some gains in peering with 3? I'm trying to =
figure out at what point is adding=20
another peering in my case a waste of money.
I haven't gone into switch and power redundancy, because I "think" I unders=
tand it. I wanted to concentrate on the multiple upstreams question. Head=
s starting to whirl right about now.
Adel
On Wed 5:27 PM , "Dylan Ebner" dylan.ebner@crlmed.com sent:
>=20
> You question has many caveats. Just having two providers does not
> necessarily get you more resiliency. If you have two providers and they a=
re
> terminating on the same router, then you still have a SPOF problem. You
> also need to look at pysical paths as well. If you have two (or three)
> providers and they are using a common carrier, then you have a problem as
> well. For example, GLBX has a small prescence in the Minneapolis metro. I=
f
> I were to use them as a provider, they would use Qwest as a last mile. If
> my other provider is Qwest (which it is), I may not have path
> divergence.Facilities are important too. We have three upstreams; Qwest, =
MCI and ATT.
> The facility only has two entrances, so that means two of these are in th=
e
> same conduit. IF you only have one entrance, all you connections are goin=
g
> to run through that conduit, and that makes you susceptable to a rouge
> backhoe.
> You are on the right track to question your resilancy. Some upstreams can
> offer good resilancy with multiple feeds. Others cannot. I would start wi=
th
> your provider and see what you are getting. Maybe you already have path
> divergence, sperate last miles, and multiple paths in the isp core. If y=
ou
> go with multiple providers, you want to make sure you don't risk losing
> something you already have.
>=20
>=20
>=20
> -----Original Message-----
> From: adel@baklawasecrets.com [adel@
> baklawasecrets.com] Sent: Wednesday, November 11, 2009 11:14 AM
> To: nanog@nanog.o
> rgSubject: Resilience - How many BGP providers
>=20
>=20
>=20
> Hi,=20
>=20
> After recent discussions on the list, I've been thinking about the
> affectsof multiple BGP feeds to the overall resilience of Internet
> connectivityfor my organisation.=C2=A0 So originally when I looked at the=
design
> proposals, there was a provision in there for four connections with the
> same Internet provider.=C2=A0 Thinking about it and with the valuable inp=
ut
> ofmembers on this list, it was obvious that multiple connections from the
> same provider defeated the aim of providing resilience.=20
>=20
> So having come to the decision to use two providers and BGP peer with
> both, I'm wondering how much more resilience I would get by peering
> with=C2=A0more than two=C2=A0providers.=C2=A0 So will it significantly
> increase myresilience by peering with three providers for example, as bot=
h of the
> upstreams I choose will be multihomed to other providers.=C2=A0 Especiall=
y
> asI am only looking at peering out of the UK.=20
>=20
> Hope the above makes sense.=20
>=20
> Adel
>=20
>=20
>=20
>=20
>=20
