[121484] in North American Network Operators' Group
RE: Enhancing automation with network growth
daemon@ATHENA.MIT.EDU (Erik L)
Wed Jan 20 22:57:20 2010
From: Erik L <erik_list@caneris.com>
To: Steve Bertrand <steve@ibctech.ca>, nanog list <nanog@nanog.org>
Date: Wed, 20 Jan 2010 22:52:39 -0500
In-Reply-To: <4B57C1FA.5000207@ibctech.ca>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
>=20
> I'm reaching the point where adding in a new piece of infrastructure
> hardware, connecting up a new cable, and/or assigning address=20
> space to a
> client is nearly 50% documentation and 50% technical.
>=20
A common problem :)
> One thing that would take a major load off would be if my MRTG system
> could simply update its config/index files for itself, instead of me
> having to do it on each and every port change.
>=20
> Can anyone offer up ideas on how you manage any automation in this
> regard for their infrastructure gear traffic graphs?=20
> (Commercial options
> welcome, off-list, but we're as small as our budget is).
>=20
Not sure how you're doing your graphs currently, but have you considered Ca=
cti?
> I use a mix of Cisco/FreeBSD&Quagga for routers, and Cisco/HP for
> switches, so it is not as simple as throwing a single command at all
> configs.
>=20
> All feedback welcome, especially if you are in the same boat. My IP
> address documentation/DNS is far more important than my traffic stats,
> but it really hurts when you've forgotten about a port three=20
> months ago
> that you need to know about now.
>=20
First, I'll throw out a bit of what we do and it might give some ideas, tho=
ugh not necessarily good ones. We use Linux/Quagga routers, in-house-modifi=
ed Linux-based LNSs, and HP switches. Some of our configuration and change =
management is done via cfengine, backed by subversion. The latter yields th=
e added benefit of revision control and all the other good stuff you can ge=
t from svn in such a scenario. Unfortunately this is only part of the confi=
g/graphs/docs/DNS/IPs/OSS equation and we don't have everything fully integ=
rated yet (nor is there a business case for it at the moment). Some of our =
OSS is based on a heavily in-house modified version of Freeside as well as =
our own app/layer that sits on top. This is essentially our base system whi=
ch allows us to push data and prov services to other internal and external =
systems - e.g. DNS, IP assignment, vendor's portals/APIs, RADIUS, etc. (bas=
ically almost every piece of hardware and software we have) and which inter=
faces with our self-service (customer portal - aka the almighty call-avoida=
nce "solution"). We also use IPPlan for managing IP assignment, but are mov=
ing away from it.
In a perfect world, everything would be integrated with everything else, se=
arching by every data element would be possible, every business process wou=
ld be automated, all of your docs would be in one place, all linked to the =
network element / customer / ticket / order / whatever, and so on. For most=
organizations, this is neither feasible nor required. Each system tends to=
do one or two things well and you have much unavoidable data duplication a=
nd data moving back and forth. Usually the goal is to minimize the amount o=
f manual data entry down to a single time and to push this aspect out towar=
ds the customer as much as possible. The extent of that will depend on your=
specific environment - everyone basically does the same thing, so often th=
ere's no need to re-invent the wheel (i.e. "let's develop everything from s=
cratch in-house" is a very bad move - you're not in the OSS business).
OSS/BSS is a huge and complex topic, so I'm only touching the tip of the ic=
eberg here and speaking mostly in general terms. It's definitely something =
that will be of greater and greater importance as your network grows, so ea=
rly planning is key, but don't get carried away trying to automate the hell=
out of everything because you'll lose focus on what you need to do in the =
short-term.
There is often a naive pursuit of perfection in OSS/BSS by those who haven'=
t been doing it for long enough - don't fall into that trap.
I'd start by defining your requirements/scope more solidly and then conside=
ring whether it makes sense to try to automate or enhance a particular proc=
ess. It may help to break things down step-by-step, perhaps based on depend=
encies or some other logical order, then think about how you would eliminat=
e what you perceive to be manual/error-prone/inefficient/slow/whatever. Fro=
m a costing perspective, you might find yourself in a (unfortunately freque=
ntly encountered by some) situation of "I just spent 50 hours writing a pro=
gram to automate a task that would have taken me 2 hours to do manually" or=
"we just spent $50k buying a product which we won't use to any reasonable =
level of capacity for the next five years".
--
Erik
*** Remove the _list part in my e-mail address to reply. ***
