[32680] in North American Network Operators' Group
Re: Operations: where are you going to sit?
daemon@ATHENA.MIT.EDU (John Todd)
Wed Dec 6 15:35:37 2000
Mime-Version: 1.0
Message-Id: <v04210100b654370fad12@[172.16.2.77]>
In-Reply-To: <20001206003259.6086.cpmta@c004.sfo.cp.net>
Date: Wed, 6 Dec 2000 14:53:08 -0500
To: Sean Donelan <sean@donelan.com>
From: John Todd <jtodd@loligo.com>
Cc: nanog@merit.edu
Content-Type: text/plain; charset="us-ascii" ; format="flowed"
Errors-To: owner-nanog-outgoing@merit.edu
Sean's starting post to this thread really wasn't a question, nor
is the content to follow consistent with the "NA" part of the acronym
(though it well suits the "NOG" part) but considering the general s/n
ratio here I figure my small bit of static with at least some useful
data won't cause anyone's mail buffers to detonate. The general NOC
discussion got me to thinking about some of the "details" of NOC
design that I've kicked around before. I'll throw a few points in
here that are often overlooked due to "practical" (read: no planning,
or no budget) reasons.
I'll preface this with the comment that we're talking about a NOC,
and not simply a call center. Call centers are where huge numbers of
people are herded into a queue to have their questions and minor
problems addressed by a mass of often underpaid, under-trained
"flaks." A NOC is significantly different, where one expects a much
higher level of expertise of staff, a much higher average
professionalism quotient (more on this in a minute), and where the
criteria for success are not things like "hold time" and "abandon
rate" but are "ETR reduction" and "network uptime."
Regardless of the particular tasks (repair? observation?
construction?) that a NOC staff must perform, there is an element of
inter-personal communication that must occur. Previous notes on this
thread have talked about how wireless phones (with headsets, I would
assume) are a good thing, since often people move from cube to cube
to get questions answered. To solve problems, very often there are
several people working on the same issue from different directions,
so quick and effective communications between staff members is
required. I am convinced that electronic communications are not the
ideal method for this quick communication.
Most cubes/office spaces are not built for effective personal
communications. They're built for personal privacy and adequate
workspace. NOC spaces should not (IMHO) generally _not_ follow these
same reasons for design, despite what NOC staff usually wish. The
concept here is to have easy verbal or visual communications between
members of the staff, and to have the manager able to view and
understand what everyone is doing. This is often directly
contradictory to what NOC staff want, which often resembles a maze of
caves from which grunts and empty candy wrappers are thrown.
The Suggestion:
Think of NASA's Houston Mission Control Center (MCC). The room is
primarily a large, open space with high ceilings. By standing up, any
one member of the staff can see all other members of the staff. The
people in the back of the room are on slightly raised platforms (1
foot?) so that they can see more of what the "front" row people are
doing. (Anyone been to the MCC lately? Is this still the case? I
know they re-modeled a few years ago.) The people in the rear
(again, I assume this to be the case) are higher in the chain of
authority, and can survey what is going on at all stations to get a
clear idea of what is happening, and anyone in the room can engage in
conversation easily with the manager without having to pick up a
phone. With this in mind, the expectation of having a professional
staff arises. Talking loudly on the phone with one's significant
other is probably going to be noticed. Shouting
The NASA control room (scaled down, of course) is an idealized
version of what I think works best in situations that are fast-moving
and complex. For 99.9% of the time, the NOC will be simply
comfortable, but in that moment where you've lost all four
cross-country strands and for some reason the out-of-band modems
aren't picking up and your IGP is melting down... you'll appreciate
the ability to have everyone in the same room able to get work done
quickly. It might even be worth the few million you spend on the
room. ;)
I've often thought about a "pit" style design, where the managers
sit in a depression in the middle of the room, and on all sides there
are raised workstations facing towards the middle. The "big board"
screens would have to be on hanging tubes directly above the manager
area. I like that method since nobody then has to turn to talk to
anyone else, but I dislike it because of the complexity of the
design. Anyone have any examples of a NOC in this style?
More minor comments on environments (network, telephone, and machine
choices aside):
- Sound reduction panels on ceiling/walls
- Personally, I prefer having tile/raised floor in the NOC, since
it allows one to slide a wheeled chair easily between multiple
stations, which often is required when working on larger
problems or when doing training. This is a matter of
preference, though, as there are some drawbacks (noise)
- Incandescent lighting instead of florescent and spot lighting
instead of "wash" lighting (easier on eyes; less glare)
- All members of team should be visible to the manager
- Airflow should not point down directly on staff - try sitting at
a workstation with a 1mph, 10-degree F cooler breeze blowing
on your head for 8-10 hours
- Multiple phones per workstation
- Headsets are mandatory, for ease of use as well as reduction in
worker's comp claims
- All "big board" items should be visible without staff having to
turn their heads more than 30 degrees from their normal focus
point
- Workstations should be labelled with numbers so easy reference
can be made to locations in the room
- Work areas should have room for a keyboard and a 8.5"x11" binder
to be completely on the surface (i.e.: about 20" deep) so
that someone can work from the operations manual and type
at the same time
- While it looks impressive, I've found that often people don't
use or look at monitors that are above eye level. It seems
from anecdotal evidence that people are more willing to use
multiple horizontal monitors than glance upwards. Besides,
stacking monitors starts to build "walls" around workstations.
Some other NOCs or NOC-like settings that I've liked:
- Akamai's facility in Boston. Nice looking, lots of $$$. Haven't
checked it out for functionality. Mistah Gilmore, do you have
pictures of this? Couldn't find 'em on your website.
- Many major TV station newsrooms or production rooms. Of course,
it's a different medium, but the same concepts: short deadlines, lots
of activity, a lot of cooperative effort.
PS: To answer someone's question about cordless phones, I've never
seen an occasion where a cordless phone has interfered with computer
equipment. Having a data center with no cordless phones is certainly
painful for your ops staff and your customers. Now, cell phones I
would have to experiment with since they are normally more powerful...
JT
>Several folks have asked me how to build a NOC for their operations
>people. Instead I thought I would post a few pointers to some of
>the larger oeprations centers with public information.
>
>AT&T Global Network Operations Center, Bedminster, NC (where are the people?)
>http://www.archrecord.com/CONTEDUC/ARTICLES/11_00_1.asp
>
>Cheyenne Mountain Operations Center (NORAD), Colorado Springs, CO (designed
>by the navy, built by the army, used by the air force)
>https://www.cheyennemountain.af.mil/cmoc/cmocimages.html
>
>Genuity Network Operations Center, Burlington, MA (cool video John :-)
>http://www.genuity.com/help/noc/tour.htm
>
>UUNET Network Operations Center, Northern Virginia (aren't there any cities
>in Virginia?)
>http://www.us.uu.net/support/noc/
>
>Verio Network Operations Center, Dallas, TX
>http://home.verio.com/company/technology/noc.cfm
>
>Key features for a NOC:
> 1) Good chairs
> 2) Quiet
> 3) Adequate ice maker
> 4) Lots of bookshelves, file cabinets, and personel storage
> 5) Lots of phone lines (including conference and analog)
> 6) Some direct phone(s) (not through PBX, i.e. Red Phones)
> 7) Multiple PCs/Workstations per operator
> 8) Private tunes (cd player & headset)
> 9) CNN and The Weather Channel (really ESPN)
> 10) Drapes across the glass window
