[687] in I/T Delivery
5ESS Upgrade and Replacement
daemon@ATHENA.MIT.EDU (Lu Keohane)
Fri Jan 18 10:24:16 2002
Message-Id: <5.0.2.1.2.20020118101042.00a10ce0@hesiod>
Date: Fri, 18 Jan 2002 10:22:07 -0500
To: delivery@mit.edu
From: Lu Keohane <lu@MIT.EDU>
Cc: njj@mit.edu, telecom-ops@mit.edu
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="=====================_5694331==_.ALT"
--=====================_5694331==_.ALT
Content-Type: text/plain; charset="iso-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable
Project Name: 5ESS replacement
Project Leader: Lu Keohane
Report Date January 17, 2002
Submitted by: Lu Keohane
URL: http://web.mit.edu/is/delivery/5ess
This Month's Accomplishments:
% Cutover date has been moved by MIT/Lucent after a preliminary test=
=20
(sampling of analog lines were tested) which proved to be only 40%=20
accurate. It was a joint decision that the accuracy must be higher and=20
must include all analog and all ISDN.
% Designed two strategies to accommodate telephone requests=20
until the new cutover. Lucent is using MIT=92s plan and=20
has ordered the necessary equipment and will begin the installation in=20
the classic at no cost to MIT.
% Finished the Frame verification in NW12 and 24, E19 is 80%=
=20
complete (dry circuits to be verified a second time)
% Re-verified specific counts for accuracy issues found by 5EOps team
% Jumpers completed in NW12 and 24 and 99% in E19 completed
% Continue to update cable and pairs found within frame verification=
=20
and day-to-day repair changes. (Approximately 100 cable/pair changes per=
=20
day).
% Continue to trace anomalies and update spreadsheets and databases
% Continue to oversee all telephone moves, adds, changes and trouble=
=20
tickets. Mapping and updating the 5E database with cable and pair changes=
=20
in chronological order (for a more accurate database at cut over).
% Continue the ICE9 database upgrade/reconciliation/updates
% Programming of new 5ESS with line translations and configurations=20
is 100% complete
% Trunk conversions are 100% complete. (Work is done by MIT after=20
hours to reduce service effecting issues)
% ISDN conversion from the E19 switchroom is in process. This work=20
is service affecting therefore 3-down is updated daily and the work=
=20
takes place after hours.
% Code that drives the business of Telecom (Service Orders and=20
billing) has been 98% complete. Fine tuning and testing are being worked=20
on at this time. Procedures are being written to test code on SO (service=20
orders) & TT (trouble tickets). The other modules that have been modified=
=20
and are being tested is: billing, cable infrastructure, vmail, jacks,=20
circuits, reports, etc. We are currently running to versions of ICE 9 on=20
two different platforms until cutover.
% Disaster Recovery spares are on site.
% Clean-up work of the modem pool started with assistance from R.=20
Hoffmann
% Testing of analog is started and anomalies found in 60% of=
=20
testing. Problems are being addressed and as a result the cutover date=20
has been moved for the best interest of the community.
% Plans for U-A to U-B conversion started for NW12 and 24 and will=20
work with the CSR=92s to gain access to locations on weekend of cutover.
Our 5ESS Database is 80% built and being verified (e.g. NNX=92s,=20
OE=92s, TN=92s, Trunks, Routing, etc.).
% Talking with Lucent everyday via conference call. Meeting=
=20
with Lucent Management Team on Wednesday afternoons and meeting with=20
supervisors of work crews and 5ESS engineers on Thursday morning
% Providing Lucent with information that is needed for the=20
new 5ESS (e.g. software releases, bmw=92s, etc.)
Next Month=92s plans:
Amis terminals have been delivered and are being set up
MIT will also install two AMIS terminals at the remote sites
Re-verify all cross connects on SO and repairs that have taken place since=
=20
August to test for accuracy on jumpers.
Continue preparation for adjunct systems (e.g. voicemail ACD, IVR, modem=20
pool, etc)
Complete the 3A translator installation/testing.
Complete the moving of analog trunks over to new switch and report=20
procedure for the night of cut over (On hold until cutover date is closer)
Continue with test plans (begin testing the existing 5ESS for features,=20
capabilities modifying documentation for accuracy. Test plan to be used on=
=20
the new 5ESS switch with additional features and higher release of software.
Set up of telephones in 3 office sites for pre and post cut over
Test the SAS announcements on new switch
Record all of the 5ESS announcements on the new switch
Test the ACD recent change/verify and MIS links
Record all the ACD announcements on the new switch
Record all the announcements for Facilities on the new switch
Test the VMail, 3A translator and 5ESS integration from the new switch=20
(off-hours) (integration, message waiting indicator functioning and outcall=
=20
abilities)
Continue tracing all special software configurations, links and other misc.=
=20
equipment running off of the 5ESS
Create procedure
Move all trunks over to new DSX panel (off hours required) (night of=
cutover)
Load all Switch Modules with current database, compare databases until=
cutover
Perform dumps of new 5ESS OE=92s for new ICE 9 database
Install TTY and AMIS terminals for cut over
Get volunteers from TNS to assist on cut over weekend when cutover date is=
=20
rescheduled
Set up testing center (test class of services, features, mlhg, msg waiting=
=20
Indicator, etc.)
Communicate new cutover date to community when ready.
Communicate moratorium to community when cutover date is announced.
Create a cut over plan and strategy for the different teams working.
Work with CSR=92s, Operators and the telephone help desk to address any=20
anomalies.
Issues:
One challenge this month was to acknowledging that Lucent was not doing as=
=20
well as we had hoped. Therefore, the project was postponed (cutover was=20
scheduled for January 18, 2002). The moving of the date was in the best=20
interest of the MIT community. We could have performed the cutover on=20
January 18, 2001 as planned, but, we would have had more than 50% of our=20
lines out-of-service, this of course was not acceptable. Lucent began=20
testing of analog in December and had a failure rate of 60% (the cause was=
=20
not clear they weren't sure if the failures were real or if the program=20
running the test was failing). Wanting to have little to no impact in the=
=20
community, the team thought it best to move the date out and perform more=20
complex testing. We also, worked with Lucent to draw up contingency plans=
=20
to provide ISDN growth capabilities within the 24 and NW12=20
switchrooms. These will provide 5ESS services to meet customer demand. We=
=20
will wait until Lucent has completed their testing of the ISDN and Analog=20
lines and will announce the new scheduled cutover. Since the first=20
meeting with management, Lucent has made a lot of progress. We do have=20
special applications that require special attention. These items will take=
=20
us longer to ensure their accuracy.
As we continued with our testing plan, we had noticed issues with SM1 that=
=20
no one had seen before. After many discussions, it was determined that the=
=20
new 5ESS while running its tests were performing inaccurate audits with the=
=20
lines that were connected to the modem pool. We now understand the problem=
=20
and are working towards a resolution within the end of the week.
We found a configuration issue with the ACD equipment. We have ordered=20
parts and software and will need to make the change out at a time that is=20
good for the Help Desk and 5E ops. This was another example of MIT telling=
=20
Lucent that they forgot to assess equipment on the 5E or as an adjunct.
The 5ESS Service Team made up of 6 staff members and one voucher employee=20
are displaying signs of stress and burnout. All individuals are working=20
110 % (full time) on the project as well as performing their day-to-day=20
responsibilities at 110%. The staff is being spread thin. The 5ESS=20
Service team tends to handle issues that have no home. Therefore, the team=
=20
will spend a significant amount of time on outside problem resolutions and=
=20
become more frustrated and stressed for the distraction. (I continue to=20
address this issue gently, cleverly and creatively). Thank you to those=20
who recognize the magnitude and complexity of this project and have offered=
=20
their assistance and encouragement.
The community was notified that the cutover plans were going to=20
take place. Letters were written to describe the =93quiet period=94=
=20
necessary before a cut and had to be re-written to notify the community of=
=20
our decision to postpone the cutover and the quiet period. When we are=
=20
satisfied with the testing results, we will inform the community again of=20
our plans and requirements of the quiet period.
Team Dynamics:
We experienced individuals disappointment in the change in cutover at the=20
same time we experienced individuals feeling relieved.
The 5ESS core team and additional resources being utilized come from a=20
strong foundation (or that the super glue is working). Increased=20
workloads, longer hours and outside frustrations are increasing stress=20
levels for all in E19 (6th & 7th floors). The TNS team is made up of very=
=20
talented individuals. Their efforts, skills, and determination are making=
=20
this project appear to the outside world as simple though in reality it=92s=
=20
very complex.
The team is doing an outstanding job with balancing the stress from the=20
day-to-day issues and the project issues (it is a good sign when the team=20
is still laughing at my jokes).
Key Leanings:
Humor doesn't always work
"Is this problem a problem, or am I looking
at it the wrong way? Is it something better?"
Thanks,
Lu
=BA=BF=BA
~=20
--=====================_5694331==_.ALT
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
Project Name: 5ESS replacement<br>
Project Leader: Lu Keohane<br>
Report Date January 17, 2002<br>
Submitted by: Lu Keohane<br>
URL:
<a href=3D"http://web.mit.edu/is/delivery/5ess"=
eudora=3D"autourl">http://web.mit.edu/is/delivery/5ess</a><br>
<br>
<b><u>This Month's Accomplishments:<br>
<br>
</u></b><font face=3D"ZapfDingbats">%<x-tab> &n=
bsp; </x-tab></font>Cutover
date has been moved by MIT/Lucent after a preliminary test (sampling of
analog lines were tested) which proved to
<x-tab> </x-tab>be only 40%
accurate. It was a joint decision that the accuracy must be higher
and must include all analog and all ISDN.<br>
<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font><x-tab> </x-=
tab>Designed
two strategies to accommodate telephone requests until the new
cutover. Lucent is using MIT=92s
<x-tab> </x-tab>plan and
has <x-tab> </x-tab>ordered the necessary equipment and
will begin the installation in the classic at no cost to MIT. =20
<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font><x-tab> </x-=
tab>Finished
the Frame verification in NW12 and 24, E19 is 80% complete (dry circuits
to be verified a second time)<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Re-verified
specific counts for accuracy issues found by 5EOps team<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Jumpers
completed in NW12 and 24 and 99% in E19 completed<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Continue
to update cable and pairs found within frame verification and day-to-day
repair changes. (Approximately 100
<x-tab> </x-tab>cable/pair changes per day).<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Continue
to trace anomalies and update spreadsheets and databases<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Continue
to oversee all telephone moves, adds, changes and trouble tickets.
Mapping and updating the 5E database with <x-tab> </x-tab>cable and
pair changes in chronological order (for a more accurate database at cut
over).<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Continue
the ICE9 database upgrade/reconciliation/updates <br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Programming
of new 5ESS with line translations and configurations is 100%
complete<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Trunk
conversions are 100% complete. (Work is done by MIT after hours to
reduce service effecting issues)<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>ISDN
conversion from the E19 switchroom is in process. This work is
service affecting therefore 3-down is updated daily
<x-tab> </x-tab>and the work
takes place after hours.
<dl><font face=3D"ZapfDingbats">
<dd>%<x-tab> </x-tab></font>Code
that drives the business of Telecom (Service Orders and billing) has been
98% complete. Fine tuning and testing are being worked on at this
time. Procedures are being written to test code on SO (service orders)
& TT (trouble tickets). The other modules that have been
modified and are being tested is: billing, cable infrastructure, vmail,
jacks, circuits, reports, etc. We are currently running to versions
of ICE 9 on two different platforms until cutover.
</dl><font face=3D"ZapfDingbats">%<x-tab>  =
; </x-tab></font>Disaster
Recovery spares are on site.<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Clean-up
work of the modem pool started with assistance from R. Hoffmann<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font><x-tab> </x-=
tab>Testing
of analog is started and anomalies found in 60% of testing.
Problems are being addressed and as a result the
<x-tab> </x-tab>cutover date has been moved for the best
interest of the community. <br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font>Plans
for U-A to U-B conversion started for NW12 and 24 and will work with the
CSR=92s to gain access to locations on
<x-tab> </x-tab>weekend of cutover.<br>
<x-tab> </x-tab>Our 5ESS
Database is 80% built and being verified (e.g. NNX=92s, OE=92s, TN=92s, Trun=
ks,
Routing, etc.).<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font><x-tab> </x-=
tab>Talking
with Lucent everyday via conference call. Meeting with Lucent
Management Team on Wednesday afternoons and
<x-tab> </x-tab>meeting with supervisors of
work crews and 5ESS engineers on Thursday morning<br>
<font face=3D"ZapfDingbats">%<x-tab> &nbs=
p;</x-tab></font><x-tab> </x-=
tab>Providing
Lucent with information that is needed for the new 5ESS (e.g. software
releases, bmw=92s, etc.)<br>
<br>
<x-tab> </x-tab><br>
<br>
<b><u>Next Month=92s plans:<br>
</u></b>Amis terminals have been delivered and are being set up<br>
MIT will also install two AMIS terminals at the remote sites<br>
Re-verify all cross connects on SO and repairs that have taken place
since August to test for accuracy on jumpers.
<dl>
<dd>Continue preparation for adjunct systems (e.g. voicemail ACD, IVR,
modem pool, etc)=20
<dd>Complete the 3A translator installation/testing.
<dd>Complete the moving of analog trunks over to new switch and report
procedure for the night of cut over (On hold until cutover date is
closer)=20
<dd>Continue with test plans (begin testing the existing 5ESS for
features, capabilities modifying documentation for accuracy. Test
plan to be used on the new 5ESS switch with additional features and
higher release of software.=20
<dd>Set up of telephones in 3 office sites for pre and post cut over=20
<dd>Test the SAS announcements on new switch=20
<dd>Record all of the 5ESS announcements on the new switch=20
<dd>Test the ACD recent change/verify and MIS links=20
<dd>Record all the ACD announcements on the new switch=20
<dd>Record all the announcements for Facilities on the new switch=20
<dd>Test the VMail, 3A translator and 5ESS integration from the new
switch (off-hours) (integration, message waiting indicator functioning
and outcall abilities)=20
<dd>Continue tracing all special software configurations, links and other
misc. equipment running off of the 5ESS=20
<dd>Create procedure=20
<dd>Move all trunks over to new DSX panel (off hours required) (night of
cutover)=20
<dd>Load all Switch Modules with current database, compare databases
until cutover
<dd>Perform dumps of new 5ESS OE=92s for new ICE 9 database=20
<dd>Install TTY and AMIS terminals for cut over=20
<dd>Get volunteers from TNS to assist on cut over weekend when cutover
date is rescheduled
<dd>Set up testing center (test class of services, features, mlhg, msg
waiting Indicator, etc.)=20
<dd>Communicate new cutover date to community when ready.
<dd>Communicate moratorium to community when cutover date is announced.
<dd>Create a cut over plan and strategy for the different teams=20
working.
<dd>Work with CSR=92s, Operators and the telephone help desk to address any
anomalies. <br>
<br>
<br>
<br>
<br>
<br>
</dl><b><u>Issues:<br>
</u></b>
<dl>
<dd>One challenge this month was to acknowledging that Lucent was not
doing as well as we had hoped. Therefore, the project was postponed
(cutover was scheduled for January 18, 2002). The moving of the
date was in the best interest of the MIT community. We could have
performed the cutover on January 18, 2001 as planned, but, we would have
had more than 50% of our lines out-of-service, this of course was not
acceptable. Lucent began testing of analog in December and had a
failure rate of 60% (the cause was not clear they weren't sure if the
failures were real or if the program running the test was failing).
Wanting to have little to no impact in the community, the team thought it
best to move the date out and perform more complex testing. We
also, worked with Lucent to draw up contingency plans to provide ISDN
growth capabilities within the 24 and NW12 switchrooms. These will
provide 5ESS services to meet customer demand. We will wait until Lucent
has completed their testing of the ISDN and Analog lines and will
announce the new scheduled cutover. Since the first meeting
with management, Lucent has made a lot of progress. We do have
special applications that require special attention. These items
will take us longer to ensure their accuracy.<br>
<br>
<dd>As we continued with our testing plan, we had noticed issues with SM1
that no one had seen before. After many discussions, it was
determined that the new 5ESS while running its tests were performing
inaccurate audits with the lines that were connected to the modem
pool. We now understand the problem and are working towards a
resolution within the end of the week.<br>
<br>
<dd>We found a configuration issue with the ACD equipment. We have
ordered parts and software and will need to make the change out at a time
that is good for the Help Desk and 5E ops. This was another example
of MIT telling Lucent that they forgot to assess equipment on the 5E or
as an adjunct.<br>
<br>
<dd>The 5ESS Service Team made up of 6 staff members and one voucher
employee are displaying signs of stress and burnout. All
individuals are working 110 % (full time) on the project as well as
performing their day-to-day responsibilities at 110%. The staff is
being spread thin. The 5ESS Service team tends to handle issues
that have no home. Therefore, the team will spend a significant
amount of time on outside problem resolutions and become more frustrated
and stressed for the distraction. (I continue to address this
issue gently, cleverly and creatively). Thank you to those who
recognize the magnitude and complexity of this project and have offered
their assistance and encouragement. <br>
<br>
<br>
<br>
</dl><x-tab> </x-tab>The
community was notified that the cutover plans were going to take
place. Letters were written to describe the =93quiet
<x-tab> </x-tab>period=94
necessary before a cut and had to be re-written to notify the community
of our decision to postpone the cutover and
<x-tab> </x-tab>the quiet period. When we
are satisfied with the testing results, we will inform the community
again of our plans and <x-tab> </x-tab>requirements of the quiet
period.<br>
<br>
<font=
face=3D"ZapfDingbats"><x-tab> &nbs=
p;</x-tab></font><x-tab> </x-=
tab><br>
Team Dynamics: <br>
<br>
We experienced individuals disappointment in the change in cutover at the
same time we experienced individuals feeling relieved. <br>
<br>
The 5ESS core team and additional resources being utilized come from a
strong foundation (or that the super glue is working). Increased
workloads, longer hours and outside frustrations are increasing stress
levels for all in E19 (6<font size=3D1><sup>th</sup></font> &
7<font size=3D1><sup>th</sup></font> floors). The TNS team is
made up of very talented individuals. Their efforts, skills, and
determination are making this project appear to the outside world as
simple though in reality it=92s very complex. <br>
<br>
The team is doing an outstanding job with balancing the stress from the
day-to-day issues and the project issues (it is a good sign when the team
is still laughing at my jokes).<br>
<br>
Key Leanings:=20
<dl>
<dd>Humor doesn't always work
<dd>"Is this problem a problem, or am I looking
<dd> at it the wrong way? Is it something better?"<br>
<br>
<br>
<br>
<x-sigsep><p></x-sigsep>
</dl><font color=3D"#0000FF">Thanks,<br>
Lu<br>
<br>
=BA</font><font color=3D"#800080">=BF</font><font color=3D"#0000FF">=BA</fon=
t>
<br>
<font color=3D"#FF0000"> ~</font> </html>
--=====================_5694331==_.ALT--
