[113] in Tooltime
Scopus Testing Plan - Second Draft
daemon@ATHENA.MIT.EDU (Steven Wade Neiterman)
Wed May 22 11:10:43 1996
Date: Wed, 22 May 1996 11:11:25 -0400
To: tooltime@MIT.EDU, tarnett@MIT.EDU
From: wade@MIT.EDU (Steven Wade Neiterman)
Cc: iggy@MIT.EDU, jsaylor@MIT.EDU
Testing Plan
May 21, 1996
Second Draft
Document Contents:
- Testing principles (guidelines):
- Philosophy
- Process
- Method
- Risks and Issues
Testing principles (guidelines):
- Repeatable testing procedure
- Results verified
- Procedure documented
- Every build should use regression testing
Philosophy:
- Testing lifecycle methodology should involve unit testing
(testing modules as created), integration tests (e.g.
movement of data transactions, deployment, access
control), systems testing (working as a complete
application) and acceptance testing (as compared to the
functional specification).
Process:
- Release 1.0b1 available on June 10
- Testing to be completed by tooltime members
- Duration of testing - two or three days
- Meeting on Thursday June 13 9:00 am to review feedback
- Release version 1.0b2 on June 17
- Help Desk "testers" to begin testing
- Duration of test three days
- Meeting on Wednesday June 19 3:00 to review feedback
- Team to assess and prioritize changes
- Release final beta 1.0b3 during week of June 24
- Meeting on June 27 9:00 am to assess state of rollout
Testing Method:
- Release 1.0b1 testing:
- Testing goal:
- Determine if (with modifications as needed for
1.0b2) it will be ready for Help Desk testers.
- How to test:
- Test version 1.0b1 for vital signs, evaluate core
functions of data entry, lookup, update and save a
record.
- Who will test:
- Release 1.0b1 testing is constrained to members of
the tooltime team.
- Possible invite a student from the Help Desk
- Testing duration:
- Two or three days from time application is released
and a meeting is conducted to assess feedback.
- What to evaluate:
- Identify "show-stopper" problems.
This is identified as an application that is:
- missing core functions (entry, lookup, save,
update, print) and there is no work around.
- has repeated crashes
- Cause high level of user frustration (e.g. poor
performance)
- Corrupts data, poor network behavior, does not
work with other applications necessary to conduct
business (e.g. email, Netscape).
- Summary: A "show-stopper" would be where a
consultant could not conduct business.
- How to report feedback:
- A discuss meeting will be created for the purpose of
archiving bugs, comments and feedback.
- Mailing list tooltime-bugs will feed into the
discuss meeting.
- Tester should report information in a common format
containing the following information: Date and time
of problem encountered, user who conducted the test,
function and/or screen being used, what was entered
(clicked or attempted), what happened, what should
have happened, what type of machine was used.
- Tester responsibility:
- Maintain a log of what is being tested.
- Follow checklist of items to test and use the
numbering scheme of the checklist to report
problems.
- ACTION ITEM: Checklist to be produced by
Peter/Lynne and submitted no later than June 7, '96.
- Testing data:
- If possible, there should be old logs in the
database to retrieve and evaluate.
- ACTION ITEM: Port a subset of existing Filemaker
data to the Scopus database.
- Release 1.0b2 testing:
- Testing goal:
- Identify "rollout" testing readiness, broaden the
testing pool and exposure of the product.
- How to test:
- In addition to retesting vital signs tested in
release 1.0b1, stress testing will be conducted at
this point, it will be facilitated by a storybook.
- Some exception tests should be conducted, for
example, evaluate error handling, loss of network
connectivity, asses response time. In addition,
exception testing includes testing field sizes for
data entry.
- An understanding of training requirements and needs
for user documentation ) should be evaluated (e.g.
is storybook enough for the consultants.
- Testing should be done on at least Windows and
Macintosh, Unix would be nice.
- Who will test:
- Tooltime members will test, however, the bulk of the
testing will be conducted by Help Desk staff,
including students.
- Four Help Desk staff and 10-20 students will be
invited to test.
- Testing duration:
- Two or three days from time application is released
and a meeting is conducted to assess feedback.
- Monday June 17 am, testing by 4-6 students and 4
staff.
- Tuesday June 18, after 5:00 pm, large student test
will be conducted to simulate stress testing.
- Logistics: New testers will need accounts and
access to machines and software. Possible use of
11-226, Training Lab and 16-720.
- What to evaluate:
- In addition to "playing", there will be a storybook
to follow.
- ACTION ITEM: Peter to write first draft of
storybook and submit to the team prior to end of day
on May 24.
- New testers need a half hour orientation to set
expectations.
- Expectation setting includes:
- Don't try to map Filemaker into Scopus, evaluate
based on conducting business.
- When evaluating an issue, ask "Can we conduct
business with this release?", "Will the level
of service decline if we do not resolve the issue
prior to release?", "Does the issue extend the
acceptable level of risk to conduct business?".
- It should be understood that we are not attempting
to create the "perfect" system in release 1.0.
- How to report feedback:
- After the Tuesday stress testing, there will be a
half hour facilitated review where comments will
be collected for distribution to the tooltime team.
- As in release 1.0b1, a template described above will
be filled out (did I hear a Web form?) and sent to
the tooltime-bugs discuss meeting.
- Wish list items, comments and nice to have features
should be captured and reported separately from
major issues.
- Tester responsibility:
- Be constructive
- Testing data:
- If possible, there should be old logs in the
database to retrieve and evaluate. This is an
important element of stress testing.
- Release 1.0b3 testing:
- Testing Goal: Confirm that changes requested in Release
1.0b2 testing were completed. Assess final release.
- Many elements in release 1.0b2 apply to the 1.0b3
release.
- Duration is short, possibly two days
- Determine go/nogo decision to deploy
Risks and Issues:
- Will application be ready to test on dates scheduled?
- Testing for enterprise rollout and acceptance is not part
of the 1.0 release. Further testing (e.g. minimum machine
requirements, low disk space and memory configurations,
access over Tether, etc.) will be conducted
after the initial rollout to the machines identified for
use by the Help Desk consultants.
- Should there be a "buddy" system to help with the testing,
where one person tests and another takes notes. Do we
need usability testing?
- Future enterprise and Help Desk testing will include
integration needs (e.g. rollout and release management,
system support, tests of automated warehouse feeds, etc.).
- The tooltime team will attempt to prioritize issues
required to be worked on for release 1.0b2 and 1.0b3.
Team members should communicate and solicit feedback to
Help Desk staff as appropriate.
