[309] in Tooltime
Scopus Acceptance Test Results
daemon@ATHENA.MIT.EDU (Steven Wade Neiterman)
Mon Nov 4 13:09:59 1996
Date: Mon, 4 Nov 1996 13:11:35 +0100
To: hogue@MIT.EDU, tooltime@MIT.EDU
From: Steven Wade Neiterman <wade@MIT.EDU>
Cc: burke@MIT.EDU, ANGER@mitvma.mit.edu, jaroy@MIT.EDU, mishakim@MIT.EDU,
terrax@MIT.EDU, legand@MIT.EDU
Bill,
As we discussed, release 1.0 of the Scopus Help Desk application is
entering a transition phase where we are ready to assess the current state
and determine next steps.
The tooltime team has completed seven beta releases of the application and
reached the acceptance phase of release 1.0. Enclosed is a report on
findings and comments from the acceptance test.
Please let us know if you have any comments or questions.
And to those who participated, many thanks for taking the taking the time
to help with this phase of the project.
----------------------------------------------------------------------------
SUMMARY:
On October 16, 1996, six Help Desk staff (three full time and three
students) participated in a two hour session to help gather information
needed to determine the status of the work to date. This session began a
phase of the project called acceptance testing. The purpose of this phase
is to demonstrate that the system meets or does not meet it's intended
project requirements in the operational environment as related to the
functional specifications and original project mandate.
A pre-requisite to participation involved becoming familiar with the
application through attending an "introduction/training" session and
spending time using the system prior to the actual acceptance test session.
Acceptance test participants followed a "storybook" document, which was
used to guide them over a series of examples. Comments from the
participants are listed below.
FINDINGS:
Acceptance testing is intended to resolve the question "Can the consultants
use Scopus as the system of record as intended by the Functional
Specification"?
It is clear that the application only presents a baseline of functionality.
Based on the purpose of acceptance testing, the following findings have
been determined:
1. The application is useable and no showstoppers were identified that
would cause the application to not perform it's intended function (with
exception of problems with the Macintosh client, which are under
investigation).
2. There is a need to continue making application improvements, including
ergonomics, automation, enhancements for efficiency, and resolve
a number of "annoying" features.
NEXT STEPS:
1. It's understood that improvements must be made in release 2.0. A task
list is currently being constructed based on feedback from the Tooltime
team and testers.
2. Post release 1.0 tasks must include support for generating reports,
continued stability refinement and quick response to issues such as
priority 1 bug resolution. These items need to be identified and
assigned resources.
3. Macintosh stability issues must be understood and determined who is
responsible for corrective actions, e.g. Scopus or MIT.
4. A first draft of rollout criteria (training, data migration, etc.)
has been assembled. Rollout principles and a further refinement of the
criteria must be addressed prior to rollout.
Since the acceptance testing, some minor items have been resolved and
others identified for team discussion. The next steps outlined above begin
a process to assess rollout and success factors for release 1.0. It also
sets the stage to begin a process to produce a scope statement and a work
breakdown for release 2.0.
----------------------------------------------------------------------
The following are comments from the six testers.
----------------------------------------------------------------------
X-Sender: burke@po8.mit.edu
Mime-Version: 1.0
Date: Wed, 16 Oct 1996 19:22:00 -0500
To: wade@MIT.EDU
From: burke@MIT.EDU (Jerry Burke)
Subject: acceptance for scopus database
Cc: sampson@MIT.EDU
for the helpdesk part:
- when opening a new help desk log the following fields automatically get
filled in:
catagory = helpdesk
queue = mac
method = voice
hardware = macintosh
these are required fields, but there are possibilities the incorrect data
will be left in them. In addition it is time consuming to go back to a
record to change, so you may not want to go back, or you forget, cuz
someone tapped you on the shoulder for help etc..
- how do you refer a to a more knowledgable consultant, only way i see in
the helpdesk log is to fill in the taken filed and hope a consultant checks
for their assignments.
why can't we make an assignment and SCOPUS sends email to the person that
got assigned, or group....
also housecalls, refer to housecalls, but how do you know the housecall
schedule, and how do they get notified when a new log gets
assigned/referred to them? the database should notify, and housecalls
should set the schedule?
- information from a previous record gets filled into housecall information
fileds. although i didn't see that happen from the mac in my office. maybe
a problem in bldg. 16 machines. just tried again and works ok.
- check to be sure this does not happen on other platforms, i am using the macs
with macos 7.5.3 v2, currently.
_ i can't generate reports for weekly status, monthly status, or quarterly.
I understand that Steve Neiterman can fix this and help me with it. for Rel. 2
- when checking my logs records form the script menu, i get a number of
other logs not assigned to me or takin by me....
- when checking my logs pending i get a number of logs i had no
responsibility for at all. however on some of them i do see my name
mentioned in the sumary fields. that a client was referred by me etc..
- the network fields --
look really good, found them to be intuitive and easy fields manipulate.
would like to see more than on entry for ip addresses, especially for the
type of network drops dcs works on and i am sure stephanie is affected by
it.
when made chnages to the miscellaneous fields, and do a save and close
it does not save the changes to the miscellanous fields.
i just tried that in my office and the changes did take, so figure it out.
i am on a mac7100/66 with 24mb ram, 500mb hddisk, virtual memory off, macos
7.5.3.
thanks steve - i don't think any of these items are show stoppers for us
(DCS) .
i understand that you will look at a few of these items for release 2.
they will report any bugs to tooltime-bugs..
thanks, jerry
----------------------------------------------------------------------
Date: Wed, 16 Oct 96 13:46:19 EDT
From: Art Anger <ANGER@mitvma.mit.edu>
Subject: Acceptance report
To: Steven Wade Neiterman <Wade@MIT.EDU>
OBSERVATIONS AND RECOMMENDATIONS ON WINDOWS SCOPUS 10/16/96
If the HelpDesk FileMaker database were to start giving serious difficulties
without hope of rapid repair, I would be willing to switch to Scopus as it
is currently implemented.
In the absence of such a compelling reason, I would not want to switch to
Scopus as I saw it today. Although most capabilities seemed to be available,
the frequent interruptions of "reasonable" steps by unreasonable warnings
and/or errors and the occasional need for other circuitous methods to
accomplish the intended action increase the training time needed, the typical
entry time, and the probability of need for correction. In all, efficiency
will suffer quite noticeably, even after familiarization with these quirks.
Problems:
--In a Housecall child window one time, Save complained about the day of the
month, failure of insertion, and non-update of key field.
--Some history entries had long strings of blanks inserted during import,
making them only marginally readable.
Major nuisance:
--After Save, CloseLog or ExitLog complains about changes since Save, but
a second Save complains that there have been no changes. (This complaint
comes up so frequently that users will often not recognize when a different
type of complaint has come up--until after they have cleared it!)
--After what I presumed to be a reasonable sequence of actions including Save,
I could not find one log by searching for several of its expected values.
It got created with only a few default values entered.
Minor nuisances:
--The opening position of the Log window did not display the full width of
the History field, even though there was room to do so.
--Alt-F4 did not warn that it would quit the whole application (rather than
just the current process).
--Having to go into Dataprobe to search on creator or modifier, when the
more intuitive FindLog looks like it should do quite well.
Suggested improvements:
--Move "pull-down" arrows immediately adjacent to related display boxes.
--Make Enter activate whichever button is highlighted in MIT_Temp_Results
dialogue box.
--Make Dataprobe Results window stay open after finishing with examination
of one of its entries.
Date: Fri, 18 Oct 96 15:05:04 EDT
From: Art Anger <ANGER@mitvma.mit.edu>
Subject: Additional observations
To: Steven Wade Neiterman <Wade@MIT.EDU>
OBSERVATIONS OF WINDOWS SCOPUS ON 10/18/96
Possibly due to a better functioning machine, and parly due to a slightly
better knowledge of the system, today's test proceeded more satisfactorily
than Wednesday's.
As I noted previously, nearly all necessary capabilities are available.
The two primary obstacles to efficient use that I now see are:
--Non-obvious effects of using Save, Exit, and various forms of window
closing in different sequences;
--Warning dialogues about automatic "changes" to otherwise untouched windows.
With suitable training literature, these awkward aspects need not have a
serious impact on daily use. I would not oppose the introduction of Scopus
after such materials are provided.
In the following release, I would hope to see some of these improvements:
--Updating the Touched field only on close after testing for other changes
--Grouping the Clear, Save, Close, Exit buttons together near the upper
right corner of each window, to promote their use in place of the system
closers
--Adding a padlock symbol (or a word) to each window which cannot be saved
--Not excluding locked records from any search, although they may be marked
somehow in the result list
--Providing a convenient means of stepping through a result set, displaying
the details of the successor without having to reselect the set window
and an itme within it
--Providing a script to show the most recent logs with latest first
Art Anger
----------------------------------------------------------------------
X-Sender: jaroy@po10.mit.edu
Date: Wed, 16 Oct 1996 23:52:37 -0400
To: wade@MIT.EDU
From: "Jeremy A. Roy" <jaroy@MIT.EDU>
Subject: Scopus Acceptance Notes
Mime-Version: 1.0
Hi Steve,
Today at approximately 5:00 p.m. I completed the storyboard for student
acceptance testing, and wish to offer my report.
The product which I worked with today, I can say in all honesty, is at long
last a step forward from our current situation. I feel that the reasons
that we chose an upgrade to the Filemaker database can finally be seen and
exploited, uninhibited by the major bugs that hindered progress thus far.
Overall, I think it fair to say that we would not lose productivity with a
release of Scopus, assuming that the two issues which I bring up in my
latest post to tooltime-bugs are addressed: the printing of child
information and the unexpected data loss in child windows. There are, of
course, many minor improvements that could be made to improve usability and
functionality, but which I do not feel are critical to an initial release.
Additionally, I feel that there are features and design elements that need
to undergo the test of "on the job" experience, for which a first release
is needed.
I was able to navigate through the storyboard scenarios at a reasonable
pace, with general ease. I was, of course, paying particularly close
attention to detail, perhaps more than I do on a normal working day,
because of the testing environment. During the course of the acceptance
test, I completed the tasks at a rate which I feel is comparable to the
speed at which I am able to complete the same tasks under the current
implementation of the Helpdesk database (FileMaker). Save a small number
of glitches and bugs, which I am documenting for tooltime-bugs, I
encountered little frustration in the process, and at no times found myself
re-arranging my work because of the software.
These are the reasons I feel that we will not lose productivity. The
product is clearly powerful, with large-scale performance increases over
the current solution, and is, in my opinion, reasonably usable. "Minor"
frustrations included items such as having to use the mouse so much,
picklists being obscured by scroll bars, an awkward searching mechanism,
and having to scroll down, minimizing and maximizing windows to see some
parts of the screen-- all issues which are not prohibitive of an initial
release, but which are areas of productivity which could be improved.
Training, I feel, will need to be more comprehensive for most consultants
than I have seen thus far. Some of the more particular aspects of using
Oracle and Scopus need to be more clearly explained and demonstrated to new
users: searching mechanisms, the "save" button, etc. The training that
most of the students (and perhaps most of the full-time staff) have
received so far IS NOT sufficient for a roll out, and should be re-touched
before a release is considered.
I wish once more to affirm my position after today's testing: With (and
only with) a resolution of the printing (adding child data) and the data
loss problems (entering in child window, clicking "close" as we discussed
today), I feel that the Scopus project will be ready for implementation in
the Computing Helpdesk.
If you have any further questions or require additional feed-back, I urge
you to contact me by e-mail.
Thank you,
Jeremy A. Roy
Consultant, MIT Computing Helpdesk
11-221
77 Massachusetts Avenue
Cambridge, MA 02139
(617)253-0001
----------------------------------------------------------------------
X-Sender: mishakim@po9.mit.edu
Mime-Version: 1.0
Date: Thu, 17 Oct 1996 00:33:31 -0400
To: wade@MIT.EDU
From: Misha K Hill <mishakim@MIT.EDU>
Subject: scopus eval
Cc: mishakim@MIT.EDU
As it stands now, most of the problems I have with Scopus are cosmetic,
bugs, and principle. Some of the bugs are critical, but they are
nonetheless bugs and not functional problems.
I find the system unreliable and I don't know whether that is due to bugs in
the database or problems in the application. If the stability can not be
resolved, then I think that we cannot use this product
Assuming that the stability problems are related to bugs that can be fixed,
then I think that Scopus is almost ready to use. My specific complaints
regarding cosmetics and functionality are included in my mail to
tooltime-bugs.
I still think that there are many features of filemaker that make it a
superior product to Scopus. Although it is not as efficient, it is much
easier to use. Multiple layouts, variable font sizes and styles, and a
scripting language that anyone can use with 15 minutes training are all
things that any consumer product would have before even going into alpha
testing in today's market. A total lack of modern usability features is not
compensated for by faster searches and better statistic generation.
I strongly recommend that we continue to use FileMaker indefinitely.
Version 3.0 provides most of the "new" functionality of scopus, and since
it's a real commercial product, Claris will continue to offer upgrades and
stability with out us paying them hundreds of thousands of dollars. We
should continue to look at solutions like scopus, in case someday they are
able to do what we want, but the MIT Computing Helpdesk is best served by
and will best serve its clients with FileMaker.
Misha Hill
Student Consultant
----------------------------------------------------------------------
Date: Fri, 18 Oct 96 10:58:40 EDT
X-Sender: micro-help@mit.edu (Unverified)
Mime-Version: 1.0
To: wade@MIT.EDU
From: Microcomputer Helpline <micro-help@MIT.EDU>
Subject: Acceptance testing
Cc: iggy@MIT.EDU, terrax@MIT.EDU
The Scopus GUI to the Oracle backend is an adequate means by which
to create, query, find, and append to logs. It is able to handle queries
and searches through the history and summary fields like the current
filemaker system. For functionality's sake it is what is needed. The
system has the ability to handle a large volume of logs and keep them for
several years. Load time is a bit longer than the current filemaker
database, but unlike the current system it requires every user to go
through a "clean" exit and load of the system, which is both good and bad:
good since every log will be marked to a consultant, bad since the process
can waste time in this reload.
The GUI lacks however in user friendliness that file maker is able
to create with its changeable screen formats, multiple fonts, and easy to
read log information, which is attributed to the large fonts and use of
bold ro make pertinent info stand out. Despite the attempts to say that
this database needs time to develop, I say that the database should wait
for these changes to occur before it is released. I am not saying that the
helpdesk could not get by with Scopus, but I am saying that as a consultant
I would rather go to a system that is BETTER than what I am using than to
go to one that is as adequate in some areas and deficient in others.
Sincerely,
Terrance Harmon
-------------------------
Consultant
MIT Microcomputing Help Line
Phone: x3-0001
E-Mail: micro-help@mit.edu
----------------------------------------------------------------------
Date: Fri, 18 Oct 96 14:28:17 EDT
X-Sender: legand@po9.mit.edu
Mime-Version: 1.0
To: Wade@MIT.EDU, Sousa@MIT.EDU
From: legand@MIT.EDU (Larry Egan)
Subject: Acceptance Testing
Per Lynne Sousa I have tested Scopus on a PC platform. I consider the
Scopus/PC platform to be acceptable for daily work. Report generation is
something that is very desirable and should be implemented as soon as
possible.
My acceptance test with the MAC platform in Bldg. 16 did not go well.
Performance problems , bugs which I cannot recreate have left me with mixed
feelings. I would like to defer my comments on the MAC platform until
Monday. I will conduct the tests in Bldg 11 and forward my results at that
times.
Thanks
Larry
