[208] in Athena User Interface
Usability testing
daemon@ATHENA.MIT.EDU (Christopher D. Beland)
Wed Jun 21 11:22:21 2000
Message-Id: <200006211522.LAA25094@w20-575-42.mit.edu>
To: aui@MIT.EDU
Date: Wed, 21 Jun 2000 11:22:03 -0400
From: "Christopher D. Beland" <beland@MIT.EDU>
Here's a core dump of planning I've done so far for formal usability
testing. What's mainly missing are application-specific tests (on
hold until we actually get working applications to play with) and the
very fine task-definition and logistical details which will be worked
out later.
Note that there is also some content which recommends changes to
existing code; please review, comment, and let me know if any of them
are actually implemented. If you have any information that refines
the many educated guesses I've made, please do send mail.
General UI Nitty-Gritties
My intuition tells me the following items represent weaknesses in the
current AUI prototype. If people agree and it's decided to implement
some of them right away, be overjoyed. If people want to wait until
the formal test results come back, that would be sensible, as my
suggestions are purely speculative.
Please do feel free to add to the list. Note that it's probable that
not everything on the list will be explicitly tested, due to length
constraints. Also, some of these will turn out to be more important
than others, and I haven't bothered trying to figure out which.
- Right-click to get pulldown menu is not obvious enough; add button
instead. Make it stay on top for very narrow windows, so they can
be closed, minimized, maximized, etc. intuitively.
- Hard to move applications if titlebar is not visible
(Better: Sides move, corners resize.)
- Meta-drag fails to move windows
- E.g. "Button-1" is much less intelligible than "Left mouse button"
- Raise, move, etc. should be on titlebar pulldown menu
- Hard to move transient windows (i.e. zwgc) - decorate by default.
- Tasklist is hard to read. Need to adjust sizing.
- Some iffy survey results came back on what people want Athena to
look like. 44% said Unix, 31% said Windows, and 7% said Mac. More
reliable numbers give 40%, 80%, and 25%, respectively, on
familiarity with interface. This, along with anecdotal evidence,
makes me suspect that there might be some amount of demand for
Windows and Mac look-alike themes, which we should be able to make
available by just making sure they're installed and don't break?
It would be very difficult to measure the actual demand for these
alternative unless they were actually available for testing, since
this is one case where explicit user desires may be considerably
different from actual preferences, especially given preconceptions
about the usability of a given (unseen) system.
In addition to these, there are some other items which I think might
come up in usability testing, but for which my intuition and/or
existing data does not provide a clear answer.
Questions:
- Is the desktop aesthetically pleasing? (colors, bg, icons, etc.?)
- What keyboard shortcuts for Sawfish should we enable by default?
- Is click-to-focus or point-to-focus more popular?
- What the placement policy for new windows be?
- Does the removable media icon clutter the taskbar?
- Should the panel be more than one row? (Window taskbar can be
heightened just by dragging; this is convenient for cluttered
desktops.)
- Is the taskbar too cluttered in general?
- Should the taskbar autohide?
- Should the taskbar have arrows on the hide buttons?
- Should the right-click popup menus onthe panel be diabled or
synched?
- Should clicking on an icon in the tasklist iconify or raise?
Testing Procedures
In addition to performing specific tasks, one useful thing to ask
people to do would be to spend some amount of time (perhaps a preset
amount of time) configuring their account the way they would like it.
(We could help motivate them by telling them that they could actually
keep the customizations for real if they wanted.)
We could observe and videotape them doing this, and collect
information on UI frustration, abandonment, and feature wishes. When
they had either finished to their satisfaction or time had run out, we
could give them a list of common customizations and see if there are
any they would have liked to implement, but didn't notice were
possible. Getting a baseline on their current customization level
would be important.
We should take some metric on how usable users found Gnome-Athena vs
Classic Athena vs Windows/Mac. With the very small initial sample we
are taking, a directed Q&A on this issue should suffice. I doubt we'd
want to do a large-scale survey on this, as I suspect there will be
clear agreement that we've made progress (rather than the reverse).
The most useful things to learn from this question are more about
details than overal ratings.
I spoke last night with a veteran UI tester from the private sector.
The following is based on her advice, which dovetails very neatly with
the Discovery Team recommendations.
- People: 1 user, 1 or more developers, 1 UI engineer
- Setup: Production computer and prototype. Video camera.
Preferably, the observers are out of view, but can see what the
user is doing at any time.
- UI engineer role: Give user set of specific and/or open-ended
tasks. Can only answer questions that pertain to clarification of
questions or tasks. May intervene to tell user skip ahead if stuck
on one thing for too long, to clarify if user has misunderstood
instructions, or to remind user to speak out loud. May not provide
assistance with UI or technical problems once testing has begun.
- Developer role: Experience has shown that developers' presence is
useful for educational purposes; they gain a much better
understanding of user mindset and behavior after observing. For
maximum effectiveness, should only intervene in case of
catastrophic technical failure (system crash, misconfiguraton,
etc.).
- Talk aloud method: Encourage users to speak aloud while working.
For maximum effectiveness, have user practice on a simple task,
such as refilling a stapler.
- Q&A: Observers conducted a question-and-answer session after
testing, being careful not to bias user response by prematurely
expressing strong opinions
Test Subject Choice
From surveys, it appears that as many as 60% of Athena users would
want to customize their logins (perhaps half of whom would only make
rather minor changes). Thomas tells me that a recent automated
dotfile survey found that 30-40% of users actually have
customizations. This corrleates with solicited and unsolicited user
feedback that simple customizations should be easier.
Combining this information with platform demographics from above, I
think that we'll need to identify at least ten test subjects, the more
the better, and preferably in the following ratios:
- 3 with no or little computer experience
- min. 1 not wanting customization
- min. 1 wanting some customization
- 3 with primarily Windows experience
- min. 1 not wanting customization
- min. 1 wanting some customization
- 3 with primarily Athena/UNIX experience
- min. 1 not having any current Athena customizations
- min. 1 wanting some customization
- min. 1 hardcore hacker who still wants to run Gnome
- 1 with primarily Mac experience
Incoming students would be useful to tap for people with no Unix
experience. Faculty and staff might be used, but current students are
greatly preferred. On the advice of that veteran UI tester, I'd
strongly advise against using I/S staff as subjects. Non-IT
adminstrative staff may be tapped, however, especially since they
generally prefer non-Unix platforms; again, students are preferred.
Effective methods of recruitment usually involve actively bribing
candidates to participate or guilting eligible friends into helping.
Hopefully, people will also want to get a sneak peek at the new AUI.
A good control question for subjects would be, "Why are you
participating in this study?"
That's all for now, folks.
Beland
===============================================================
Christopher Beland - http://web.mit.edu/beland/www/contact.html
Got spam? Stop it at the source. http://spamcop.net
===============================================================
