Methods
Participants
We sent out a call for
volunteers to the students@sims mailing list. From the respondents, we
selected three students that closely matched our personas. None of the
students were participants in our earlier focus group. The students included:
-
Participant 1: A female
first-year Masters student (native English speaker)
-
Participant 2: A male
first-year Masters student (non-native English speaker)
-
Participant 3: A female
second-year Masters student (native English speaker)
Task Scenarios
Task 1: You are a SIMS
student getting ready to sign up for your second semester classes. You
are thinking about taking a networking class and want to look at the course
reviews for IS 250.
For this task we
were interested in what method (e.g. browsing vs. searching) the participant
chose to use and how successful they were in first locating the specified
course page and then reading the comments.
Task 2: You are a
SIMS student getting ready to sign up for your second semester classes.
You are thinking about taking a user interface design class.
(If user
used "Search" mechanism in task 1, go to part 1. If user used "Browse"
mechanism in task 1, go to part 2)
-
IS 213 has been recommended
to you, and you want to read more about it. Browse the course listings
to find the comments about this class.
-
IS 213 has been recommended
to you, but you can't remember its exact name. Search for user interface
design classes to find comments about IS 213.
For this task we were
interested in the success of the participant using the method they did
not initially choose in finding the specified class.
Task 3: You are a
SIMS student who has just finished taking IS 250. You really enjoyed the
class and want to write a comment so future students can read it. You liked
the course, thought the professor was excellent, and thought the coursework
was challenging but worth it.
For this task we
were looking to see both how the participant chose to go about commenting,
whether they were successful in adding a comment, and how they worked with
the commenting features (ratings, etc.) available.
Procedure
The team took on the
following roles during the test:
| Facilitator |
James |
| Computer |
Jean-Anne |
| Recorder |
Linda |
| Recorder/Photographer |
Sonia |
The tests were scheduled
in one-hour blocks on a Saturday afternoon. Each participant, on arrival,
was greeted and provided with tea and cookies. The facilitator explained
the purpose and background of the project and outlined the procedures and
expectations of the test for the participant using a prepared script. The
facilitator then answered any additional questions and provided the participant
with an informed consent document to sign.
The facilitator read
the first task scenario to the participant and allowed the participant
to ask questions regarding the task. When the participants were satisfied
that they understood the scenario, they were asked to begin. The initial
screen (before the first task) showed the "Browse by Subject" page in the
main panel.
While the participant
navigated through the prototype, the facilitator attempted to keep the
participant "thinking aloud," the computer operated the paper prototype
and signaled the participant when the prototype was done and ready for
more input, and the two recorders took notes on the critical incidents
and photos of the participant's interaction with the prototype.
When each task was
completed, the facilitator asked the participants general questions about
their experience with the task. The facilitator then read the next task
to the participant. The standard protocol was to leave the prototype in
the final state of the previous task when commencing the second task, but
we "restarted" the interface for Participant 3 at her request (she was
unclear as to how to proceed from some of the screens).
After all the tasks
had been completed, the facilitator and the rest of the team led the participant
in a general discussion of her experience with the interface prototype.
Test Measures
We were interested in
the following informal test measures.
-
Can users navigate through
the system to accomplish the given task?
-
This was the most important
test measure. If users could not find their way to the screens where their
given task could be accomplished, then our entire navigation scheme would
have to be redesigned.
-
Do users prefer browsing
or searching?
-
We implemented both
searching and browsing mechanisms. We were interested in whether users
preferred one over the other so that we could focus more heavily on one
mechanism.
-
Do users make use of
the various browsing categories?
-
The courses were presented
in categories to help with browsing. We wanted to see whether users would
find the categories useful.
-
Is the placement of
"static" elements logical to users?
-
We had a number of "static"
elements in the interface: a toolbar, navigation elements, and the screen
where information was displayed. We wanted to know if these elements were
in logical places on the screen or if users had any difficulties finding
them.