Participants
The four participants
were recruited from the SIMS student body. Three students responded
to an e-mail query broadcast to the student body, and one to a verbal
invitation. All four respondants were 1st year Master's students. Two
were female; two were male. Detailed
demographic information was collected using the post-test
questionnaire.
Setup
The tests took place
in a SIMS computer lab at a Windows NT workstation using Netscape Navigator
as the browser. Before the test sessions began, we cleared from the
database all of the test ratings and comments and then seeded the system
with coherent ones to provide our users with realistic reading interactions.
Procedure
The homepage of
the prototype was loaded and the browser was minimized before the user
arrived. Before each test, the moderator welcomed the user and briefed
him or her on the purpose and procedure of the test. The user was given
a chance to ask questions about the test and was provided with a consent
form. We chose not to demonstrate the system for the users before
the tasks so that we could observe how users might interact with the
system with no guidance.
For each task, the
moderator gave each user a paper description of the task, and asked
the user to signal when he or she had read the task, understood it,
and was ready to begin interacting with the prototype. The user's signal
marked the start of the task for the purposes of timing. The moderator
chose the second task (either a search or browse task) to be the opposite
of the method chosen by the user in the first task. In cases where the
user ignored the specified method, the moderator did not intervene.
During the test,
one observer timed the duration of the tasks, recording both how long
it took for the user to find the correct page and the how long the user
spent reading or interacting with a page before being satisfied that
the task was complete. All three observers took written notes regarding
each user's interaction with the prototype.
After all three
tasks had been completed, each user was asked to complete a written
questionnaire and was offered snacks. After the questionnaire was
completed, the users were asked a series of open-ended interview questions:
some prepared, some based on the individual user's interactions and
questionnaire answers.
Tasks
Task 1:
Assume it is your first semester at SIMS and you are getting ready
to sign up for next semester's classes. You are interested in taking
a class on networking, so you want to find course reviews on IS250.
Task 2:
Again, assume it is your first semester at SIMS and you are getting
ready to sign up for next semester's classes. You are interested in
taking a class on user interface design. You know there is a class
on user interface design being offered, but you can't remember its
exact name or course number. Try browsing the course listings to find
a relevant class.
or
Task 2:
Again, assume it is your first semester at SIMS and you are getting
ready to sign up for next semester's classes. You are interested in
taking a class on user interface design. You know there is a class
on user interface design being offered, but you can't remember its
exact name or course number. Try searching to find a relevant class.
Task 3:
This time, assume you have completed your first semester, and you
want to express your opinion on IS206. You can go ahead and say whatever
you want, either your real opinion or make something up. (Keep in
mind that what you enter will be viewable on the web, at least for
the duration of this test, but your real name will not be shown.)