Test results
The test revealed
a number of issues with the initial design, ranging from cosmetic to
quite serious. The main findings are listed below:
- The main entry
screen was confusing, and did not help a new user to understand the
purpose of the site. For instance, users expressed confusion about
when they would need to login.
- The location,
order, and grouping of the navigation controls did not assist the
users in finding commonly used functions.
- The search options
drop-down menu was apparently not helpful, or at least did not catch
the users' eye, since no one tried it. For one user, the search options
actually made the search function appear too complex and "intimidating".
- The categories
used for browsing courses (adopted from the existing SIMS web site)
did not map to the tasks or meet the user's expectations.
- The comment pages
need to describe the scale for users to be able to interpret the ratings.
For instance, to understand a rating of "two thumbs up",
the user must know that the highest possible rating is "three
thumbs up".
- Users want to
see the individual commenters' ratings with their comments, not just
aggregated ratings.
- Features such
as the links to past semesters' information and "people who liked
this course also liked..." were not easily noticed or understood.
- After viewing
a course description, users want an explicit "exit" function.
The initial design had provided only an implicit exit, e.g., when
done viewing a course description, select another function such as
browse.
- Access to the
add comment function needs to support a verb-noun interaction (decide
to add a comment, then select the course) as well as noun-verb (view
a course description, then decide to add a comment).
- The use of a
forced "Preview" function on the comment page elicited mixed
reactions ranging from initial surprise to annoyance.
- The "Shopping
Cart" button, even though it related to a function not modeled
in this prototype and not needed by the tasks, attracted attention
and caused confusion.
- No users selected
Help/FAQ, even when verbally expressing confusion about the site.
On the positive
side, users liked the following features:
- Although links
to past semesters' information had to be explained, the function elicited
positive reactions once explained.
- Although the
"people who liked this course also liked..." function took
a while for users to notice, the users thought it was a great idea.
- The use of "thumbs
up" and "thumbs down" images for the rating scale was
understandable as well as amusing to the users.
In addition, users
suggested several additional functions:
- Users requested
a way to communicate with commenters, specifically by providing their
email addresses.
- One user questioned
the absence of an anonymous posting function.
- One user requested
a way to view all comments by a particular commenter. This would allow
the user to gauge the commenter's credibility by viewing their comments
on courses that the user had taken.
Discussion of findings
and plans to address
The test results
provide a wealth of valuable feedback for modifying the design. Essentially,
the findings can be considered as falling into three categories:
- Layout and clarity
issues that we will attempt to address in the next design iteration.
- Complex functional
issues that will require additional investigation to resolve.
- Issues which
seem at least partly due to limitations of the test set-up or the
paper prototype itself. These issues will need to be monitored in
subsequent iterations.
Layout and Clarity
Issues
Clearly, both the
main entry page and the set of navigation tools which appear on every
page need to be modified significantly. The focus of the main entry
page needs to be on providing information which will help a new user
understand the site, while allowing an experienced user to proceed quickly
to the task at hand. The navigation tools layout needs to be redesigned
to ensure that the frequently used functions are placed in prominent
and logical positions and are not overwhelmed by the number of less
important functions. In addition, a number of concrete improvements
to the comments pages were suggested, such as the addition of information
on the rating scale and an explicit "return to browsing" or
"return to main page" function.
Issues requiring
additional investigation
The set of course
categories used in this initial prototype did not seem satisfactory.
Unless these categories can be improved, the browsing options add complexity
without actually assisting the user. Beyond simple categories, additional
metadata might improve both browsing and searching. The team would like
to use card sorting to determine better course categories or appropriate
facets. This in turn will influence the design direction of the search
and browsing functions.
Several new functions
were requested by individual users, such as a mechanism to view all
comments by a particular commenter. Before expending substantial effort
in designing them, we want to obtain information from a larger set of
users confirming the desirability of such functions. This may be accomplished
with a questionnaire.
Issues to be monitored
This paper prototype
did not model all of the functions envisioned for the site. Specifically,
we did not include the function to allow a user to save a list of courses
in a personal "portfolio" or "shopping cart". Although
this test clearly showed that the "shopping cart" metaphor
was not clear in this context, little else can be inferred. A later
iteration of the design will include this function, and allow us to
determine more about its usefulness.
One specific observation
concerned the possibility of accidentally adding a comment to the wrong
course. An error of this type was observed once (i.e., the task statement
was "add a comment to IS250", but the currently displayed
course was IS213, so the user added the comment to IS213). This may
be a case where the interface was genuinely confusing, or may be due
to the artificial nature of the testing situation. In other words, the
user may have reasonably expected the interface to be "reset"
to a correct starting point for the task being assigned. No immediate
changes are planned to address this, but the team will be aware of this
potential issue when structuring later tests.