Evaluation
Our evaluation was casual and small-scale. Colleen showed the second
iteration of the prototype to three members of the target
audience at the California Digital Library. She asked each participant
a short series of questions grouped into two
conceptual sets. Colleen asked the question, then observed and took notes
as the participant used the visualization.
Because the interface is still in a very rudimentary phase, with no
explanatory text, help features or tutorial, she provided a brief verbal
overview of the type of data accessible through this visualization, along
with a short demonstration of navigation.
The first set of questions was task-based and data-centered, with the
goal of ascertaining whether the prototype met the basic need of helping
users to identify the major patterns in circulation and holdings across
time.
- What is the approximate number of charges for 2003 in Music,
Dance, Drama and Film? How does it compare to the number of charges
in other subject areas at that time?
- Which subject area tends to have
higher circulation over time:
Engineering or Literature?
- Are there particular months of the year
when circulation seems consistently high or low in general? Do the
patterns appear to be subject-specific,
or applicable across subjects?
- Are there any items that have both
very high holdings and high circulation in languages and literature
in 2001?
- Did you identify anything of particular interest in the holdings
and circulation patterns as you browsed?
A second set of questions was open-ended and design-centered, aimed
at understanding whether the user interface enhanced or impeded understanding
of the data.
- Is the process of zooming in from left to right intuitive?
- How do you
like the look and feel of the interface?
- What do you find most appealing?
Least appealing?
- Is loading time reasonable?
- What additional data and/or features would
you find most helpful?
- Any additional comments on this visualization?
The three participants, all California Digital Library staff, were:
- Programmer, female. Trained in former UCB library school, very familiar
with bibliographic data and library practice. Chosen because she
is a member of a project team that is experimenting with the use of
circulation
and holdings data as weights in information retrieval. [link to notes]
- Programmer, male.
Has worked with bibliographic data extensively while working
on retrieval systems for CDL. Chosen because he is a member of
a project team that is experimenting with the use of circulation
and holdings data
as weights in information retrieval. [link to notes]
- Analyst, female. Chosen because she
has expertise in UI design. [link to notes]
Participants successfully used the visualization to explore the patterns
of interest: circulation by subject area, circulation over time, and
the relationship of circulation to holdings. All three commented on the
clean, simple look of the interface, and noted that graphs and text were
easy to read. They all liked the ability to drill down for more detail.
There were a number of user interface problems that became apparent
in the course of the testing:
- All participants could see the value and desirability of being
able to narrow down the number of subjects on view in the left-hand
window, but
two out of three abandoned the use of that filter because they didn’t
want to deselect so many checkboxes.
- Although we made more graphs visible at a time by reducing size
and going to a 2-column layout, a minimal amount of scrolling was still
needed
on the left-hand side. This slowed participants down when asked the second
question about comparing the value across subjects.
- All participants liked being able to drill down to the item-level
view, but were impeded by the clumping of values at the bottom of
the scatterplot when large numbers of points were visible.
- Subject filters reset when a year filter is chosen, which caused
problems for 2 of the 3 participants. The filter values should
be persistent.
-
All three commented on the need for clearer and more consistent
labeling of the axis values.
During the open-ended questions, participants offered many excellent
ideas for improving and expanding upon the visualization. Among the best:
- Add the ability to toggle between two alternative holdings dataset
to explore those differences.
- Change the subject filter so that they
see all by default, or can build a custom view by adding subjects
rather than subtracting them.
- Consider allowing participants to combine subjects
in a single bar graph, and then generate the scatter graph on the
combined subjects.
- Add mouseovers on the image maps to display the column values.
- Highlight
selected items, to reinforce the notion of “drilling
down” into the dataset.
|