Group 3: California Digital Library - Prototype Description

California Digital Library

Assignment 9: Pilot User Study

FORMAL EXPERIMENT DESIGN

Our pilot study gave us valuable insight into planning a formal usability study. Including a larger number of test subjects will allow us to more accurately measure the specific effects of individual interface changes. The most important interface element which remains to be improved is the understandability of the two-level search and the differentiation between database resources ("catalogs") and full-text resources ("holdings"). We will compare several alternative interface changes and measure the quantitative effects on users' search patterns.

HYPOTHESES:

We hypothesize that specific changes in the layout of the site will have a measurable effect on users' seach accuracy, as evidenced by their choice of resources which are more appropriate for their specific information need, and which result in the retrieval of more relevant information.

We also hypothesize that there will be an inverse relationship between search speed and search accuracy. Accuracy can be improved by the development of an interface that encourages users to read the instructions rather than proceeding blindly; this increased time spent reading may result in slower search processes for novice users. However, user satisfaction should be more strongly correlated to accuracy than to speed. Users will be willing to spend longer searching if they feel they are successfully retrieving their desired information. We do expect that the added search time will be reduced substantially as users gain familiarity and expertise with the system, but we do not expect this reduction to be measurable over the short length of our study.

INDEPENDENT VARIABLES (FACTORS):

We will hold constant the overall site architecture and vary only two specific design elements between the alternative interfaces. For each change, the baseline will be our existing Prototype #2.

Variable #1: A graphical redesign of the front page to more closely integrate the descriptive text with the links, including the screen location of the text and the relative font sizes, in order to increase the visibility of the explanatory text and encourage more in-depth reading.

Variable #2: A graphical redesign of both the front page and the browse & search pages to emphasize the distinction between "catalogs" (e.g., journal article databases) and "holdings" (e.g., electronic journals on the web). We may include images to connect the different types of digital resources with their physical-library equivalents.

These two factors, each with two levels, will result in four alternative interface designs.

In our analyses, we will also control for the users' domain of knowledge, experience with the California Digital Library and other digital libraries, and experience with physical libraries.

DEPENDENT (RESPONSE) VARIABLES:

First, we will measure the total time for the completion of each scenario. Completion is defined as when the user feels satisfied with their search results.

Second, we will use two different measures of search accuracy:

Initial choice of search path - a dichotomous variable: Did they choose an appropriate path for their scenario (as arbitrarily defined)

Recall - number of valid search results retrieved (as a percent of the total valid results available)

Finally, we will measure self-reported satisfaction, using a post-test questionnaire.

BLOCKING AND REPETITIONS:

Because our response variables are so closely tied to user learning and understanding of the system, it will be impossible for us to test multiple interfaces on the same subject; it would be impossible to separate the "learning" due to one interface from that due to another. Therefore, we will test each subject with only a single version of the interface. We will, however, test multiple scenarios with each subject (all on the same interface). Because of the wide range of expected responses, the confounding variables for which we will control, and the four alternative interfaces, it will be important for us to test a large number of users. With five indenpendent variables (2 analysis and 3 confounding), we will need at least 100 to 150 participants.