Formal Experiment Design

The user interface for the SFMOMA Digital Asset Management System has evolved to support two broad classes of users: 1) those who want to find, view, and use digital images, and 2) administrators and digital imaging specialists who want to catalog images and manage the digital asset collection. Accordingly, formal usability testing would involve designing two experiments. Here we describe only one experiment, one which targets administrators and digital imaging specialists. The goals are 1) to measure how long it takes to enter new record sets (object + one set of images) into the system; 2) How many data entry errors are made entering the record sets; and 3) whether the rate of record creation improves as more records are entered. The interface will vary on two dimensions: 1) The order of record creation and 2) display of parent / child relationships.

The study uses a within-groups design (all participants evaluate the same set of interfaces).

 

Participants:

The study design requires a minimum of 18 participants. Participants should be evenly split between imaging center administrators-those responsible for coordinating image creation and cataloging, and digital imaging specialists-those responsible for actually creating the images. Participants should also have moderate to advanced computer skills. Given these criteria it is likely that participants would have to come from multiple institutions.

 

Tasks
  • Add 15 new record sets, comprising of one object and one set of images.
  • Ten of the new records are for simple objects (no children).
  • Three are child records.
  • Two are grandchild records.
  • The order of objects for which record sets must be created is the same for each participant.
  • The amount of information entered for each record set is approximately the same for all objects

 

Hypotheses
  1. Allowing users to chose the order of record creation will result in faster record creation.
  2. Requiring users to add object records first, then image records will result in fewer data entry errors.
  3. Hiding Parent / Child relationships will result in faster record creation.
  4. Always displaying Parent / Child relationships will result in few data entry errors
  5. The time it takes to add a single record set (object and images) will be slowest for records 1-5,
    will increase dramatically for records 5-10, and will level off between records 10-15.

 

Response Variables (dependent variables)
  • Time: Time to add a new record set (object and images) *
  • Errors: Number of data entry errors made while entering the new records **
 

* Record set = The information that needs to be entered into the database to adequately describe one object plus one set of images related to that object.
** A data entry error is defined as 1) entering incorrect or partial data into a field, 2) attempting to proceed without entering any data into a required field, 3) not adding both components of a record set: that is both object and image records for a given object.

 

Factors / Levels (independent variables)
  • Order of record creation
    • Object Records first
    • Image Records first
    • Either Object or Image: User choice
  • Display of Parent / Child relationships
    • Always displayed
    • Hidden: User chooses to display

Blocking and Repetitions
Three trials per block, 18 total participants.
Record Set order (1-15) same for each block.

Displayed Hidden
Obj Img
Img Ch
Ch Obj
  Displayed / Hidden: Whether parent / child relationship information is always displayed or hidden until user requests it.
  Obj:

Enter object records first
  Img: Enter Image records first
  Ch: Choice of user