The current prototype of SFMOMA-DAM emphasizes getting images into the system--the creation and management of object and image records in the DAM database, and search--discovering what images are cataloged in the system. This is the core functionality upon which the rest of the system will be build. We conducted a pilot usability study in an attempt to assess system flow and functionality in relation to new task scenarios. The current prototype is the third iteration of the interactive prototype. It is inspired by design recommendations resulting from a formal heuristic evaluation conducted by peer review, and a reassessment of user needs.

It was our hope for the third prototype to make the core functionality of Quick Search and adding object and image records dynamic with a set of ASP scripts connecting to the DAM database. These scripts were not ready for the usability testing, however, so the tested version remained hard-coded. Unfortunately, given the push to create dynamic pages we didn't leave ourselves enough time to fully hard-code all of the pages either. We feel that the in-between state of the interface contributed significantly to the problems test subjects had completing the scenarios. Ideally, usability testing would have been postponed until after the interactivity was in place.

The purpose of this evaluation is to identify usability problems in the SFMOMA-DAM second interactive prototype. This report includes the following:

• Method
• Test Measures
• Results
• Discussion
• Formal Experiment Design
• Appendices

Method

Participants

The pilot usability study included four participants from SFMOMA. Volunteers were solicited from a group of Museum staff involved in the system development. All four participants reside in the Collections Department. The following table summarizes participant demographics:

Apparatus

The test required web access.

Tasks

The task scenarios have changed in recent iterations. Previous tasks emphasized searching for objects, requesting images, and the assigning metadata to images. A new priority emerged in the process of developing the system database. This change in priority effects our task analysis. A new emphasis is differentiating the creation of object and image records, and searching for images associated with objects. The tasks given to test participants included the following:

1a. Search for all objects with "blue" in the title.

1b. Locate all digital image versions of the object titled "football blue".

1c. How many objects are related to the "Drawer 3 of Blue Chest of Drawers" object? Which one is the root object?

2.a Add new object record for new acquisition.

2.b Catalog an image file for the new acquisition.

Procedure

Participants were scheduled for user observations on site at SFMOMA. Four participants were tested in three sessions. Participant 2 and Participant 3 worked collaboratively on tasks and were observed simultaneously. Testing took place in participants' workspace at the Museum. One group member served as facilitator and two served as note-takers and observers. Roles remained consistent throughout testing.

While all test materials were prepared and organized at the start of each test, the on-site location, use of participants' workspace, and consecutive scheduling limited the group's preparation of the testing space as recommended by Nielson. Each participant was greeted and oriented to the testing process by the facilitator. Each participant consented to the process by signing consent forms. Participants were asked to complete a brief form describing demographic information and related work experience.

Test Measures

• Navigation --Measure system flow to support tasks.
• Clarity of terminology --Assist user in system visibility and match system to organizational context.
• Ease of adding new records --Assess new design.
• Search --Assess new design.
• Learnability --Assess ease of use.
• Overall satisfaction --Assess total user experience of new system.

Results

The pilot usability study yielded significant feedback on the third iteration of the interactive prototype, as well as on the process of the usability testing.

Task completion:: All participants completed all tasks.

Time to complete tasks:

Errors: A total of two errors occurred during testing. Both were IE errors as a result of missing data in input fields.

Reported Satisfaction: Participants rated the system satisfactory on each test measure.

Written Feedback:

Discussion

The user observations inspired significant changes and rethinking in the design of the user interface.

Navigation : Users followed the system flow as designed to complete assigned tasks. Home, Search, and Search results appear easy to navigate. Problems arose due to the lack of system visibility on some pages, though. Some of the pages that caused confusion and proved challenging included: view image versions, add object and image records. Users resorted to using the BACK button in the browser to assist navigating between these screens. An internal "back" button that returns a user to the previous page could provide assistance within the interface. In addition, we would like to re-implement breadcrumbs on main navigation pages to assist in system visibility. However, addtional documentation including descriptive titles and instructions will also improve system visibility and assist users with navigation.

Clarity of terminology : We continue to struggle with terminology in use at the Museum. Problems arise in describing art objects as objects in an image management system and differentiating records of images of art objects. The Museum has a history of using terms such as item and main to represent objects in a catalog. In addition, while there seems to be a positive response towards the delineation of relationships between objects and between objects and images, there appears to be confusion on the descriptive terms. For instance, the third prototype uses "related objects" to identify all objects within a family of art objects. Users identified a mismatch in their experience using "family" or whole and parts. Other troublesome terminology includes subtle distinctions between root/parent/child for objects and view/versions for images.

Ease of adding new records: These screens are meant to be used only by the few users who will have permission to enter records into the database, which suggests that learning the terminology may not be a problem as long as it is logical and consistently applied. Nevertheless, trying to better match the users existing vocabulary would be a good idea, if such a vocabulary existed. We need to improve feedback so that users always know where they are in the add record/add image process. This might be able to be cleared up with additional informational text on the screens and with more consistent labeling. But there also seemed to a be a general resistance to the idea of dealing with objects first, when the real goal is to catalog (in this case) images.

Search : The newest iteration of quick search appeared unnecessarily complicated to users. Our recommendation is to return to a previous iteration and set a criterion default that is consistent with other Mueseum search interfaces that utilize "starts with".

Additional concerns:

• Redesign pages where content "above the fold" does not match user expectations.
• Object records need to be Confusion about the "object record" should diminish when the missing "three images" are more readily visible.
• All pop-up help windows should be reviewed and possibly redesigned to highlight key points first, and then show more complex details.
• Need to provide additional labeling and visual feedback to assist system visibility.

Formal Experiment Design

The study uses a within-groups design (all participants evaluate the same set of interfaces).

Participants:

The study design requires a minimum of 18 participants. Participants should be evenly split between imaging center administrators-those responsible for coordinating image creation and cataloging, and digital imaging specialists-those responsible for actually creating the images. Participants should also have moderate to advanced computer skills. Given these criteria it is likely that participants would have to come from multiple institutions.

Tasks

• Add 15 new record sets, comprising of one object and one set of images.
  
• Ten of the new records are for simple objects (no children).
• Three are child records.
• Two are grandchild records.
• The order of objects for which record sets must be created is the same for each participant.
• The amount of information entered for each record set is approximately the same for all objects

Hypotheses

1. Allowing users to chose the order of record creation will result in faster record creation.
2. Requiring users to add object records first, then image records will result in fewer data entry errors.
3. Hiding Parent / Child relationships will result in faster record creation.
4. Always displaying Parent / Child relationships will result in few data entry errors
5. The time it takes to add a single record set (object and images) will be slowest for records 1-5,
   will increase dramatically for records 5-10, and will level off between records 10-15.

Response Variables (dependent variables)

• Time: Time to add a new record set (object and images) *
• Errors: Number of data entry errors made while entering the new records **

Factors / Levels (independent variables)

• Order of record creation
  • Object Records first
  • Image Records first
  • Either Object or Image: User choice
• Display of Parent / Child relationships
  • Always displayed
  • Hidden: User chooses to display

Blocking and Repetitions

Appendices

Data:

• Critical Incident Log

Materials:

• Facilitator Script
• Observation Log
• Tasks
• User Questionnaire
• Participant Consent
• Demographic Information