Manis Interface Project
Assignment #8 - Pilot Usability Study
Contents
Introduction
Application Description
The MaNIS application (Mammal Networked Information System) is a network of distributed databases of mammal specimen data. The project is a collaboration between 17 research institutions and natural history museums, funded by the National Science Foundation. MaNIS makes information available for nearly a million museum specimens.
We are developing a user interface prototype for the MaNIS application. The interface offers two major features: 1) the ability to execute searches of the MaNIS database using a set of search criteria such as taxon, location, date, and institution and 2) the abilitiy to manipulate or view search results using various utilities such as sort, print, export, and field display.
Purpose of Study
The overall purpose of the study was to determine if users could understand, navigate, and utilize the interface without any training or explanation. More specifically, we wanted to find out if there were marked improvements between the first interactive prototype and this second interactive prototype as a result of design modifications made according to Paparazzi's heuristic evaluation and Marti's feedback.
Method
Participants
All of our participants were selected because they were representative of the different types of users who would use MaNIS in the course of their work. They were recommended by either sponsors of the larger scale ManIS project or interviewees/participants from our prior project activities.
| Participant 1 | Participant 2 | Participant 3 | |
|---|---|---|---|
| Gender | F | F | M |
| Age | 43 | 40+ | 48 |
| Occupation | student and mom | faculty curator | research information manager |
| MaNIS Usage Frequency | 0 times | ~10 times | 20-30 times |
| Comfort Level with Computers | pretty comfortable | moderately comfortable | very comfortable |
Apparatus
Usability tests were conducted at several locations: a work station in the SH210 lab for participant 1, a work station in a faculty office for participant 2, and a laptop at an MVZ breakroom for participant 3. A stop watch was used to time task execution. All 3 particpants used an Internet Explorer browser to render the MaNIS interface prototype. The prototype was developed using a combination of php, html, javascript, css, and MySQL. Participants were also asked to sign a consent form, complete a demographic survey, and answer an interview questionnaire.
Tasks
These scenarios are slightly modified from those suggested for testing the first prototype during Assignment #5.
Scenario A: Which museums should I visit to review specimens?
You need to review the taxonomy of the Hairy-tailed bats, genus Lasiurus, as a foundation for ecological fieldwork observations in California [see persona Robert Mismer] . By examining DNA from museum specimens caught in the past 80 years and the ones you catch in the field, you hope to be able to say something about the robustness of the hairy-tailed bat population's genetic diversity.
- Task 1: Get a count of available specimens for Lasiurus.
- Task 2: Narrow your results to specimens collected in California since 1920.
- Task 3: Specify that the results include only the following information: Catalog Number, Institution, Scientific Name, preparation, and Collection year.
- Task 4: Download the results
What we looked for...
This scenario was created to illustrate a common interaction a user might have with the MaNIS application. It was used to determine how easily and effectively search conditions such as taxon, location, and date could be specified. It was also used to assess the usefulness of features such as "total record count", "show/hide fields", and "export."
Scenario B: Where is the best place to capture a bat?
You're beginning a study on Lasiurus cinereus in New Mexico [see persona Helen Williams]. You'd like to capture a few animals, and will have the best luck going where they have been found before. By looking at the locality information for where specimens of these bats, you can plan where in New Mexico to go to set up mist nets. Ideally the collection record will be relatively recent, to avoid traveling to someplace where the correct habitat no longer exists.
- Task 1: View a list of all Lasiurus cinereus from New Mexico.
- Task 2: Organize the records by when they were collected.
What we looked for...
This scenario was created to test the "sort" feature of the application.
Scenario C: What species is this?
As a curatorial assistant [ see persona Debbie Twist] at the Museum of Vertebrate Zoology (MVZ), you are bringing a new specimen into the collections, and need to identify what species it is. Field notes indicate that it is a bat of the genus Lasiurus, and was collected in Pasadena, California. Once you see which Lasiurus species' distributions include Pasadena, you'll be able to go to the specimen cabinets and compare the new specimen with the identified specimens, and determine which species the new specimen belongs to.
- Task 1: Pull up the records of Lasiurus bats collected in Pasadena, California.
- Task 2: Limit the results to those specimens held in the MVZ collection.
What we looked for...
This scenario was created to assess whether it was clear to the user that the location panel could be used to conduct two types of location searches - by state or locality. It was also used to verify the effectiveness of the insitution search.
Procedure
Each team member played one of the following roles:
- Facilitator - responsible for any direct communication and interaction with the user
- Timer - responsible for recording the time it took for the user to execute each task
- Error Recorder - responsible for recording user errors during task execution
- Comment Recorder - responsible for recording postive and negative user comments
One hour was set aside for each session and the the following procedure was followed with each of the 3 participants:
- User was briefed about the purpose of the project and the usability test.
- User was asked to sign a consent form.
- Facilitator demonstrated to the user how to "think aloud."
- Facilitator read the complete scenario to the user.
- Facilitator re-read the scenario to the user one task at a time.
- User executed each task.
- Timer recorded amount of time it took to execute each task.
- Error Recorder logged any actions the user took that deviated from the expected action.
- Comment Recorder jotted down any positive or negative statements made by the user while executing tasks.
- Upon completion of all 3 scenarios, the Facilitator answered all user questions that were deferred to the end of the session.
- User was asked to complete a demographic survey.
- User responded to follow-up/interview questions.
- Facilitator ended the usability test session.
Test Measures
We measured 2 items: 1) Task Time - time it took for the user to execute each task within the scenario and 2) Number of Errors - number of errors committed while executing a task. Tasks in each scenario are closely tied to features of the application. Features that cause long task times or errors might be obscure, confusing, or non-intuitive, and are likely candidates for re-design.
Results
The overall results from our pilot usability tests were very positive. People generally found the interface easy-to-use, and we also got a lot of good feedback about areas for further improvement.
The following list summarizes our observations during the usability tests. See the document User Comments for our detailed notes on the tests.
- (+) Users liked the step-by-step search refinement. One user commented that it made her feel like she was browsing through the collection and that she might discover things she would otherwise not see.
- (+) Initially, the process of setting search conditions was a little confusing, but once users had completed one successful search they didn't have further problems.
- (+) Users described the interface as "easy-to-use" and "straightforward."
- (+) Users liked the "Did you mean" feature, because they frequently misspelled typed-in taxon search terms.
- (+) One user liked that it was "OK to make wrong choices."
- (-) The Show/Hide Columns link was hard to see, and the panel didn't allow users to select/deselect all checkboxes.
- (-) Users didn't see the New Search link.
- (-) All users had problems with the Location text entry box; they weren't sure how to correctly specify a location.
- (-) Users did not read the text that was on the screen.
- (-) Use of links for list of institutions suggested that clicking on them might lead to the websites for the institutions.
- (-) One participant wanted feedback on which fields were being searched by the query.
The following table shows the time it took our participants to complete each task. We were unable to collect timing data for one of our participants, but in general this person was able to complete each task in a shorter amount of time than the other participants.
| Task | Participant 1 | Participant 2 | Average Time |
|---|---|---|---|
| Scenario A | |||
| Task 1 | 55 sec | 120 sec | 87.5 sec |
| Task 2 (California) | 90 sec | Technical difficulties (was taking a while anyway) | 90+ sec |
| Task 2 (since 1920) | 25 sec | 7 sec | 16 sec |
| Task 3 | 50 sec | 60 sec | 55 sec |
| Task 4 | 10 sec | 20 sec | 15 sec |
| Scenario B | |||
| Task 1 | 20 sec | 20 sec | 20 sec |
| Task 2 | 25 sec | 30 sec | 27.5 sec |
| Scenario C | |||
| Task 1 | 75 sec | 40 sec | 57.5 sec |
| Task 2 | 40 sec | 15 sec | 27.5 sec |
The following table shows the number of errors for each scenario. For detailed descriptions of the errors, see User Errors.
| Scenario | Participant 1 | Participant 2 | Participant 3 | Average Errors |
|---|---|---|---|---|
| Scenario A | 4 | 6 | 3 | 4.3 |
| Scenario B | 1 | 1 | 1 | 1 |
| Scenario C | 5 | 1 | 1 | 2.3 |
Here are the questions from our followup interviews with summaries of the answers.
- How would you rate ease of use?
- Overall satisfaction level?
- Overall did you like the new interface? How does it compare to using the current MaNIS interface?
- Given the choice, which interface would you use? Why?
- Are there any other tasks that you would like to do using MaNIS?
- Was any part of the inteface confusing? If so, which part(s)?
- Do you think anything is missing?
- Would you like to make any other comments/suggestions?
Average answer: 4.3 (on a scale of 1 to 5, 1 = very difficult, 5 = very easy)
Average answer: 4.3 (on a scale of 1 to 5, 1 = not satisfied, 5 = completely satified)
Most users liked the new interface and found it easier to use than MaNIS.
Most users preferred this interface to the current MaNIS interface. One user mentioned that she couldn't understand how to build a query using the old interface.
Searches of specimens from other parts of the world; world museums.
There was some learning curve in trying to understand how the search conditions worked. Would like to know what database fields are being searched. Show/Hide Columns initially seemed like it would narrow the search.
Searching by collector, preparation, and specimen number.
Make instructions larger.
Discussion
Because of the small number of users, our test measures are not statistically significant. However, both the time-on-task and the errors made suggest patterns that support what we observed during the interviews: users needed to find their way around the interface the first time, but once they did they were able to complete similar tasks again quite quickly. The need to learn during the first task should be true of any interface that is new to users, and so the more interesting question is whether this interface takes less time to learn than the current MaNIS search interface. Our study was not structured to address this numerically; however, qualitative responses from users who had experience with MaNIS suggested they prefered the new interface design.
Given a user preference for the browsing/search-refinement-with-feedback approach embodied in the new interface design, it is not clear that time-on-task is a critical issue. Our previous interviews indicated that users (other than collection curators) were likely to access the MaNIS database very infrequently, perhaps once or twice a year. Under these conditions, it is more important that users feel they are able to understand and work with the interface as beginners than that the interface is able to support timing improvements for expert, daily users.
Interestingly, although the task timings and error rates indicate that our participants with more MaNIS experience were able to figure out the new interface quickly, they were also mislead by their expectations that the search interface interactions would be complex. They were concerned about getting the spelling of search terms correct, attempted to use wildcard characters, and expected a need to somehow explicitly indicate that they wished to combine search terms. In other words, they didn't know they could trust the interface to provide them feedback, and they attempted to pre-emptively guess what contortions might be required in order to get the desired search results. The user with moderate experience on MaNIS expressed relief at the guidance the new interface provided; the heavily experienced MaNIS user expressed a desire for more feedback about how the search terms were being used to create the result set, and some ability to tweak that. Taken together, our interviews suggest that a) the new interface does a good job of supporting common search tasks for all levels of users; b) even more feedback should be given; and c) tools with a power similar to the current MaNIS' boolean-query construction feature should be developed for power users.
As with any opportunity to watch users with an interface, these interviews provided us with ideas for features and with feedback about what users expected the interface to have or to do. For example, one user expected to be able to reverse the sort order by clicking on the sort link again; users found consistency lapses on the Show/Hide and Institution panels; and another user reminded us that the MaNIS interface would eventually be used by scientists in other countries, suggesting that language localization features should be developed.
We were also aware, however, that the rate of new ideas was rather low, and that the interviews felt a little like those staged commercials where the "real customer" decides they like the new product, yes, it's much better than the leading brand.... In other words, probably because of the extensive iterations the design had undergone during the paper prototype stage and the heuristic evaluation, although we were just starting the testing of the online-prototype we were already reaching the point where the rate of feedback begins to tail off, and the benefit of additional user interviews is declining. We found this a strong testament to the power of the design process undertaken during the semester.
Formal Experiment Design
Hypotheses
- Hyperlinking the column headings to let users to control the left, search condition entry panel will allow users to enter their search conditions more quickly and effectively than just having hyperlinks across the top of the page.
- Graphically representing Boolean ANDs and ORs will help users understand what their query is saying better than displaying the corresponding SQL statement.
Factors and Levels
Factors (Independent Variables)
- Hyperlinking of Column Headings
- Levels: On, Off
- Query Representation
- Levels: Stacked Boxes or Comma-Separated Boxes, SQL Statement
Response (Dependent) Variables
- Time
- Number of Errors
If users take less time to complete a task (by using hyperlinked column headings) and/or make fewer errors in entering search criteria when column headings are hyperlinked, then we will be able to accept our first hypothesis. If users learn how to interpret the representation of the query more quickly and they make fewer errors in identifying or recognizing the type of Boolean query represented when shown a graphical version, then we will be able to accept our second hypothesis.
Blocking and Repetitions
Because of the difference between reading a SQL statement and viewing a graphic, the query representation will be between subjects. The hyperlinking of column headings will be within subjects.
| SQL Statement (16 participants) |
Comma-Separated Graphical Representation (8 participants) |
Stacked Graphical Representation (8 participants) |
||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
Graphical Representation Examples
Comma-Separated
Stacked
