IS213 Projects Home

Assignment 1

• Problem Statement
• Project Members

Assignment 2

• Personas
• Task Analysis
• Appendices
• Work Distribution

Assignment 3

• Scenarios
• Comparative Analyses
• Initial Design 1
• Initial Design 2
• Initial Design 3
• Work Distribution

Assignment 4

• Introduction
• Low-Fi Prototype
• Methods
• Test Measures
• Results
• Discussion
• Appendices
• Work Distribution

Assignment 5

• Design Revisions
• Revised Scenarios
• 1st Hi-Fi Prototype
• Storyboards
• Instructions for Use
• Slide Presentation
• Work Distribution

Assignment 6

• Heuristic Evaluation
• Work Distribution

Assignment 7

• 2nd Hi-Fi Prototype
• Responses to HEs
• Work Distribution
• Instructions for Use

Assignment 8

• Introduction
• Method
• Test Measures
• Results
• Discussion
• Appendices
• Work Distribution
• Slide Presentation

Final Write-up

• Problem Statement
• Solution Overview
• Final Personas
• Final Scenarios
• Revised Storyboards
• Final Interface Design
• Design Evolution
• Interactive Prototype
• Instructions for Use
• Work Distribution
• Slide Presentation

Method

Participants

We first attempted to recruit participants from the body of pre-existing IMG users who had previously registered for an IMG tutorial. Unable to schedule enough experienced users during the week of the study, we chose to use two volunteer professional annotators, as they have good domain knowledge and experience with the annotation process.

The three participants we engaged came from backgrounds in professional annotation, molecular biology, and genomics. Their ages ranged from 35 to 45 and their education level, from Bachelor's to PhD. Only one of them was previously familiar with IMG, although all had some experience with manual or automatic annotation.

A fourth study was conducted with an individual between the ages of 25 and 35, with a Master’s degree in molecular and cell biology and experience doing professional annotation. However, much of the data from that interview was irrelevant to evaluating this add-on system, since the user had a strong tendency to ignore our tasks and start exploring other, unrelated elements of the IMG website. Also, we erred in failing to log out the previous user, and this participant said that not having to log in affected her behavior. Thus, we have decided to omit the data from her study.

Apparatus

Usability tests were conducted on-site, in the laboratory and offices of the users, in order to make the environment as realistic as possible. Since the functional prototype was available online, the only materials we used were a computer with internet access, an audio recorder, and pen and paper or a second computer for note-taking.

Tasks

The task sets for this study were drawn from our existing personas and scenarios. However, in order to facilitate the quantitative measurements, we broke down the three primary tasks into six sub-tasks, detailed below.

Task Set 1

* Start from the gene list: http://groups.sims.berkeley.edu/annotation/
* View the annotation history for the first gene on the list.
* Modify the existing annotation for that gene with data from a homolog:
  - GO function; EC number; Gene Name.
* Submit your annotation.

Task Set 2.

* Start from the gene list: http://groups.sims.berkeley.edu/annotation/
* Express your opposition to the annotation for the first gene on the list.

Task Set 3.

* Start from the Gene Cart: http://groups.sims.berkeley.edu/annotation/genecart.php
* Change an annotation by adding a COG group.
* Add this gene to your list of watched annotations.

Task Set 4.

* Start from the Gene Cart: http://groups.sims.berkeley.edu/annotation/genecart.php
* Comment on annotations for multiple genes.

Task Set 5.

* Starting from anywhere on the site, find where you can view your watched annotations.

Task Set 6.

* How would you go about determining whether this annotation is a valid one or not?

Task 1 was a complicated task, which we broke down into two sub-tasks: finding the annotation history and modifying the gene annotation. For the first task, we recorded start-time, end-time, and errors, while for the annotation task we recorded start time, end time, errors, and other resources referenced. For tasks 2 through 5, we recorded start time, end time, and any errors or useful comments. Task 6 was more of an open-ended, qualitative exercise, with variable outcomes, and so such quantitative data would be irrelevant. We instead observed where in the site the participants navigated in order to answer the question. We hoped to determine the relative importance of our collaboration options, hypothesizing that users would favor navigating to the annotation history.

Procedure

A complete script for this usability study can be found in the Appendix of this report.

1. To begin, we briefed participants on the purpose of the study, the think-aloud process, the value of their honest input, and their freedom to quit at any time. After having them review and sign an informed consent form, we thanked them for participating.
2. We explained that the test would be conducted with a prototype, noted the fact that not all data in it was real, and explained how it was intended to integrate into the existing IMG system. We showed them how to get to the initial “start page” and asked them to bookmark it.
3. We gave each participant a printed list of tasks and read aloud each task as they worked through the list. While the participants worked, we observed and recorded qualitative and quantitative metrics of their performance. We interrupted them only if they became lost within the existing IMG system.
4. We went through the list of qualitative debriefing questions.
5. Finally, we gave them a quick survey to collect basic demographic information.

Screenshots of this prototype