IS213 Projects Home

Assignment 1

• Problem Statement
• Project Members

Assignment 2

• Personas
• Task Analysis
• Appendices
• Work Distribution

Assignment 3

• Scenarios
• Comparative Analyses
• Initial Design 1
• Initial Design 2
• Initial Design 3
• Work Distribution

Assignment 4

• Introduction
• Low-Fi Prototype
• Methods
• Test Measures
• Results
• Discussion
• Appendices
• Work Distribution

Assignment 5

• Design Revisions
• Revised Scenarios
• 1st Hi-Fi Prototype
• Storyboards
• Instructions for Use
• Slide Presentation
• Work Distribution

Assignment 6

• Heuristic Evaluation
• Work Distribution

Assignment 7

• 2nd Hi-Fi Prototype
• Responses to HEs
• Work Distribution
• Instructions for Use

Assignment 8

• Introduction
• Method
• Test Measures
• Results
• Discussion
• Appendices
• Work Distribution
• Slide Presentation

Final Write-up

• Problem Statement
• Solution Overview
• Final Personas
• Final Scenarios
• Revised Storyboards
• Final Interface Design
• Design Evolution
• Interactive Prototype
• Instructions for Use
• Work Distribution
• Slide Presentation

Discussion

In general, the interaction flow through the annotation area tested well with participants. The one user familiar with the IMG system performed every task with virtually no hesitation and felt that the system was very intuitive and easy to use. The two professional annotators, who had never worked with the IMG system, needed some time to get familiar with it first. In the first task, they both explored the Gene Details page a bit before clicking on the Update Annotation button. One of the annotators said he felt that the site was not intuitive, but that once he had a chance to look around he "understood the intuition of the designers" and that in that sense it was intuitive. His timings after the initial task were much quicker, though not as fast as the IMG user. The other annotator took less time with the initial task and completed other tasks very quickly. In retrospect, we concluded that we should have given users not familiar with IMG an introduction to the original system.

There were several issues with the Update page highlighted in user testing. One of these was the fact that the ontology terms were not set to show up until a user entered a change in an entry field. Users noted that these should always be present, so we plan to change this on multiple pages in the next prototype. The participants also noted that there was no explanation of how the homologs in the Compare and Transfer menu were chosen. We therefore plan to add information about the algorithm used in IMG for identifying homologs and explain that the menu excludes genes listed only as hypothetical. The homolog menu's labels were also once mistaken for selectable options for comparing and transferring. We plan to break this menu into two submenus so that each can have a complete label outside the menu and prevent user selection errors. It was also suggested that it would be helpful to be able to transfer annotation data from protein database hits, so we plan to make that available as well, possibly in a third menu. Participants also noticed small bugs in the handling of ontology ID numbers that caused them to be treated as numbers rather than text strings, truncating leading zeroes or presenting an empty string as a zero. One participant had trouble with the popup window while poking around in the system after completing the test tasks, not realizing that it was already open. We are rethinking the use of this window and at a minimum plan to make it come forward when a user attempts to open it if it is already open.

Several issues were identified on the page on which users compare and transfer annotations. This page was consistenly rated as the most useful part of the system, and users all said that it was a good way to work with this data, but there were still problems to be addressed. The radio buttons to "transfer all" data from one gene to the annotation subject were somewhat confusing. One participant wondered why the "transfer none" radio button was selected after he had transferred an item from a homolog, though he dismissed the issue and quickly completed the task correctly anyway. We now plan to change the "transfer all" buttons to "select all" buttons and position them at the top of each column. Participants noted that the table on this page should also show ontology terms to match the IDs, which we intend to add. The percentages given for annotator track records were also mistaken for percent identity scores, so we will remove the track record percentages from this table. The ratings based on voting were also found to be unclear (we accidentally failed to change them from "agrees/votes" to "x agreed, y disagreed" when we changed the format on the Gene Details page), so we will present this in a more self-explanatory format. Users also said that they would like to see evidence information in that table, so we plan to add that.

In this study, we found some interesting differences between our scientist target users and the professional annotators. During both this and the earlier study with the paper prototype, the scientists were much more happy with the existing IMG tools. The annotators mentioned things they would like to see that none of the scientists mentioned. While it is possible that we simply needed to get input from enough users for these things to come up, it is notable that both the annotators mentioned them. They wanted to see more graphical representations of alignment data, and they wanted access to the original data sources rather than relying on the data in IMG. We plan on adding links to the data sources in our revised prototype. The graphics are a larger issue that we don't feel that we have the time needed to address in the next prototype. The IMG system does provide small alignment graphics with no labels, but these are poor representations of the data and rely on user memory to know which part of the graphic represents which sequence. IMG tends to offer textual and numeric data rather than graphics generally, so we feel the numeric approach is at least consistent with the existing system.

Our testing has also led us to plan changes to the Gene Details page, most notably moving the agree/disagree buttons to the discussion page. One participant pointed out that there is a tradition in scientific discourse that calls for a scientist who expresses disagreement with another one to provide a reason for their disagreement. As we have wondered all along whether scientists would use the voting buttons in the current interface, we hope that this will provide a flow that better encourages use of the voting mechanism. Participants also were slow to recognize the meaning of the version number links on the Gene Details page, so we plan to expand them, including dates in the link text. One user noted the similarity of the sofware version number in the footer of all IMG pages to the annotation version number. We plan to address this in the next version, either by identifying annotation versions differently or by removing the software version number in the footer.

Only one issue came up related to the Gene Cart page. One user, when asked to comment on multiple genes, considered simply clicking on multiple "discuss" buttons but then thought there must be another way to comment on multiple genes. He did find the button to comment on multiple genes, however. To be consistent with the Gene Details page, we plan to remove the voting buttons here. We hope that this will simplify the interaction options so that users can more quickly scan the options available. The near use of many Discuss buttons provides an example of one tendency we noticed in participants, which was to look for already-known ways to accomplish a task if they had done a similar one before. We hope to design tasks for future studies that control for this tendency in users.

Finally, there was some confusion on the My Annotations page, because of the similarity in labels for "My Annotations" and "My Watched Annotations". We plan to split the two lists onto two different pages, titling one "Genes I've Annotated" and the other "Genes I'm Watching". One user commented that she did not know what the "My Annotations" feature was though she found it quickly, which suggests that we should have used a more neutral description of the task.

We were surprised to discover how participants addressed our experimental question about how they would go about deciding whether an annotation is valid. We thought that they would use the annotation history, the discussion area, or the voting results, and we hypothesized that the former would be favored. We couldn't have been more wrong. Every user went into analysis mode, studying the data the IMG system presents as evidence available for function prediction. This provides a little perspective on the relative importance of our collaborative widgets. In future studies, we would need to find another way of assessing the relative usefulness of the collaboration options.