Results

Overall, we were pleased with the main characteristics of the prototype. All users found the flow natural and expressed their approval of the basic functionality. Much enthusiasm was shown for the "Compare and Transfer Annotations" tool that allows one to transfer annotation information from another gene based on similarity. For full transcripts of the user sessions, please see User Transcripts.

Time to Complete Tasks 1-5 (to the nearest minute)

Task

User A

User B

User C

Task 1
(part 1 + part 2)

1 min + 3 min

2 min + 8 min

1 min + 4 min

Task 2

1 min

1 min

1 min

Task 3

3 min

2 min

1 min

Task 4

1 min

3 min

1 min

Task 5

2 min

1 min

1 min

Please note that the responses to task 6 were not timed. The intent was to determine what kind of information the users would use to determine the credibility of the annotation.

Following is a summary of observations and discussion by task:

Task 1

View the annotation history for a specified gene on the list. Modify the existing annotation for the gene using data from a homolog and submit updated annotation.

  1. All users navigated to the "Gene Details" page and then used a version link to retrieve the annotation history.
    • User A had a brief confusion over the annotation version number/system version number.
  2. Overall, all users found the "Compare and Transfer Annotations" tool very helpful and intuitive. However, the following issues were identified:
    • User A found the "Transfer Nothing" radio button in the subject column confusing. (He expected the state of the button to change to unchecked, as he had chosen data to be transfered from another column);
    • User A was not certain that the data was transfered to the "Update Annotation" page.
    • All users wanted to see descriptions next to the GO, EC and COG Group numbers.
    • User B found % agreement confusing, he wasn't sure if that was a similarity measure of % identity.
    • User B suggested that the evidence be included in the Compare and Transfer table.
    • Users B and C wanted to have a graphical representation of the homolog/subject alignment in the comparison table. (We note that the alignment is available in the Evidence for Function Prediction section of the "Gene Details" page. Also, both had never used IMG before, and both are annotators (rather than scientists) who use a different JGI annotation tool exclusively.)
    • User B suggested that an explanation of how homologs were chosen be included (e.g., best Smith-Waterman hits).
    • User B attempted to perform a selection on a label in the homolog menu.
    • User B noted that the data did not update in the parent window once the annotation was finished and the window closed. (He expected for the window to reload to display the updated annotation).
    • User B noticed that the EC number was not updating on the "Update Annotation" page upon transfer.

Task 2

Express your opposition to the annotation for the specified gene on the list.

  1. All users clicked on the "I disagree" button, some noting that they had noticed it before, almost all in a matter of seconds.
  2. While performing the task, none of the users expressed a preference to support their disagreement. (This, however, might be an effect of the controlled environment.)
    • User B commented in debriefing that he would like to be able to submit a comment explaining the nature of his disagreement, the inclusion of which he believed to be fundamental to scientific discourse.

Task 3

Change an annotation by adding a COG group. Add this gene to your list of watched annotations.

  1. All users followed the familiar steps of using the "Compare and Transfer" tool for updating a COG group. None of them attempted to simply type in a number. This could be attributed to the learning effect of the testing process.
  2. All users used the check-box to add the gene to "My Watched Annotations" on the "Update Annotation" page.
    • User C commented in debriefing that even though she had no difficulty performing the task, she wasn't sure what "My Watched Annotations" was.

Task 4

Comment on annotations for multiple genes.

  1. All users were conscious to select/deselect genes on the list using checkboxes.
  2. With the gene list being longer than the average window height, all users had to scroll to uncover the "Discuss Selected" button.
    • User B first thought of clicking on the "Discuss" button for a specific gene, but then realized it will only work for that specific gene.

Task 5

Find where you can view your watched annotations.

  1. All of the users remembered seeing "My Annotations" on the navigation bar from the previous tasks they performed.
    • User B noted the lack of distinction between "My Annotations" and "My Watched Annotations."

Task 6

How would you go about determining whether this annotation is a valid one or not?

Task 6 was designed as a way to determine what information users would need to assess the credibility of an annotation, particularly whether or not they would rely on voting results.

  1. All users attempted to perform their own analysis of evidence to determine if the annotation was a valid one.
  2. Because the experiment did not use a real gene for annotation, the users could not perform a true and complete analysis.
  3. In contrast to our expectations, none of the users gave any weight to annotation history, discussion, annotator "track record" rating or voting results.

In addition to the points outlined above, in conducting debriefing we also learned that:

  1. All users liked how the annotation functionality was resolved and specifically liked the "Compare and Transfer" tool.
  2. User A, who had prior experience with the IMG system, noted that the annotation tool was very well integrated with the existing IMG system.
  3. Users B and C, who had never used IMG before, had to spend time to get familiar with it. User B also seemed stressed about having to take the time to familiarize himself with it and therefore possibly taking too long on the first task.
  4. Aside from being performed on a hypothetical gene, our tasks approximated what users would do in performing an annotation.
  5. There was no function noticeably absent from the system.