Response to Heuristic Evaluation

Following is the list of usability issues found by our heuristic evaluators, along with explanations of how we addressed each one or why we did not.

1. [H1 Visibility of system status] (Severity 1, Found By: 2)
On the Gene Details page the "Loading" text in the upper right-hand corner of the page seems slightly redundant with the browser's own loading feedback. Not really a problem, but it might be nice to have the word "Loaded" removed after the page is loaded. Users who missed the beginning loading, might wonder what has actually been loaded. "Loaded" doesn't convey much information. Too informal?

We chose not to address this, because the "Loading" text is part of the existing IMG system, used throughout to deal with extremely slow-loading pages, and is beyond the scope of this project.

2. [H2 Match between system and the real world] (Severity 2, Found By: 2)
On the Gene Details page, the text "V.1.1 and V1.0" exists. It is unclear to me what the linked V1.0 does, or why the V1.1 is not linked. (the extra white space before V1.0 is also linked) Maybe to a targeted user, this link is clearer. It is not clear which one is active.

We reworked the versioning indications a little bit, making current and previous version numbers function as links. We also added the current version number in the heading for the functional annotation area, to clarify which version is current.

3. [H4 Consistency & Standards] (Severity 2, Found By: 2)
Under "Discussion" the user name is linked. I expected it to bring me to a page that had details about the user, much as clicking on a gene brought me more details about the gene. I was not expecting it to be a mailto link. Maybe if there is no info about that user, a small "envelope icon" might make the functionality more clear. For many users, mailto links cause work flow disruptions when a new application is launched.

We decided to use an envelope icon for the link to the email. Also, we are now using real names where we attribute an annotation to a specific user, to make them more recognizable to colleagues.

4. [H2 Match between system and the real world] (Severity 2, Found By: 3)
On the Gene Details page, under "discussion," there is a "More" link. I expected this link to show me more of the text of the message shown. It instead drops me to a full Annotation History page and I have to scan the page to find the quote that is begun on the Details page. Oddly, I can't even find the quote shown. Perhaps it is just hard-coded for this prototype. There should be a value to indicate how many "more" comments to view. Users will know if the annotation is "hot."

The "more" link became "More Discussion", and we are now including the entire text of the most recent discussion comment. The "More Discussion" link only shows up when there is more than one discussion comment for the current version of the annotation.

5. [H4 Consistency & Standards] (Severity 2, Found By: 2)
The Gene Details page has two buttons for submitting opinion: "I Agree", and "I Disagree". They are both red, which is troubling since, to me, that seems to indicate they are both negative feedback. Maybe "I Agree" should be green.

We changed to a more neutral color, brown.

6. [H1 Visibility of system status] (Severity 3, Found By: 4)
When clicked, both the "I Agree" and "I Disagree" buttons stay blue (highlighted) even after the results have been "saved." They should revert back to normal state to avoid giving the impression the system has failed to complete the save. I would suggest instead you "gray out" the button that was clicked, to indicate you can't agree (or disagree) twice. Maybe a set of 3 radio buttons, with one being "No Opinion." Votes get submited as soon as the user changes the vote off of the default "No Opinion."

The state changes to the voting buttons have been altered so that the color change only occurs after the save operation has completed. So, the text changes to "saving..." to indicate that the user's input has been received, and then it changes back to "I agree" or "I disagree" but with the alternate button color, so that users can still see what they picked but have an indication that the system state changed to reflect their vote. We do not agree with the idea of graying out the buttons, as they can still be used to correct a vote. We chose not to use radio buttons, as their expected behavior goes against the immediate action associated with these buttons.

7. [H4 Consistency & Standards] (Severity 2, Found By: 2)
"Update Annotation" and "Discuss" buttons have slightly different windowing behavior. It looks like "Update" brings up a new window, while "Discuss" redirects the current one. I would suggest making them consistent, or at the very least, change the frame and look of the new window. Make it clear that it is a "dialog" rather than the new window of interaction. I tend to find myself using the new window to finish my work flow and forget about the parent window from which spawned. Generally windowing is behavior inconsistent.

We have made the pages that appear in the popup visually distinct from the rest of the site by simplifying them, removing the irrelevant content above the IMG logo and removing the potentially misleading navigation bar.

8. [H2 Match between system and the real world] (Severity 1, Found By: 1)
I find the phrase "0 of 1 voters agreed with this annotation" cumbersome. Maybe it would better be served with "Agreed: 10, Disagreed: 30, Total: 40" or something like that.

We changed the wording of the voting information to "Votes on this annotation: 1 agreed, 2 disagreed".

9. [H4 Consistency & Standards] (Severity 1, Found By: 1)
On the "Annotation" page, the use of colons is inconsistent. The first set of questions has none, but the "Evidence" set mostly uses them. I suggest you use them for all fields, even checkboxes, as ungrammatical as that seems.

We prefer no colons, but we agreed that this should be consistent and changed the page accordingly.

10. [H4 Consistency & Standards] (Severity 0, Found By: 1)
On the Annotation page, change "Comment (will be..." to "Comments (will be..." as most users think of comments in the plural.

We disagree with the assertion that users do not think of an individual comment as singular. In addition, we want to discourage posting diatribes in the discussion area, so we chose to leave this as "Comment".

11. [H4 Consistency & Standards] (Severity 1, Found By: 1)
At the end of the Annotation page, two checkboxes exist. They break the pattern of label first and then input field, as established earlier on the page. On the second field, "Notify me via Email of discussion and changes," capitalization should be kept consistent with the label above it, either title-case it, or convert both to normal case style. Example: "Notify Me Via Email of Discussions and Changes."

We don't feel that the checkboxes at the bottom need the same format as the annotation data, as they are a preference setting. We continue to capitalize My Watched Annotations, as that is a named page in our system. We did change the case of "email".

12. [H3 User control and freedom] (Severity 3, Found By: 1)
From the Gene Cart, if you choose to "Discuss Selected" one is brought to a discussion page. On this page, there is a cancel button which does nothing. There should be a clear way to get back to the Gene Cart if the user decides not to discuss those genes.

This was a bug that clearly needed fixing. The Cancel button now takes one back to the Gene Cart.

13. [H2 Match between system and the real world] (Severity 3, Found By: 2)
Discuss Selected Genes and Gene Cart page lists a set of "Gene Object IDs." This might make more sense to a target user, but I am guessing this is an example of the code peeking through into the UI. If the ID is needed, maybe the Discuss page should list the descriptions as well. Otherwise the user must remember the IDs and be confident they are the ones they wish to comment on.

We chose not to do this, because people like our metagenomics user, Martha, could select hundreds of genes for a specific note, and that would be cumbersome. The list is there to remind the user that they are discussing multiple genes.

14. [H3 User control and freedom] (Severity 4, Found By: 3)
"Discuss Selected Genes" allows you to submit your comment, but after you do so, you are brought to a page that only lets you "close the window." This button didn't actually work, but nor did I want it to as this was the only window I had open. It does work if you get to that page through the Update Discussion page, at which point it really does close your browser window! The user's only escape is the browser back button.

“Close the window” was changed to “Back to Gene Cart,” with the associated behaviour of bringing the participant back to the gene cart.

15. [H4 Consistency & Standards] (Severity 1, Found By: 2)
The Annotation page refers to the free-form text areas as "Comments," where as elsewhere they are noted as "Discussions." If these are the same thing, they probably should be consistently referred to as one or the other.

We chose not to change this label, as we deem "Comment (will be added to discussion about this annotation)" sufficiently clear. Changing the word "comment" to "discussion" would create a redundancy and reflect less well what the user is expected to enter. Removing the parenthetical phrase would make the label less informative.

16. [H8 Aesthetic and minimalist design] (Severity 2, Found By: 3)
The "Discuss This Annotation" page, and elsewhere, includes a full time stamp of when the annotation was made. It is probably gratuitous to include the time full information. At least the "seconds" are unlikely to be useful and it clutters the display. I would suggest visually separating the timestamp, author, and comments a little more. A slight use of color or a gestalt grouping might be enough to make this page easier to read. Long comments could quickly become a long jumble of text otherwise. I am not sure what the "-0800" code is for, but it is unlikely to mean much to the user.

All of these are valid suggestions. Problems arise from the fact that our initial design output data in the default format, listing time relative to GMT. We have implemented all the suggestions above. The current comments look like this:

Test User (member), 2006/04/19 15:27: "hello!"

17. [H8 Aesthetic and minimalist design] (Severity 1, Found By: 1)
On the list of current annotations, the green flag denoting that an annotation has changed in the last seven days should be placed next to the checkbox instead of at the end of the line. This will yield a cleaner, less cluttered look as well as reducing the amount of time a user spends on scanning the line for information.

We accept this proposal.

= Annotation updated in the last seven days.

1520850 Hypothetical protein X1 [Escherichia coli O6 CFT073]
1482290 Hypothetical protein c0002 [Escherichia coli O6 CFT073]
1482340 Hypothetical protein yaaX [Escherichia coli O6 CFT073]
1482350 Hypothetical protein c0008 [Escherichia coli O6 CFT073]

The Gene List page, originally of the style above, was modified to follow the style below.

= Annotation updated in the last seven days.

18. [H2 Match between system and the real world] (Severity 4, Found By: 2)
It is not clear that the Homolog Selection dropbox is functionally related to everything below it (including the 'Add Selections to Gene Cart', etc. buttons). Make it more obvious that everything including and below the Homolog Selection is in one group.

This may indeed be a catastrophe; however, it is a part of the exisitng IMG system that has no relationship with our project and hence lies completely outside of our scope.

19. [H1 Visibility of system status] (Severity 1, Found By: 1)
Buttons under the 'Functional Annotation' column are too close together and cluttered. It may be easy for the user to accidentally click on the the wrong button (e.g. Clicking on Agree when the intended action is to click on Update). Additionally, the use of a red highlight on the Agree button may be a source of ambiguity.

The "I Agree"/"I Disagree" buttons were decreased in size and spacially grouped closer to the rating statement, "Votes on this annotation: X agreed, X disagreed." This makes the association between clicking on those buttons and the change in rating even more obvious. Additionally, the buttons were re-done in a more neutral tone, as shown in figure 19.1 and 19.2

Figure 19.1 Before HE 19

Figure 19.2 After HE 19

20. [H5 Error prevention] (Severity 1, Found By: 2)
The separate sections (Export Genes, Sequence Alignments, etc.) should be "Gestalted" together better. Especially since the buttons resemble tabs in many ways, users may erroneously interpret the buttons as the headers of each section.

Similiar to HE 18, this lies outside the scope of our project.

21. [H1 Visibility of system status] (Severity 3, Found By: 2)
The layout of each page makes it very hard to understand. It seems like there's no order, or, if there is an order, no distinction between sections. The buttons and the headings seem to act as section delimiters, but it's really hard to perceive the different groups. It seems like this would be easy to fix, adding horizontal lines or distinguishing the section headings.

Again, this is a problem with the existing IMG system, particularly on the Gene Details page. We have attempted to make the grouping of our portion of that page more clear by making it compact and providing white space below it. We also introduced new buttons that look less like tabs. Addressing this in the rest of the IMG site is beyond the scope of our project.

22. [H2 Match between system and the real world], [H4 Consistency and standards ] (Severity 2, Found By: 2)
Batting Average is confusing, seems unrelated to gene annotation and is plausibly unprofessional. Additionally (agrees/votes) is in ratio while 0.788 is in decimal format. Most websites today use visual scales or rating systems to indicate approval.

Based on feedback from users, we believe that visual scales or rating systems are an inappropriate medium to address a scientific community. We have made every attempt, throughout this site, to steer clear of cartoony, patronizing, or misleading representations of data. Admittedly, "batting average" is culture-specific. We have accordingly replaced all mention of this metric with " track record:", followed by percent agreement. So, for example, a batting average of 0.5 would become "track record: 50% agreement."

23. [H8 Aesthetic and minimalist design] (Severity 0, found By: 2)
"Evidence" table row is highlighted at the bottom of the first table on the functional decomposition table.

We have evaluated this suggestion and decided to leave the background as is: the evidence is one of the most important elements in evaluting an annotation and we suspect that our users will appreciate the ease of finding it.

24. [H1 Visibility of system status] (Severity 3, Found By: 3)
In general, it was difficult to perceive location within the greater website; we were often lost and did not know how to return to previous pages without using the BACK button.

Unfortunately, this is a navigation problem with the IMG system. Currently, only one section of the site employs "you are here" indicators in the submenus. We have decided to equip our pages with these indicators as well. We also decided to strip the "Update Annotation" pop-window to the bare minimum to make sure that it looks different from the main parent window and could not be navigated away from by accident.

25. [H1 Visibility of system status] (Severity 1, Found By: 1)
There was an error message in the "Ortholog Clusters" section. Since this section is below the initial view of the screen and possibly not in the control of the current designers they should atleast check for errors and make a notation on the top of the page in plain view of the user.

We don't believe it's a problem considering the scope of our project. The "Ortholog Clusters" section is drawn from the IMG system. However, in order to provide the ortholog cluster data to users in testing, we have corrected this problem.

26. [H8 Aesthetic and minimalist design] (Severity 1, Found By: 1)
The "your log in information is not valid" message is poorly aligned under the menu options; it is only in partial view.

The login messages have been adjusted to appear as intended on testing systems.

27. [H4 Consistency & Standards] (Severity 3, Found By: 1)
The buttons in the interface do not appear clickable. There is no mouse over event or change in mouse cursor to indicate clickability. This is especially true on the Gene Cart page where the blue or red outline does not suffice to show button or link behavior exists.

We have redesigned the buttons. They are now elevated and appear clickable.

28. [H5 Error prevention] (Severity 1, Found By: 1)
You can select all the genes and still click Select All. Instead, the button should become disabled if the action has already been completed and become activated again when it makes sense.

We do not feel that the suggested behavior is consistent with user expectations of web sites. The page provides a "clear all" button next to the "select all" button, and clicking either one leaves the page in the desired state.

29. [H4 Consistency & Standards] (Severity 1, Found By: 1)
The wording "Add to My Watched Annotations" is not consistent with "MyAnnotations."

"My Watched Annotations" is a subsection in the "My Annotations" section. We can see how it might not be very apparent at first where to go to find "My Watched Annotations", but as the users learn about this new section we hope that they will find this placement logical.

30. [H2 Match between system and the real world] (Severity 3, Found By: 1)
It seems that the annotation rating is very important. Therefore, I think that the score (batting average or whatever scale is used) should be shown prominently at the top of the page. Currently, it is very small and hard to pick out. Also, it might be helpful to show it when the genes are listed in a table.

We disagree strongly. The annotation rating concept has not been tested yet. We cannot know if it might skew user perceptions of annotation credibility if it is given excessive prominence. "Does voting work?" still remains a question that we believe could be truly answered through the use of the system. We have decided to leave the rating as is—displayed along with the annotator name (following the proximity principle). We do plan to seek additional user feedback on this issue in the consequent round of user testing.

31. [H4 Consistency & Standards] (Severity 1, Found By: 1)
There is a combination of serif and sans-serif fonts on the page. These should be made consistent as the difference does not seem to serve any real purpose.

We do not agree with this comment: serif and san-serif fonts serve a purpose. (Headings are sans-serif bold, content is serif font).