Evaluators
Mary Hodder
Jeff Towle
Dan Perkel
Heuristics used
Individually, Dan and Mary used Nielsen's first version of the heuristics.
Jeff used the version updated in 1994. They are similar. As a group, we decided
to go with using the updated version rather than the originals.
Nielsen's updated
heuristics
| H1 |
Visibility of system status |
The system should always keep users informed about what is going on, through
appropriate feedback within reasonable time. |
| H2 |
Match between the system and the real world |
The system should speak the users' language, with words, phrases and concepts
familiar to the user, rather than system-oriented terms. Follow real-world
conventions, making information appear in a natural and logical order. |
| H3 |
User control and freedom |
Users often choose system functions by mistake and will need a clearly
marked "emergency exit" to leave the unwanted state without having
to go through an extended dialogue. Support undo and redo. |
| H4 |
Consistency and standards |
Users should not have to wonder whether different words, situations, or
actions mean the same thing. Follow platform conventions. |
| H5 |
Error prevention |
Even better than good error messages is a careful design which prevents
a problem from occurring in the first place. |
| H6 |
Recognition rather than recall |
Make objects, actions, and options visible. The user should not have to
remember information from one part of the dialogue to another. Instructions
for use of the system should be visible or easily retrievable whenever appropriate. |
| H7 |
Flexibility and efficiency of use |
Accelerators -- unseen by the novice user -- may often speed up the interaction
for the expert user such that the system can cater to both inexperienced
and experienced users. Allow users to tailor frequent actions. |
| H8 |
Aesthetic and minimalist design |
Dialogues should not contain information which is irrelevant or rarely
needed. Every extra unit of information in a dialogue competes with the
relevant units of information and diminishes their relative visibility. |
| H9 |
Help users recognize, diagnose, and recover from errors |
Error messages should be expressed in plain language (no codes), precisely
indicate the problem, and constructively suggest a solution. |
| H10 |
Help and documentation |
Even though it is better if the system can be used without documentation,
it may be necessary to provide help and documentation. Any such information
should be easy to search, focused on the user's task, list concrete steps
to be carried out, and not be too large. |
Severity ratings
| 0 |
Don't think this is a usability problem |
| 1 |
Cosmetic problem |
| 2 |
Minor usability problem |
| 3 |
Major usability problem |
| 4 |
Usability catastrophe |
Results of evaluation
Note on results format
We realize that this report will not present the results in the requested format.
We agreed as a group that the best way to present the results would be to title
each problem with a name, then note which heuristic is violated and the severity
of the violation. Past experience with the use of heuristic evaluations and
also best practices with bug reports influenced this decision. We were concerned
that naming each violation by the heuristic it violates would make it difficult
to scan the list, as many would have the same name (i.e.: Consistency and standards).
The heuristics have been grouped by area of the prototype and sorted internally
by severity.
Total number of violations: 35 violations (but 3 of them violated
two heuristics)
Broken down by heuristic violated:
| H1 |
Visibility of system status |
5 |
| H2 |
Match between the system and the real world |
9 |
| H3 |
User control and freedom |
1 |
| H4 |
Consistency and standards |
15 |
| H5 |
Error prevention |
3 |
| H6 |
Recognition rather than recall |
5 |
| H7 |
Flexibility and efficiency of use |
1 |
| H8 |
Aesthetic and minimalist design |
0 |
| H9 |
Help users recognized, diagnose, and recover from errors |
0 |
| H10 |
Help and documentation |
1 |
Broken down by severity of violation:
| 0 |
Don't think this is a usability problem |
1 |
| 1 |
Cosmetic problem |
8 |
| 2 |
Minor usability problem |
17 |
| 3 |
Major usability problem |
9 |
| 4 |
Usability catastrophe |
0 |
Violations in Filters/Search conditions
- Users may be unable to remove a date with only an upper bound
Heuristic Violated: H4 - Consistency and standards
Severity: 3
The “X” button in the date search condition is only shown on one
of the cells (lower bounds). If you do a search with an only an upper bound,
the x-box does not show.
# of evaluators: 1
- “Institution” is the only screen that narrows the search
with links
Heuristic Violated: H4 - Consistency and standards
Severity: 3
The two other formats are free-text and checkboxes. It does not seem to allow
a user to select multiple institutions, as the advanced search allows a user
to select multiple options (i.e., sex).
# of evaluators: 2
- Date input does not specify date format
Heuristic Violated: H6 - Recognition rather than recall
Severity: 3
Date does not specify date format. A look at the data reveals that it is in
years. There is a 4-digit limit, but it will not accept 2 digit years.
# of evaluators: 1
- Relationship between widgets and actions on "Advanced"
tab is unclear
Heuristic Violated: H5 - Error Prevention
Severity: 3
There is only a single button on the “Advanced” tab, and it specifically
searches for collectors. Since it is not clear how one should submit an update
to the sex or preparation criteria, it is very likely that a user will click
that button by mistake. How would I apply the Sex and Preparation filters?
# of evaluators: 2
- Location finder input and output is inconsistent
Heuristic Violated: H4 - Consistency and standards
Severity: 2
Location finder is by country according to the label. But you can input state
information. Display is by state/county which sort of makes sense if you know
you're seeing one country.
# of evaluators: 1
- Left bar (controls) relationship to top bar (criteria) is unclear
Heuristic Violated: H2 - Match between the system and the real world;
H4 - Consistency and standards
Severity: 3
The relationship between the top search conditions and the left nav bar was
really unclear at first. It appears to be a navigation bar at first. Which
it isn't. There should have been more of a graphical relationship between
the links at the top and the stuff on the left. This includes layout (proximity
issues) and color (the links were grouped with the logo which indicated nav
bar to me).
# of evaluators: 2
- "Lasiurus" and "Lasiurus " result in different
results (as do "California" and "California ")
Heuristic Violated: H5 - Error prevention
Severity: 3
A space after search criteria results in erroneous results. Some error prevention
such as trimming of white space would help. Especially if you're copying and
pasting terms into the search box.
# of evaluators: 1
- Advanced link is inconsistent with others
Heuristic Violated: H2 - Match between the system and the
real world
Severity: 1
I was confused as to what the "Advanced" filters were.
# of evaluators: 1
- "Advanced" Search Condition houses multiple filters
Heuristic Violated: H4 - Consistency and standards
Severity: 2
This panel is the only one to house multiple filters.
# of evaluators: 1
- It is not clear what is meant by a sex of “null”
Heuristic Violated: H2 - Match between system and the real
world
Severity: 2
In the advanced filter section, there are four options for "Sex."
The term "null" is likely to be unfamiliar to users who are not
computer scientists.
# of evaluators: 1
- Inconsistent filtering on null entries
Heuristic Violated: H4 - Consistency and standards
Severity: 2
Some filters (sex) allow include null values. It is not clear how to include
null values in other searches, for example, the date search.
# of evaluators: 1
- User is forced to remember free text input format
Heuristic Violated: H6 - Recognition rather than recall
Severity: 2
The user is forced to remember complex rules about inputting free text. For
example, it is not clear whether a partial name will be accepted, how Boolean
operators will be handled, or if exact case matches are required.
# of evaluators: 1
- The “Click Map To Search by Country” graphic may be
confusing
Heuristic Violated: H4 - Consistency and standards
Severity: 2
In the rest of the areas, all search controls are confined to the left sidebar.
Is the user required to select a country directly from the tiny map? Try to
find, say, Nepal. Or will the map open up in the center/content area. In the
other searches, the center area continues to display results, so it would
be confusing if a larger map opened up in this area.
# of evaluators: 2
- Location search label/prompt is misleading
Heuristic Violated: H2 - Match between system and the real world
Severity: 2
The search box has a graphic that urges the curser to search by country, however,
no results are returned for US, even though many (or all?) of the entries
are located in the US. It looks like the search box only takes states.
# of evaluators: 1
- "Search Condition" Links
Heuristic Violated: H4 - Consistency and standards
Severity: 2
While it is not really a universal standard to have navigation links across
the top (usually top or left), the converse is true: if I see links at the
top, I'm assume they are navigation. Especially if the shading/coloring separates
it from the content and other elements of the screen. I didn't even see the
title "Search Conditions" until much later in the evaluation.
# of evaluators: 1
- Filtering by more than one criteria can get tedious
Heuristic Violated: H7 - Flexibility and efficiency of use
Severity: 2
There should be a way to filter by more than one criteria at the same time.
Having to click a criteria, hit apply, then find another criteria and hit
apply again is time consuming. This would be frustrating to frequent users.
# of evaluators: 2
- "Find Dates" is misleading
Heuristic Violated: H2 - Match between the system and the
real world
Severity: 2
I understood what was going to happen because of the scenario, but even still,
I was confused by the button that seemed to be telling me to "find dates".
I wasn't trying to find the dates. I should note that I was not as confused
or affected with "Find locations" or "Find institutions,"
though neither really map to what's going on, either.
# of evaluators: 2
- Date fields take characters
Heuristic Violated: H5 - Error prevention
Severity: 1
It is possible to enter alpha-characters in the date fields, and the system
will accept them and display them on the "Search Conditions" panel.
# of evaluators: 1
- Use of left panel is inconsistent
Heuristic Violated: H4 - Consistency and standards
Severity: 1
Show/hide link and Export link is inconsistent with filters/search criteria
# of evaluators: 1
Violations in search results
- Show/Hide column selection does not reflect current system status
Heuristic Violated: H1 - Visibility of system status
Severity: 3
The show/hide columns area is reset after every change in the show/hide columns
area. Thus, it is possible for all boxes to be checked (show all), when only
a few columns are shown.
# of evaluators: 2
- If you sort a column, it reverts to show all columns.
Heuristics Violated: H4 - Consistency and standards, H2 -
Match between system and real world
Severity: 3
This might just be a bug, but it's still major.
# of evaluators: 2
- Sort does not specify the type of sort performed
Heuristic Violated: H6 - Recognition rather than recall
Severity: 2
Just having "sort" doesn't help the user figure out what kind of
sort will be performed or help provide feedback with regards to the sorted
order of items in the table. In summary, at a glance that user has to recall
two things: how the entire table is sorted and what the sort order is.
# of evaluators: 2
- Institution data forces recall
Heuristic Violated: H6 - Recognition rather than recall
Severity: 2
In the results table, the institution name is displayed using the initials,
so one must return to the institutions panel/filter to see which institutions
is which.
# of evaluators: 1
- Terminology on Show/Hide column selection may be confusing
Heuristic Violated: H2 - Match between the system and the real world
Severity: 2
Show/Hide columns may make sense the first time it is used (hiding columns),
but it is not clear what action hides and what action shows. Is a checked
column going to be hidden?
# of evaluators: 2
- "X" button next to "Institutions" column header
does not seem to do anything
Heuristic Violated: H1 - Visibility of system status
Severity: 2
If this button is doing something else besides being used to hide/close a
column, it's not clear what it's doing
# of evaluators: 1
- Results navigation links are unclear
Heuristic Violated: H6 - Recognition rather than recall
Severity: 2
We realize that this functionality has not yet been implemented. However,
their presentation is still unclear: they could either link to pages of results
or to anchors in the result set.
# of evaluators: 1
- The initial system state is a blank screen
Heuristics Violated: H2 - Match between system and the real
world; H1 - Visibility of system status
Severity: 2
Looking at the blank screen in the middle, it was unclear what to do. The
first page (and the page you get when you have eliminated all of the criteria)
is just blank and somewhat confusing. The mental model of the app is you apply
a series of filters to reduce a result set. So what is the start state? Furthermore,
criteria were eliminated, there were more and more results... until there
were no criteria and a blank screen appeared. This seemed inconsistent. It
was certainly jarring in terms of experience. It forced a shift in attention
into thinking why there were "no results" which wasn't accurate
# of evaluators: 1
- Advanced search results in strange behavior
Heuristics violated: H1 - Visibility of system status; H4 - Consistency
and standards
Severity: 2
When searching in the Advanced page, if there are no results, the left nav
changes colors and menu, and nothing is returned in the body of the page.
Looks like an error in the search, verses that the search had no results.
No results for other searches looks different. This may be a bug.
# of evaluators: 1
- Not all columns are shown in the show/hide columns area
Heuristic Violated: H4 - Consistency and standards
Severity: 1
This is not that big a usability problem, but it is inconsistent. We realize
that it might not make sense to eliminate those columns.
# of evaluators: 2
- "X" button next to "Institutions" column header
is confusing
Heuristic Violated: H4 - Consistency and standards
Severity: 1
It is not clear what this is supposed to do. If this is to close the column
it is inconsistent with the other column headers and the "show/hide columns"
link.
# of evaluators: 3
- Results are displayed in mixed caps
Heuristic Violated: H4 - Consistency and standards
Severity: 1
Some results are displayed in lower-case, and others are displayed in all
caps.
# of evaluators: 2
- Columns going off the page is confusing in relationship to "Show/Hide
Columns" functionality
Heuristic Violated: H2 - Match between the system and the
real world
Severity: 1
Columns that I could choose to hide weren't there on the screen because they
were off the page to the right. It was confusing trying to map a far right
column with information on the far left.
# of evaluators: 2
- Results navigation links may be confusing
Heuristic Violated: H1 - Visibility of system status
Severity: 1
All links showing result numbers are linked, indicating that they take the
user elsewhere. It appears as if the first page of results shows both a back
button, indicating you can go to a previous page.
# of evaluators: 1
- Catalog numbers are difficult to read
Heuristic Violated: H4 - Consistency and standards
Severity: 0
Catalog numbers are displayed in incomparable formats for searching. Some
start with numbers, some start with letters, but it is possible to sort them.
# of evaluators: 1
General
- No way to start a search over
Heuristic Violated: H3 - User control and freedom
Severity: 3
I wanted to start a search over from scratch and I felt that I had no options
other than trying to click on the logo. The back button seemed my only recourse.
Looking back, I realize that I could have probably eliminated search criteria
from the top, but it didn't occur to me.
# of evaluators: 3
- No help provided
Heuristic Violated: H10 - Help and documentation
Severity: 2
No help or prompts are provided to the user.
# of evaluators: 3
Individual reports
Mary's evaluation (,doc)
Jeff's evaluation
Dan's evaluation