A user centered approach to designing, building,
  and implementing a

Digital Asset Management System

for the San Francisco Museum of Modern Art
      
Thoreau Lovell
Margo Dunlap
Joanna Plattner

IS213 
Spring 2001
       

 

home

Heuristic Evaluation of the Content Management System

 

The subject of this heuristic evaluation is a Content Management System (CMS). After reading and synthesizing the problem statement, task analysis, and scenarios that have been prepared, we believe we have a good understanding of the system's intended functionality. At its most basic level, the CMS is a web interface to a specialized data repository. More specifically, the repository is a backend database tailored to support the creation and management of html formatted text, images, and html style sheets for distribution over the internet or some other computer network protocol (wap).

The purpose of this evaluation is to identify usability violations in the CMS First Interactive Prototype. The evaluation report includes the following:

 

Methodology:

This evaluation was conducted in two parts: individual assesment and group evaluation. The following description represents the steps of individual assessments. First, project documentation was reviewed which included the First Interactive description, project vocabulary, and scenarios. Second, the interactive prototype was reviewed. We made note of "planned" but not yet implemented design changes such as adding more instruction to each page and replacing the term "document" with "text". Next, we went through the interface more carefully in search of specific heuristic violations. Usability problems were identified according to Nielsen's 10 point heuristic guideline scheme. Severity ratings were applied to each violation.

Individual assessments were evaluated in a group discussion. The following documentation summarizes the group evaluation.

 

Summary Findings & Recommendations:


Conceptually, this prototype might work in the context of a very small web site with just one or two contributors. However, a large organization with complex information publishing requirements would find it inadequate due to following key interaction problems.

1. Document management convention dictates that a standard set of metadata about an object should be visible. For example, who created the objects, when? Who edited them, when? Add histories and details about how items are used. Without such information, users can experience a number of difficulties such as not being able to locate desired items or unwittingly modifying or deleting objects other people are using because they share the same name.

2. A versioning or archiving feature is critical for a complex web site, especially one with many contributors. What if I accidentally write over the home page and it accidentally gets authorized by the editor? All versions of published content should be archived and accessible via the CMS system.

3. Another mandatory feature that is not present in this prototype is the ability to search for documents across a variety of attributes such as text, title, author, date, location, etc.

4. There is a significant Gulf of Evalution problem with this design. There is no feedback regarding the impact of modifications made through the system. Users, whose actions could affect thousands of published web pages, need to know how their changes will affect the site. Will their new/modified content fit on a particular device's display? How many web pages will be effected? Users should be able to answer the following question: "were my intentions realized?" A preview feature would be ideal, but at minimum upon "save" users should be told which web pages will be affected upon authorization by the editor.

5. There is no visible way to cancel or gracefully exit from the editing process. Users can log out, but not cancel, or clear modifications. Add functionality that allows users to undo previous actions.

6. The error messages (both javascript and cold fusion) are incomprehensible. Errors encountered on nearly every interactive screen. If possible, hide the details and display custom messages that offer recovery instructions.

 

 

Summary of Problems by Heuristic Violation:
Violation
Total
H1 Visibility of system status
11
H2 Match system to real world
9
H3 User control and freedom
2
H4 Consistency and standards
6
H5 Error prevention
2
H6 Recognition and recall
3
H7 Flexibility and efficiency of use
1
H8 Aesthetics and minimalist design
0
H9 Error recovery
1
H10 Help and documentation
3

 

Summary of Problems by Severity:
Severity
Total
0
0
1
1
2
13
3
13
4
12

 

Compiled Heuristic Evaluation Worksheet
#
Severity
Violation
Problem
Evaluators
1
2
[H 1 ] Visibility of System status
 Login- No description of system
1
2
3
[H 10] Documentation and Help
 Login-No help with login password.
1
3
2
[H 2] Match system to real world

"Editor's Console" confusing vocabulary. All users are editing content.

One of the choices on the persistent menu bar is "editors consol". This option should be relabeled to reflect the tasks it supports, rather than the "role" of editor. Users with editor privileges, such as the HR administrator in scenario 4, may not consider themselves to be "editors" and therefore may not think to select that menu option. Suggested labels: "authorization console" or "make public console".

1
4
2
[H 2]Match system to real world
 Why have to go through category before selecting file. Seems like an extra step. Files could be in directory.
1
5
1
[H 4]Consistency and Standards

Documents--Inconsistent action link for 3. Delete document. 1 & 2 have file selection, then button.

The links on the main page clearly marked as hyperlinks. Users shouldn't have to guess where links are on a web page. The design should follow convention by underlining links.
1
6
2
[H 4] Consistency and Standards
 Documents--Inconsistent operation. verb-noun, instead of noun-verb in 1 & 2.
1
7
2
[H 2] Match system to real world
 1, 2 & 3 Seem like steps, not separate operations.
1
8
3

H2 Match system to real world

[H 5] Error Prevention

 

 Danger of overwrite. What are the standards for creating new documents rather than modifying existing ones? Possibility for error in deleting relevant doc information by modifying document content.
2
9
2
[H 5] Error Prevention
 Text entry editing screen--slow processing. Text does not appear as user types. Creates errors. Doesn't meet user expectation.
3
10
3
[H 9] Error recovery
 Icons on editing page return errors-"Web page not found" --no instruction.
2
11
3
[H 4] Consistency and Standards
 Ezedit introduces interaction design problems that the content management team may not have any control over. For example, after "rolling over" the unconventional icon on the lower left of the edit page that looks like a magnifying class on a piece of paper, a pop-up note revealed that it was a "Edit Source" icon. After selecting it I realized that it is toggle that hides/shows html tags. At the very least, the CMS system should include a guide to using the Ezedit system so that all unconventional, nonstandard features are accessible to users who are unfamiliar with them.
1
12
2
[H 4] Consistency and Standards

Inconsistent rollover titles on menu icons

1
13
2
[H 2] Match system to real world

Confusion over Make it Live and Searchable-terminology "Searchable" not clear.

1
14
4
[H 1]Visibility of System status

The "make it live" and "make it searchable" check boxes exist both in the ezedit window, and at the foot of the "editor" page (and the modify document page - see below). It isn't clear if the check boxes control the same function. Confusing which to use.

2
15
4
[H 1] Visibility of sytem status
 After saving document user recieves "Congratulations you've added a document" AND graphic/stylesheet selections. Does user have to select graphic or stylesheet?
2
16
2
[H 2] Match system to real world
 No apparent default stylesheet
2
17
4
[H 1] Visibility of sytem status
 Unclear feedback. After adding document, user recieves "Congratulations you've added a document" when is it really added? When I save it initially, or after I select stylesheets and graphics?
2
18
3
[H 6] Recognition rather than recall
 pageID-do I have to re-type the name of the file I just created?
1
19
4
[H 2] Match system to real world
 Stylesheets -don't know how to use stylesheets in order to complete scenario 3.
2
20
      
21
2
[H 4] Consistency and Standards
 Editor's console: does not go select category -then document like in scenario 1.
1
22
3
[H 2] Match system to real world
 System driven language-WAP, FLASH-Don't know what exactly to select
2
23
4

H6 Recognition rather than recall

 After selecting any of the primary options (Documents / Style sheets / Graphics) from "Main," it is unclear what you are looking at. Take "Style Sheets," for instance. It would be very helpful to know what pages would be affected if a style sheet is modified. This same violation is present in "Documents," and "Graphics." At the very least the CMS should report whether an object is in use on an active site. Without knowing what pages are using a particular style sheet it's possible that undesirable and unintended changes could be made to the site.
1
24
4
H2 Match between system and real world
 Not enough information is given about any of the objects accessible through the CMS. Who created the objects, when? Who's edited them, when? Convention generally dictates that a standard set of information about an object should be visible, such as indicated in the description of this violation. Without such information, users can have difficulty locating desired objects and may unwittingly modify or delete objects other people are using.
2
25
4
H1 Visibility of sytem status
 It's unclear how much of what a user sees is based on his / her login. Nor is it clear who can edit what and what the approval process is for style sheets and graphics. Some kind of "Content Management Flow Chart" on "Main" would help. A user should be able to indicate where a document, graphic and style sheet should be used. For instance, a user should be able to say "take this text, format it with this style sheet, and use it on this web page."
1
26
4
H1 Visibility of sytem status

No preview options to show the user how the text, graphic, style sheet will look. Without a preview function users are unlikely to have confidence in the system.

The interface doesn't explicitly instruct users on how to "view" content. Some users might find the listed options (add, modify, delete), intimidating.

It appears that documents must be opened in order to authorize them for publication. This is cumbersome and would be tedious for an editor in cases where dozens of documents need authorization. At minimum, there should be an auto-preview feature (similar to e-mail) so that editors can see the content without "opening" the document.

3
27
3
H10 Help and documentation
 It's clear that the CMS group intends to add more help and documentation features. A few specific suggestions: indicate how many characters of text can be used in specific places on specific pages and indicate the acceptable size range for graphics. Some explanation of style sheets (see below) would also be critical. It would be very frustrating for users (and damaging to Lobotech) to have documents truncated because they were too long.
3
28
3
H4 Consistency and standards
 [This problem is referring to standards in the sense of adhering to external standards.] How does a user know if his or her style sheet conforms to Lobotech standards? Is there any validation of style sheets? Most users are likely to be intimidated by Style Sheets. Other users may try to use the add new style sheet option to add text. Once you click delete (doc, graphic, style sheet) there's no way to back out, no "Are You Sure?" message. Might result in accidental deletions.
1
29
2
H2 Match between system and real world
 It's clear that Editor's Console is still in development. Here a few suggestions on making it conform to real world conventions: Editors should be able to make suggestions for changes and they should be able to reject documents. Both actions should result in a message being sent to the author. They should also be able to assign a text to a specific location on the site.
1
30

4

 
H1 Visibility of system status
 A user should be able to indicate where a document, graphic and style sheet should be used. For instance, a user should be able to say "take this text, format it with this style sheet, and use it on this web page." The user isn't given enough information to know how the work that they are doing in the CMS will ultimately be used.
1
31
3
H3 User control and freedom

No undo; back; or edit. For instance, once you click delete (doc, graphic, style sheet), no way to back out.

There should be a means for users to "undo" or "cancel" "publish to the web" commands.
3
32
4
H10 Help & documentation
 There is no help or documentation at this time.
1
33
3
H1 visibility of system status
 It isn't clear that users are creating text marked up in html when they use ezedit. This essential characteristic is completely obscured until the "edit source" icon is clicked. Users may not need to see the HTML that is embedded in their documents, but they should be informed that HTML is being generated.
1
34
2
H1 visibility of system status
 The commands (submit, make it live) on the edit page aren't visible until I have scrolled down to the bottom.
1
35
3
H1 visibility of system status
 After selecting "add, modify, delete document" only two of the three options presented on the subsequent page were visible on my notebook computer's screen. The third option was "below the fold". Although the heading said "you have three options", I still missed it.
1
36
2
H1 visibility of system status
 After selecting the "editors console" link, there is no confirmation that I am in the right place. The page doesn't have any sort of title or heading confirming that I successfully reached my destination.
1
37
4
H3 User control & freedom
 There is no search function in this interface. This is a standard information access tool that users expect, particularly in a database application. This violation could also be classified under "recognition not recall""
1
38
4
H5 Error prevention
On the "modify document" page where documented are created or edited, users are given the option of "make it live", which I thought was only accessible to editors. This may or may not be an error, but if it is I assume that selecting the checkbox would lead to an avoidable error message along the lines of "you don't have the authority to publish to the web".
1
39
3
H6 Recognition over recall
 During the modify/edit document process there is no indication of the "category" of my document (About us) in the case of scenario 1.
1
40
3
H7 Flexibility & efficiency of use
 The interface does not provide accelerators or short cuts for experienced users.
1

 

 

 

 

 

 

 

Individual Heuristic Evaluations