California Digital Library

Interface Design Project

Low Fidelity Prototyping

[ Home | Introduction | Prototype | Method | Test Measures | Results | Discussion | Appendices ]

Test Measures, Results, and Discussion

We combined the active CDL website search mechanism with a mock-up of our prototype initial web pages to the site.

• We looked for the ability of users to find their way from our front-end pages to the many resources and search or browse paths available in the CDL.
• We evaluated users' ability to select the appropriate database or resource from the CDL within which to locate their specific topic.
• We attempted to discern users' conceptions of the site structure, specifically their awareness of the need for a two-level search.
• We observed users' abiltiy to orient themselves within the different parts of the CDL and their perception of their current location relative to other parts of the site.
• We watched to see if our assumptions on more "Help" choices (Help, Site Map, Go To drop-down menus) were viable for the users.

View raw data in test logs (MS Word format):

Participant #1
Participant #2
Participant #3

Results of Post-Test Questionnaire

Clearly all of our participants would prefer the ability to search the content of the resources (or in the case of journals at least the article titles) rather than just the database title or subject headings under which the resource catalogs and databases are listed. They appeared unaware of the need to first select an appropriate database, and were repeatedly confused and frustrated by the failure of their keyword searches to function at this level. Our prototype attempted to open up the options already offered on CDL, but failed to provide a clear enough indication of the types of searches that the system would accept. Users seem far more familiar with web-based commercial search engines than any library cataloguing system, so Boolean or free text/keyword searches were sought often.

In general, participants were frequently disoriented within the searching and browsing functionality of our interface and the CDL and Melvyl interfaces. Upon entering the CDL, they were uncertain where in the search process they were, or lost track of steps they had taken--not knowing how to recover a previous system state that they remembered. Sometimes they did not realize why they were "going in circles".

With respect to our redesigned front-end pages, the participants appeared to find the high visibility of more of the CDL features useful, but found our attempt at distinctions among the Browse, Search, and Quick Search features unclear. In some cases they did not take advantage of our front-end pages as a home base for repeated exploration in the way that we had envisioned.

Even though we prominently displayed Services on the front-page, this was repeatedly ignored altogether. Even when the participants wanted to use the Profile service, they didn't intuitively think it would be available through the CDL. They went directly to Melvyl to find it.

When the system page offered a "Back" button, the participant would defer to the Browser 'Back' button instead to make sure he/she would be going to the previous page that they remember. They didn't "trust" where a system 'Back' button might take them.

The length of some of the explanatory text in the CDL site (as well as some of our screens) was indeed too long. Some of the library specific lingo (e.g., archival finding aid) was not clear. Also participants found it difficult to ascertain the difference between for example "Electronic Journals" "Mags & Journals" and "Periodical Article Database" even with our pop-up window text. They often wrongly predicted the content of selected links; moreover, even upon loading the resultant screen, they often failed to realize that it did not lead to the information they were seeking.

The success of our reworking of the CDL front-end webpages was mixed. In our Search & Browse page, participants spoke of appreciating having the formats and topics visible as multi-selectable check-boxes, but we did not observe them using this multiple-selection feature. The user path through the options we made available was highly unpredictable. We observed first-hand evidence of the observation noted in many usability readings that users often miss what the designers view as obvious paths to success or to assistance simply because they have a different mental model of their task or because those paths are one-too-many mouse-clicks away.

It was interesting to watch the participants move forward blindly. Rather than try to read a User Guide up front, it seems that all of them would rather attempt to do various searches by hacking away and repeatedly getting zero results eventually "learning" the system, but becoming very frustrated. Some looked for "help" but had mixed preconceptions that Guides and Site Map would be "useless" and therefore did not seek them out. We're not sure we can change their haste even if we offered more help upfront. Even when clues to a more appropriate path were present, they were often overlooked or ignored. The challenge will be to give context-sensitive help to them when they want it.

After the first two participants had not used Site Map, we were questioning our assumption that there should be a site map (this does not exist in the current, live CDL site.) The third participant, however, relied heavily on the Site Map, but was not getting out of it what she had expected, which was a little more detail and not just a re-hash of the page layouts. The index style was appreciated, but still did not give them any different knowledge than the other pages did.

All participants felt they would have abandoned the search at some point. Two participants explicitly stated that they would have asked a "real librarian" to help get them where they wanted to be, or even begin to get oriented..

The Melvyl search screens have more choices to refine the search than it seems CDL does, and, once there, participants stayed in this environment to search for their resource (even though, that particular resource may not have been available in the Melvyl system, but rather back out through CDL.)

What we learned:

People may ignore or dismiss significant portions of a web interface rather than exploring these options and links for assistance. In a complex interface like this one, people can easily get stuck in dead-end search paths and then may not elect to go back to the beginning to reformulate their strategy. Instead they may try to get their initial path to work by cycling around a small portion of the options available to them, with the goal of eventually learning the system. This lead to a good deal of frustration.

For the most part, users presumptions were logical (as we heard in their thinking aloud), but they missed one or two crucial bits of information or options. Some looked for "help" but had preconceptions that User Guides would be too time consuming. This brings up an interesting issue: among our participants, some had aversions to certain obvious keywords that we used, or considered using (this also came out in the post-testing discussion), to denote help-like resources, not only "guides", but also "site map", "index", and so on. This seemed to be due in part to prior bad experiences with site maps, and with the fear that an Index or Guide might be a large, unwieldy and time-consuming detour away from getting things done.

What we will change:

Browse/Search/Quicksearch options need clarification and consolidation in some form. The search functions in our interface to the CDL should be clearly labeled with examples and hints so that users will understand the limited topic/subject search that it utilizes. We will attempt to make more clear CDL's need for a two-level search, and will consider using different terminology for the initial database/resource selection, to avoid confusion with the standard Search Engine model.

The nature and contents of the Help or guidance available for such a complex and potentially maddening search space needs to be foregrounded. One difficult issue we need to study further is what to call such a function and how to structure it, in order to draw the positive attention of the largest number of users since some of the obvious candidates had negative associations among our prototype users (as discussed above). The current CDL uses a question mark icon (?), that might be able to be enlarged and better equipped with useful information for this type of interaction.

Immediate access to more guiding explanations is needed. Text may become too lengthy, so a form of icons or bulletted text needs to be addressed. One possible modification to the homepage is an explicit link/source/mouse-over pop-up that offers to give users concise aid in how "to browse [CDL resources] by broad topics or by title or to search by keyword to find electronic journals, databases, archival finding aids, and other resources." We will work to make the homepage more of an orientation and hint page for successful searches throughout the CDL system. One participant offered a suggestion for "more details" hyperlinks to bottom of page with more text IF the user wanted to get it.

What the evaluation could not tell us:

While quick, there are limitations to the experiment using lo-fi protypes. The rough mock-ups limit graphical design choices that may have helped in the participant's use of the system (e.g., we had a search button on the home page, that was not always seen right away.)

[ Home | Introduction | Prototype | Method | Test Measures | Results | Discussion | Appendices ]