California Digital Library

Assignment 9: Pilot User Study

[ Home | Introduction | Prototype | Method | Test Measures | Results | Discussion | Formal Design | Appendices ]

DISCUSSION

It appears that we have improved the site from the original CDL site simply by observing that the participants of this pilot study had fewer questions, were significantly less frustrated with the system and found fewer (if any) dead-ends compared with the participants of our original lo-fi design (and anecdotely compared with the users of the original CDL as well who say they find it "very difficult" to use.) However, we concede that while the demographics of the participants of the lo-fi and those of the pilot study were very similar, the scenarios tested on each set were different and may have lead to some of the frustration felt by the lo-fi participants over the pilot study users.

While we measured time to complete scenarios, for instance that Scenario 1 was completed between 2 and 3 minutes for each participant, the small sampling coupled with no real benchmark timing of any other interface does not allow us to appropriately analyze this data. It simply stands as a good initial sampling that could be tested against others in a larger scale experiment. For Scenario 1 in particular, each of the three participants of the pilot study immediately sought Melvyl, and was fairly efficient in retrieving it. In a more formal study, the demographics could be varied to cover new students not familiar with the workings of the UC library system, who would have to seek out a source to find a book on campus. Our prototype does not clearly indentify Melvyl as such a source and presumably this might cause confusion for new students.

It would also be effective in a more formal experiment to better benchmark more than one interface against another as far as time and overall satisfaction of results. Ideally, we could test a user's ability to use the current CDL or any iteration of a redesign versus our current Prototype. To avoid having the users learn the UI as they move from one UI to the other and thereby effecting the data, this formal test would probably have to be performed between groups of similar population mixes working on one UI each. This would probably garnish the least corrupt data in quantifying time and successful pathways to a source.

Our pilot study was also partially affected by a few technological difficulties. We chose to implement the Search mechanism in the CDL versus the Browse (noting that ideally we would have like to connect the two options as described in our Prototype 1 discussion.). Implementing the CDL Search mechanism in place of their Browse mechanism, disconnected the Topics selection on our Browse Sources, while Keywords was now a live selection. One participant used the Keywords search on that page, although they were fairly sure that their query would not be relevent (the result supported her assumption). In the other cases, participants used the Topics search (though it has a note saying it is not functional in this prototype), instead of the Keywords search which resulted in a second search step directly on the CDL site.

While minor errors in implementing Frames in this UI need to be fixed, overall, this proved to be very effective design change to Prototype #2 in helping the user to keep status of where they were by having the navigational bar always on the leftside, even when they scrolled the LONG lists of databases. And, also in indicating to the user that they were entering a different site when the link took them into a new browser window.

Some of the items we will look into changing in our final design:

[ Home | Introduction | Prototype | Method | Test Measures | Results | Discussion | Formal Design | Appendices ]