Assignment 8: Pilot Usability Study

Page Contents:

Introduction
Method
Participants
Apparatus
Tasks
Procedure
Test Measures
Results
Discussion
Formal Experiment Design
Appendices
Work Distribution

Introduction

BookMarket is an online book exchange interface for SIMS students to buy or sell used textbooks. Users of BookMarket will be current and former SIMS students, other students taking SIMS courses, and SIMS faculty and staff. BookMarket supports two actions: buying and selling. The goal of the buyer is to find textbooks and to have an easy way to contact the seller to arrange the transaction. The goal of the seller is to easily and quickly post the books s/he wishes to sell. It is likely that many users will wish to both buy and sell books, and BookMarket accommodates these users by offering easy navigation between the two functions. BookMarket puts buyers and sellers in contact with each other. It does not facilitate online currency exchange; instead, it gives buyers a one-stop place to find used books for sale within the SIMS community, and it gives sellers the opportunity to post books for sale in a central location with a targeted audience.

The purpose of this pilot usability study was to observe users working with BookMarket, completing three main tasks: 1) buying a book and selling a book; 2) comparing book prices and contacting multiple sellers to bargain down their prices; and 3) posting a book for sale and then later editing the book posting. The rationale for the test was to see whether BookMarket's interaction flow is logical and the tasks are quick to complete. We also wished to see whether BookMarket's revised text effectively manages user expectations about the purpose of the system.

Method

Participants

We tested three subjects from the SIMS community. Demographic information follows:

• Subject 1: Female age 20-25, single first-year MIMS student
• Subject 2: Female age 35-40, married, two children, first-year MIMS student
• Subject 3: Female age 45-50, single, second-year MIMS student

We picked these students because they are not in 213, so they have not been previously exposed to our prototype or idea. They represent different ages, ethnicities, marital status, and life experience. We made sure to test one second-year student because we felt her perspective on selling books might be different than first-years'. We would have liked to have tested a male subject to balance any gender biases, but given the availability of testers at this busy time of year, we had to make do with whom we could find.

Apparatus

We ran the usability study on the PCs in the upstairs SIMS lab. We used the Netscape Navigator 4.75 browser.

Tasks

We will attempt to answer the following questions in all three of our task scenarios:

• Does BookMarket's revised language sufficiently explain the purpose and functionality of the system?
• Does the user understand the meaning of the navigation tabs and use them appropriately to complete tasks?
• Does the user try to use Help? If so, does she find the information she needs to complete the task?
• Overall, are test subjects satisfied enough with BookMarket that they would use it in the real world?

The chart below details the specific task scenarios and what we looked for in each scenario.

We altered the tasks slightly from previous iterations. We changed the title of the book in Task 2 so that in each task, the user buys or sells a different title. In Task 3, we reduced the number of books to sell from three to one. We changed this task because the functionality of selling multiple books at one time is not implemented in the second prototype.

Procedure

1. Collect demographic info.
   

2. Administer Informed Consent Form.
   

3. Show the test subject the BookMarket home page. One of our main goals is to discover whether the system is understandable and usable by first-time users. Therefore, we choose not to give a full demo before the test. Instead, we gave them a brief description of BookMarket.
   

4. Read directions for Task 1.
   

5. Ask a comprehension question about Task 1 (see Questionnaire)
   

6. Read directions for Task 2.
   

7. Ask a comprehension question about Task 2 (see Questionnaire)
   

8. Read directions for Task 3.
   

9. Ask last verbal comprehension question. Administer written Questionnaire.
   

10. Debrief test subject.
    

Test Measures

In our usability study, we recorded the order of actions that the user carried out and her verbal comments. We also recorded the time it took for each user to complete each of the tasks. The user answered an open-ended, verbal question after each task and filled out a questionnaire after she completed the entire test.

We decided to ask questions between tasks to minimize the bias from the learning effect that occurs when the user progresses through the system. We also wished to take a snapshot of the user's understanding of BookMarket at the time of the test and not in hindsight.These questions pertained to the user's comprehension of BookMarket's functionality. For example, after the user posted a book, we asked, "In your understanding, what happens next?" We chose open-ended questions to encourage the user to explain her understanding of BookMarket without cues that come from a multiple-choice questions.

While the verbal questions sought to gauge the user's understanding of BookMarket, the written questionnaire solicited the user's opinion about the system.We asked the user for her overall impressions, likeliness of using the system in the future, and any possible negative comments. Since we actively solicited criticism of the system, we felt that a written questionnaire would allow her to make negative comments more freely.

What we measured and why

Time - We measured the time lapsed between the end of the reading of the task by the administrator and the appearance of the relevant buy/sell confirmation screen. We chose to measure time because we feel it is a general gauge of ease-of-use. Looking back on our time data, we can identify which areas caused specific problems for specific people.This data will be even more useful when it serves as a benchmark for future tests in which we alter the interface.

Navigation Flow- We noted exactly what users clicked during each step of the task. This data tells us how users prefer to move within the site. It tells which entry points are the most visible, the most preferred, and the most useful. It also gave us a step-by-step view of what users do when they encounter trouble. Navigation is a natural reflection of what the user is thinking and what the user expects will happen next.

User Comments-- We paraphrased user think-aloud comments. This data tells us precisely what the user is thinking while she performs a particular action.

Comprehension - We asked verbal questions after task to measure the subjects' comprehension of the task they had just completed and their subsequent expectations. This is important because much of BookMarket's real work is done off-line. For the system to be successful, all users must understand what to do after they leave the site.

Results

Observation Notes by the recorder (includes participant comments during the test)

Participant Verbal Survey Responses

Asked after Task 1:
1. Did you notice a way to discover how many other buyers have contacted a seller about a particular book? (Note: We're testing if the subjects noticed the "Responses" column in the seller table.)

Subject 1: Yes - it's the "Responses" column.
Subject 2: No.
Subject 3: No, not off the top of my head, I'd have to back and look at it.

Asked after Task 2:
2. You've just made an offer to buy The Social Life of Information from several sellers. In your understanding, what happens next? (Note: We're testing if the subjects understand that they should expect an email from sellers who are interested. We want to be sure the subject doesn't think a monetary transaction will happen online.)

Subject 1: I will be contacted by email.
Subject 2: Sellers may or may not respond to me.
Subject 3: Someone will contact me via email if they want to sell it.

Asked after Task 3:
3. So you've just posted Handbook of Usability Testing for sale. In your understanding, what happens next? (Note: We're testing if the subjects understand that they should expect an email from buyers who are interested. We want to be sure the subject doesn't think a monetary transaction will happen online.)

Subject 1: Nothing. I just changed the condition of the book. (Administrator clarifies question.) People will contact me by email to arrange a transaction.
Subject 2: I wait for a person to buy the book.
Subject 3: I wait for an offer. I will see what offers come through via email. They may want to negotiate with me.

4. In your own words, what does BookMarket help you do? (Note: We're testing whether first-time users correctly comprehend what the system does.)

Subject 1: It's a bulletin board to tell others what books are for sale, and a way to browse other people's books.
Subject 2: Helps you sell books in an environment where people might be interested.
Subject 3: It's a communication device. It helps you post books, see the best offer, and find books quickly. It helps you resell and buy in a very quick way. It sounds pretty valuable.

Participant Written Survey Responses

5. If BookMarket was fully implemented and run by Simians, would you use it? Why or why not?

Subject 1: Yes. I think it's a very useful tool that also helps build a community in SIMS.
Subject 2: Yes. I see this system as easier to use than an eBay or other similar auction site.
Subject 3: Yes. New and used book are so expensive! This would undoubtedly save me money.

6. To buy and sell books today, you had to give BookMarket some personal information. Did you feel comfortable giving BookMarket this information? Why or why not?

Subject 1: Yes. Since the community is SIMS, I'm not likely to be spammed
Subject 2: As long as its only my name and email address, I have no problem with it. I don't see how the site would work without that info.
Subject 3: Yes, I felt comfortable.

7. For each question, place an "x" in the column that best describes how you feel. (Numbers represent the number of uses who marked that box)

*Subject 3 wrote: "I seemed to choose the wrong item to edit."

**Subject 2 wrote "Information on the price of the book new would be helpful."

Time Taken to Complete Each Task (MM:SS)

Discussion

Summary of Results - Surprises and Overall Findings
Overall, users were able to complete the tasks quickly and successfully. They had positive feelings towards the system, and they indicated their desire to use the system should it be implemented. We are encouraged by these results.

The tests highlighted some rough areas in our prototype. First, all of our subjects commented on the large number of identical forms they had to fill out to complete the tasks. Our subjects found that repeatedly filling in the same information was laborious and frustrating. We plan to fix this problem by implementing a log-in feature of some sort that pre-fills the forms with the current user's personal information. Were we to conduct the formal usability test outlined below, we would test various different implementations of log-in features to find the one that best satisfies users.

All users hesitated while filling out the Sell Book Form. No one knew how much to ask for the book because our current prototype does not inform the user what prices other sellers (both retail and BookMarket sellers) are asking. We plan to fix this problem in the final prototype by including a table above the Sell Form that states the current BookMarket price range, retail cover price, and competing online bookseller prices.

All users hesitated when filling out the Buy Book Form. No one quite knew what to put in the optional "Comments" box. Would the email being sent by BookMarket already include necessary information such as book title and price to be offered? We plan to fix this in the future in one of two ways: by inputting standard message text into the field, which will let the user know what's being sent and will give them an opportunity to change the default text if necessary; or by adding a "preview message" function so they can see the entire message as it will be sent. We might also implement a combination of these functions.

From our verbal questions during the test, we realized that users get a strong idea of what BookMarket is about and how buying and selling works. We are pleased that our revised text is sufficient to explain these things to a new user. However, we also learned from our questions that not all users notice the "Responses" column in the Buy Form-- that is, they do not notice that they can tell whether a certain seller has been contacted multiple times by other potential buyers. To attempt to correct this problem, we plan to move the "Responses" column closer to the price column, which we know people notice. We may also add another line of explanatory text in the highly-visible table heading, or we may add color to the column. We think this problem is particularly urgent because if users ignore the "Responses" column, the lowest-priced book will always be flooded with offers, while higher-priced books go unsold. We don't want to unduly annoy sellers who have low prices. We also don't want to frustrate buyers by appearing to offer them books that are no longer for sale.

We also noticed that 2 out of 3 users used the Quick Search box on the left navigational bar. Since this search function is currently hard-coded, this test informs us that if we choose to pursue this project and implement BookMarket in the real world, we will need to add legitimate search functionality.

Answers to Our Original Questions

• Does BookMarket's revised language sufficiently explain the purpose and functionality of the system? YES
• Does the user understand the meaning of the navigation tabs and use them appropriately to complete tasks? YES
• Does the user try to use Help? If so, does she find the information she needs to complete the task? UNDETERMINED
• Overall, are test subjects satisfied enough with BookMarket that they would use it in the real world? YES

Changes as a Result of This Study - Ideal Implementation

• Implement login functionality that pre-fills forms
• Create search engine back-end for Quick Search
• Allow user to sell multiple books at once
• Allow user to automatically input books that aren't on book list into system
• Implement administrative functionality of all kinds, so someone can update required books for each class and edit inappropriate entries
• Implement Account functionality such as changing password, retrieving lost password, or deleting account
• Implementing pop-up help
• Implement edit posting and delete posting functionality

What Changes are Feasible for Third Prototype

• Include comparative prices on Sell Form so users know how much to ask for a book
• Include offer information on confirmation page after user contacts sellers to buy a book. This confirmation page detail allows the user to review the offer and print the page for her records if desired
• Clarify what's sent to sellers when a user fills out the Buy Form. Introduce a "preview" screen and/or use textual clues to tell the buyer what an appropriate entry in the Comments field would be.
• Highlight "Responses" column, perhaps by moving its location in the table, changing color, or adding text in table heading.
• Remove "Quick Search" box from Edit Postings pages
• Revise Help page for organization and clarity

Formal Experiment Design

Questions

A. Which method of filling out forms is the least error-prone and frustrating?

• No login (fill out a form for each book bought or sold)
• Mandatory login (user must login or create a new account before first buy or sell is attempted)
• Auto-remember (the system automatically remembers the user's information after the user fills out her first form. The user's information is only remembered during that single session)

B. Which method of selling multiple books is the most efficient (i.e., produces the smallest error rate, most satisfaction, and the fastest completion time)?

• Select one book at a time. Fill out a form for each book.
• Select multiple books and be presented a form for each book sequentially.
• Select multiple books and be presented with a form for all books at once.

Hypothesis

Hypothesis A: Users will be most satisfied and commit the fewest errors with the auto-remember system.
Justification: During our pilot usability test, users remarked that they did not like filling in the same information several times. However, they did not express the desire to log in nor did they go to the Account button to find a login feature.

Hypothesis B: Users will be most efficient at selecting multiple books and filling out one form to sell them all.
Justification: A single form will reduce the number of steps a user must complete to post multiple books.

Factors and Levels for Testing

Between Groups:
Factor 1: Login Type
Level i: No login (NL)
Level ii: Mandatory login (ML)
Level iii: Auto-remember (AR)

Within Groups:
Factor 2: Method of Selling Multiple Books
Level i: One book at a time (OBAT)
Level ii: Multiple selection, multiple forms (MSMF)
Level iii: Multiple selection and single form (MSSF)

Blocking and Repetitions

Here's how we would block out the experiment:

We would employ 36 participants for our test: that is, three per square. The limitation of the number of participants is based on the small size of the SIMS student population. Each Login Type factor will have nine repetitions, while each Selling Method also has nine.

Appendices

Informed Consent Form

Script for Demo and Directions for Task Scenarios

Questionnaire