Introduction,
Task Flow,
Work Distribution
Method & Test Measures
Method
Participants
We recruited three Bay Area drivers to test our low-fi prototype. We selected
participants who had no prior exposure to our system, and furthermore
no personal connection to any of our group members (i.e., no friends).
We found participants through friends-of-friends and through casual acquaintances.
We looked for a range of ages, driving frequency and familiarity with
Bay Area freeways. The demographics of our participants are as follows:
- Participant A is a 40-year-old woman whose daily commute varies from day to day. A lifelong resident of San Jose, she is extremely familiar with Bay Area freeways and traffic patterns. She attends grad school in Berkeley, works a part-time job in Milpitas and frequently spends time with her boyfriend in Hayward. This lifestyle requires her to zig-zag back and forth across Bay Area freeways frequently. Her schedule is flexible and somewhat unpredictable.
- Participant
B
is a 25-year-old woman who lives in San Jose. A Bay Area resident for
10 years, she is very familiar with the freeways and traffic patterns.
She commutes a half-hour every day to work, and she often drives to
Berkeley on weekends to visit old friends from school.
- Participant C is a 26-year-old woman who attends grad school at UC Berkeley. She lives in Berkeley and walks to school every day. She is primarily a weekend driver: she likes to take road trips to Tahoe, Napa and other recreational spots. She moved to the Bay Area one year ago and still considers herself new to the freeway system.
Task Scenarios
Task 1: You are a commuter who travels from home to the same office each day through city and highway traffic. You almost always follow the same route and the trip generally takes you about an hour each way. You're at home on a weeknight and you'd like to learn what traffic will be like during your commute tomorrow. A friend recently told you about so you decide to consult it. Find out how much traffic you can expect tomorrow morning.
Task 2: You've already registered and you've already told the system
your home and work addresses. For your next morning commute, determine
what type of traffic you can expect and determine whether or not to take
a route different from your usual route. What time will youl need to leave
home in order to get to work on time?
Task 3: It's Thursday afternoon, you have a few spare minutes at work and you'd like to plan a weekend trip to [ask user to specify a destination within a day's drive that they'd like to visit, but that they haven't visited and don't know the precise route to]. You'd like to meet friends there in the afternoon but you're not committed to a specific time. You're more interested in avoiding unnecessary traffic than in arriving at a certain time. Find out what some of the best possible routes are to your destination, browse the alternatives, choose the best one and print out the directions.
Procedure
Each participant committed to an hour-long session of testing our paper
prototype. Participants signed a consent form (see Appendix) that guaranteed
their privacy and specified that the test was voluntary, without paid
benefits.
Three team members were present to administer each testing session. Our roles, which we rotated between sessions, were as follows:
- Facilitator: This person was the officiator, the only one who was allowed to speak to the participant. He/she conducted the test, presenting the task scenarios and encouraging the user to think aloud as much as possible.
- Computer: This person played the computer, enabling the paper prototype to behaves similarly to the true logic of our system. The "Computer" arranged the paper prototype to simulate system response to user input. For example, when the participant pressed a button on the paper prototype, the "Computer" replaced the old screen with whatever new screen should appear.
- Observer: This person silently observed the testing session, taking detailed notes on a laptop.
Each test
began with introductions to our team and our respective testing roles.
The facilitator explained our testing procedures, informing the participant
that she would be given two tasks to complete using our system. The facilitator
emphasized that we intended the system to speak for itself; we could not
answer questions or explain our design to the participant. Any lingering
questions could be answered after the test during a debriefing. The facilitator
then handed written tasks to the user one at a time, also reading each
task aloud. The facilitator tried to elicit the participant's thoughts
as he/she proceeded with the given task. All team members strictly refrained
from explaining the system or reacting visibly to the participant's actions.
The observer and computer occasionally interjected with questions if something
was unclear. The computer used Post-It notes to immediately create buttons
on-the-fly if a user desired added functionality to complete the assigned
task. After both tasks were completed, we held a 10-minute debriefing
session in which everyone present could ask questions and talk freely
about the design.
Test Measures
We looked for a variety of responses from the subjects including but
not
limited to (1) focus of attention, (2) interpretation of symbols,
explanative text, and underlying implementation, (3) perception of
differentiating features of the system, and (4) perceived reliability
of
the predictions produced by the system. We evaluated each of these test
measures by asking the subject to vocalize their thought process as they
completed tasks. Users were also asked to give their overall impressions
in a common period at the end of their evaluation. Generally, we were
able to get clear and consistent feedback from the users.