Method & Test Measures
Method
Participants
We found our participants by recruiting housemates of our friends.
Participant A is a 31-year-old male who is native to the Bay Area. He lives in Oakland and commutes to work on bicycle. He also has a motorcycle, which he uses once or twice a week to pick up groceries or visit friends in SF. He describes his experience with web-based systems as "average"; he frequently uses Transit Info to explore public transit options.
Participant B is a 23-year-old female who moved to the Bay Area six months ago. She teaches high school science in Oakland. Every day she drives twenty minutes to work. She also drives regularly on weekends, to party in SF and to hike or camp in various locations throughout the area. She frequently uses MapQuest to find directions and maps for her travel plans. Though she is certainly computer literate, she is largely technically ignorant-- she much prefers people to computers.
Participant C is a 24-year-old male who also moved to the Bay Area six months ago. He commutes daily from Oakland to San Francisco, which takes him about 40 minutes each way. He too drives regularly on weekends, for recreational hiking trips and for hanging out with his work buddies in SF. He typically consults his paper map for directions, though he has occasionally used MapQuest or Yahoo! Maps.
Our testers represented a diverse range of backgrounds and a reasonable spread in terms of age. Participants B and C had only some rudimentary computer training, and thus were especially valuable to us since their interaction with the computer was less biased by their technical know-how. Their commute needs also varied significantly: Participant A mainly commutes by bike and uses his car for roads trips and short local commutes, while Participants B and C commute to work. This variation was also significant, since our system is designed to accommodate the needs of different types of commuters.
Apparatus
We performed our tests in the respective homes of our participants; a ThinkPad laptop computer with a desktop mouse placed on top of a kitchen table was the setup that we used for all tests. We used Internet Explorer 6.0 running on Windows XP Professional, with a screen size of 1024x768 and 32-bit color.
Procedure
All testers received the same set of instructions before the start of the testing session. We briefly described to the participant what the system was supposed to do, and that they were among our group of initial testers. We explained that we were not supposed to answer any questions, but encouraged them to think out loud as they performed the tasks. We also explained that all session information would remain confidential. Finally, we clarified that it was the system that was being tested, not the testers themselves, and that they shouldn’t hesitate to give us whatever feedback came to their minds as they made their way through the system. Each participant was asked to complete both of the tasks described below, one task at a time. When they were finished, we asked some follow-up questions, as described below.
Task Scenarios
We used the same tasks that were developed for our first round of usability testing, as repeated here:
Task 1: You are a commuter who travels from home to the same office each day through city and highway traffic. You almost always follow the same route and the trip generally takes you about an hour each way. You're at home on a weeknight and you'd like to learn what traffic will be like during your commute tomorrow. A friend recently told you about so you decide to consult it. Find out how much traffic you can expect tomorrow morning.
Task 2: You've already registered and you've already told the system your home and work addresses. For your next morning commute, determine what type of traffic you can expect and determine whether or not to take a route different from your usual route. What time will you need to leave home in order to get to work on time?
Test measures.
For each participant we measured number of errors, type of errors and qualitative satisfaction with the system. To measure errors, we took detailed notes during each test, documenting all mistakes and misconceptions.
For qualitative assessment of overall satisfaction, we assessed a follow-up interview, consisting of the following questions:
- What is your general impression of the Road Sage interface?
- What did you like about the system?
- What did you find particularly confusing?
- Any recommendations on how to improve the things you found confusing?
Our findings are summarized in the Results section.