McInterface: Pilot Usability Study Setup

McInterface
User Interface Design & Development Project
SIMS 213, Spring 2001

: Linda Harjono, Saifon Obromsook, John Yiu Chi Wai

Pilot Usability Study

Introduction

We are designing the user interface for McDonald's food ordering system to be used by the customers inside the restaurant. The system is a walk-up station that operates on a touch screen. Customers would place orders and make payments using the system, then get a receipt from the system that later will be used to get the food. In our design prototype, we simulates the touch screen interaction using mouse-clicks. For this study, we focus on evaluating the ordering-process aspect of the system. We do not and will not test the interactive payment mechanisms (such as verifying the ATM card's PIN).

The goal of this study is more formative than summative. Our pilot usability study is intended to evaluate which aspects of the interface are good and bad, and how the design can be improved. In addition, the study is also intended to test our new design decisions and help us deciding between some design alternatives (please refer to the section on test measures for more detailed descriptions). We encourage participants to think aloud during the usability test and encourage them to give us suggestions at any time.

Prototype Preparation

We made several design changes to our prototype since our Second Interactive Prototype. Here are the elements of our prototype that we changed:

We added the items for Mini Meal Deals and New Taste Menu. We realized that New Taste Menu can apply to both individual and meal items. Therefore, we put it as a separate menu.

The toys are now displayed on the same page as other menu, not on a separate window.

The 'Quit' and 'Check Out' buttons are changed into 'StartOver' and 'Pay' buttons.

To make the Main screen less cluttered, only the medium price is displayed on the Main screen if a food category has size options.

The Item Detail screen for all Extra Value Meal items now indicates that the size applies to the fries and drink.

The 'Next' and 'Previous' buttons are changed into 'Show More' and 'Go Back' buttons. Now those buttons only appear when needed. When the Order Summary is filled up, the system will automatically put the next ordered item on the next page of the Order Summary and display the 'Go Back' button.

The drink and size of the ordered item on the Order Summary are displayed inside parenthesis.

We changed the 'Options' button into 'Edit' button.

After making these changes, we prepared two versions of prototypes, one using 'direct' ordering process and the other one using 'indirect' ordering process.

Direct

In response to the heuristic evaluation on our First Interactive Prototype, we modified the ordering process in our Second Interactive Prototype by:

Displaying the size and drink choices on the Main screen. If the default setting does not apply, users can change the desired size and drink options here.
When users touch on the item's picture, the selected item goes directly to the Order Summary.
To change other options (i.e. special requests options), users need to touch on the 'Edit' button next to each item in Order Summary.

Also, to resolve some design issues, we would like to highlight some characteristics of this version, which are original in our second prototype:

To remove an item from the order summary, users need to reduce the quantity to zero.
The instruction 'Touch the picture to order the item' is on the top of the menu display screen.

Indirect

We were not sure if the above design is a good design decision because the first obvious impression we had was that the menu display screen was very cluttered. So, we also retained the original design with the following flow:

No size and drink choices on the Main screen
Displaying the Item Detail screen each time users select an item from the menu.

In addition, we also added some new design alternatives to this version of prototype:

We placed an 'Order' button next to each item. With this design, users need to touch this button in order to order the item, instead of touching the picture. The label ' Click on the picture to order the item' is thus eliminated.
We replaced the quantity changer with a simple 'Remove' button to be used to remove an ordered item from the Order Summary. The quantity is displayed together with the item description in the Order Summary, and it can only be changed through touching the 'Edit' button. The Special Request icons are placed below/next to the end of the item description.
When the Happy Meal and Mini Meal Deals menu, which do not have size options, are displayed, there is an explanation at the top saying that the items come with small fries and a small drink.

See appendix D to try both versions of our prototype.

Test Measures

During the revision process, we found many controversial design issues. Therefore, we will use the participants' incident logs and comments to evaluate the following aspects of each version of our design and also of the overall design:

Direct vs. Indirect ordering process: how well they perform the tasks using both of our prototype.
'Remove' button vs. the quantity changer on Order Summary: whether it is obvious enough for users that they could remove an ordered item from the Order Summary by simply changing the quantity to zero, and whether it is obvious enough for users to change the quantity via the 'Edit' button in the case when the quantity changer is replaced by the 'Remove' button.
Price display: whether displaying only the price of the medium item size is enough or whether they prefer seeing all the prices on the Main Menu screen.
Menu categories' placement: how easy it is for users to find certain items (to figure out where a certain item is categorized and how to get it).
Food categories' tabs: how obvious it is for users that the tabs can be touched when browsing the food categories.
Special request icons: whether the Special Request icons are meaningful enough to users.
Instructions and button labeling: whether the wording of buttons and instructions are clear, descriptive enough, and make sense to users.
Users' overall feelings towards the design: what they like or dislike about the design and how comfortable they are with the interface. This measure is intended to give the participants opportunities to give us other suggestions that might not be what we initially looked for.

We also record their positive and negative comments, new bugs and design problems discovered by them, some of the their actions that alerted us to potential problems that our design may have, and suggestions that they made while testing our prototype. In addition to logging their actions, we also measure the following variables and use them to help evaluate our prototype:

Time to finish (in minutes)
Number of errors (along with their descriptions)
Number of re-tries before achieving the task

Method

Participants

We conducted the pilot usability test on three participants who we are regular McDonald's customers. We tried to find participants who had similar backgrounds as our personas. One of them participated in our low-fi prototype testing. (The design of our current prototype is different enough from that of our low-fi prototype that the familiarity they got from the earlier testing should little affect their interactions with our current prototype in this experiment.
The table below shows the demographics of the participants.)

Participant	Age	Gender	Country of Origin	Occupation	Frequency of visits to McDonald's	Experience with McInterface
1	40	Female	Costa Rica	Housewife with 3 children	Twice a week	Participated in low-fi testing
2	23	Male	Thai	Master Student in Civil Engineering	Three times a week	Never, but participated in need assessment interview
3	33	Male	American	Software Engineer	Once every two weeks	Never

Apparatus

Our prototype is developed using Microsoft Access and VBA. The equipment we use for the whole testing were:

A desktop computer with a mouse and Microsoft Access 2000 installed
An unzipped pencil bag to resemble the card swiping machine
A stop watch to measure the time used to complete each task

We conducted the usability test in the SIMS' computer lab.

Task Scenarios

In order to test new design changes, we slightly modified some of the task scenarios we had for the First-Interactive Prototype design. Here are the task scenarios:

1. Free-Style

You come to McDonald's and spot this kiosk for the first time, so you want to try using it. (You can do whatever you want with it - ordering food, browsing the menu, or simply playing with the system)

The purpose was to see how the participant interact with the system without being constrained by a pre-defined scenario. Asking him/her to follow a scenario for the first time could shift his/her focus from using the system to remembering the task itself. In addition, since we did not do a demo of the system in this usability study, this is a good way for them to try out the system before we asked them to do a specific task.

2. Quick BigMac

You did not have breakfast this morning, so you decide to come to McDonald's to get your lunch. You are starving at this point and you want to have a Big Mac meal.

The purpose of this task scenario is to evaluate how well the interface supports the participants in ordering an Extra Value Meal, which we see as the most frequently ordered food item at McDonald's.

3. Many Items

For 1st participant: You and your husband just picked your kids and some of their friends up from school. You stop at McDonald's and decide to buy some food for your family and all the other kids. You want two McDouble sandwiches and want to share one medium Coca Cola and one large fries with your husband. Then you realize that you are very thirsty, so you add another medium Coca Cola. Your husband also likes ketchup a lot. You then order other 7 different items for the kids.

For 2nd participant: You just left a party with your girlfriend and a bunch of friends late at night. You decide to stop at McDonald's. Since you just knew that you passed the comprehensive exam, you want to buy some food for everybody. You want two McDouble sandwiches and want to share one medium Coca Cola and one large fries with your girlfriend. Then you realize that you are very thirsty, so you add another medium Coca Cola. Your girlfriend also likes ketchup a lot. You then order other 7 different items for your friends.

For 3rd participant: You just left a party with your date and a bunch of your colleagues late at night. You decide to stop at McDonald's. Since you just knew that you got promoted, you want to buy some food for everybody. You want two McDouble sandwiches and want to share one medium Coca Cola and one large fries with your date. Then you realize that you are very thirsty, so you add another medium Coca Cola. You date also likes ketchup a lot. You then order other 7 different items for your colleagues.

The purpose of this task scenario is to test how easy it is for the participants to find individual food item, to make special request, and to change the quantity of an ordered item using both prototype versions. In addition, this scenario is intended to test how well the Order Summary presents the ordered items when users order more than eight items (in this scenario, users will need to use the 'Show More' and 'Go Back' buttons of the Order Summary section to see all ordered items).

4. Toys and Allergy

Imagine you have 2 kids, you bring your kids to McDonald's to buy lunch because they kids saw the Robo-Chi Pets TV commercial last night and desperately want the toys. You decide to get one Extra Value Meal for yourself and whichever items that offers toys for your kids. One of your kids is allergic to pickles. Before making payment, you then change your mind about your choice of Extra Value Meal, so you cancel your old choice and order a new one.

The purpose of this task scenario is to test how easy it is for the participants to make special requests, to find which food comes with the toys, and to cancel an ordered item using both prototype versions.

Procedure

Here are the procedures of our pilot usability study:

1.We prepared the needed equipment and had the prototype ready to use. Then, we invited the participants to sit in front of the prototype screen.

2. We gave a brief introduction about our project and explained the purpose of the pilot usability test to the participants. We also assured them that the test results would be kept confidential and emphasized that it was the prototype that we wanted to test, not the participants. Then, we asked them to sign a consent form. We also encouraged them to relax, "think aloud" while performing the tasks, and not hesitate to ask questions during the sessions.

3. Then, we gave the participants a general instruction of how the prototype worked and told them about the limitations of our prototype. However, we did not show any demo to the participants since we expect them to be able to use our interface without any prior training. (Please refer to appendix E for the scripts of system preview and instructions we read to the participants).

4. We first asked them to do a Free-Style. Then, we asked them to do three task scenarios on both versions of our prototypes, first on the 'direct' version, then on the 'indirect' one. We wanted to know whether the participants will feel cumbersome using the 'indirect' version or not after they used to work with the 'direct' version. Or, whether the advantages of seeing simpler interface on the Main screen and seeing more detailed options are worth the extra step in the 'indirect' version.

5. During the test, we made critical incidents log to record their positive and negative comments, new bugs and design problems they discovered, some of the their actions that alerted us to potential problems, and their suggestions during the testing (see appendix E).
We tried to measured the time used by the first participant to perform each task. However, since she often stops doing what she did and gave her opinion about the interface, we could not measure the time she took to perform each task accurately. Since direct feedback while participants are doing the tasks are more important to us than the accurate time they took to complete the tasks, we then decided not to record the time needed to complete certain tasks for the other participants.
We also recorded number and descriptions of errors the participants made performing each task and number of retries before they can achieve the task.

6. After that, we had a follow-up interview with the participants to discuss the test and to get some more suggestions to improve our design (See appendix E for the list of our interview questions). At the end, we thanked the participants for their participations.

We understand that the learning effects might distort the quantitative measures of this experiment. However, this is a formative usability study, so we are more interested in the qualitative results of the testing such as their opinions and suggestions.