The scenarios should give good coverage of your interface; if they do not, or if you need to change them for other reasons, you should redesign them in advance.
To prepare for the evaluation you should assign team members to the different tasks (i.e., computer, facilitator, etc.) and practice with someone playing the participant. Get any bugs out of your procedures. It may take about 4-8 hours to create the materials and several more hours to practice playing computer.
Find three participants (i.e., volunteers who are not in your group) to work through your task scenarios. You should not use friends. The type of people you recruit should be based on your task analysis.
You should ask the participants to sign an informed consent form saying the test results will be confidential. Be sure to let participants know they are free to change their mind about doing the study if they do not want to sign the form, but if they don't sign the form then do not do the study with them.
Example documents can be modified to fit your project's description.
Procedure for Informal Usability Study:
You should write-up a script of how you will proceed and follow the same script with each participant. Tell each participant what they are trying to achieve, but not how to do it. When they are finished, you will give them the directions for the next goal and so on. Each participant will perform tasks to achieve all three goals. Keep the data separate for each task and participant.
Give each participant a short demo of how to work with someone "playing computer". Do not show them exactly how to perform your tasks. Just show how the system works in general and give an example of something specific that is different enough from your scenarios.
Instruct the participant to "think aloud" to let you know what they are thinking as they work. Remember not to give them hints or ask them questions while they are executing their tasks, but you may need to prompt them to keep them talking.
After each participant has finished the tasks, "debrief" them about the interface in general, and follow up on questions and issues that arose during the course of the task execution.
It would be preferable if you can find an empty room or a space with only a few other people in it.
As discussed in class, testing the interface should take about an hour per participant.
Measurements and Observations:
Keep a log of critical incidents (both positive and negative events). For example, the user might make a mistake and you notice it or they might see something they like and say "cool". Keep track of when they hesistate and when they proceed confidently. You may want to write this on a copy of the script or of sketches showing the interface at different stages. Later you should prioritize these events and give severity ratings to the problems you observed.
You should concentrate on process data (i.e., what is happening in the big picture) rather than detailed quantitative data (i.e., time or number of errors). Observers should jot down ideas for alternatives to the design based on what they observe the participants doing.
Report your results (summaries of the process data) and draw some conclusions with respect to your interface prototype. This should be the most important part of the write-up. We want to understand how you would change your system as a result of what you observed.
Your write-up should be no more than five pages (again, please put it on your project web site). The write-up should follow this outline (number of pages/section are approximate):