PROTOTYPE Experiment KIT
SFnight project
Procedures:
- “Facilitator
1”
1. Ask user to Fill up User Questionnaire and the consent agreement:
2. Will remind the user that we are not testing the user ability, but the system itself.
3. Also will explain the tester about the technique “thinking loud”
4. Runs the Demo and introduces
the subject to the site
- “Facilitator
2”
1. Read the Task. The facilitator would ask the subject to read each Task before start each site evaluation. The time started to be counted only after reading the task 2. The task description sheet should be available for subject’s consultation anytime.
2. Start the Task #: The tester begins to perform the Task 1 in the prototype.
3. Finish the Task #: Once the subject has specified that the task is done, give him/her a small evaluation form, see item 6.
4. Task Questionnaire # of Site: After each task, ask the tester to complete a questionnaire about the task related.
- “Observer 1” “Observer 2”. Will do the following
· Follow each Task Observation Requirement.
· Take notes of the user’s expressions and words, when the user is “thinking out loud”.
· Make note when the user clicks in the wrong place. This requires high concentration; otherwise we loose data.
· Be responsible for counting the time.
- “Facilitator 2”
After all tasks are finished, finishes the post-test interview/Questionnaire with the subject, if necessary by him/herself. If the subject is tired, skip this part.
Statement of Informed Consent
A group of Masters and PhD students in the School of Information Management and Systems and the Computer Science Department at UC Berkeley are conducting studies to assess the viability of several different methods for searching the contents of large World Wide Web sites.
If you volunteer to participate in this study, you will be asked to type some queries into a computer interface, look at the results returned by one or more different computer user interfaces, evaluate those results, and answer some questions. The questions will be asked orally before, during and after the search sessions. There will also be written questionnaries before and after the search sessions. Your interactions with the computer will be recorded automatically in a log file.
There are no direct benefits to you for participating other than what may be an educational experience in viewing interesting ways to search web sites. There are also probably no risks to you; you will simply be using a computer, typing some queries, and viewing and evaluating the results. We will associate an ID number with your log files but will keep a record of the link between your name and ID number. However, we will not name you if and when we discuss your behavior in research publications.
If you accept these terms, please write your initials and the date here: ___________________
1 About the Tester
1.
How old are you?
Over 45 years old
31-45 years old
21-30 years old
18-20 years old
2.
How long have you been using the Internet
2-5 years
1-2 years
Less than 1 year
3. Have you used one of the following web sites about entertainment?
Sfgate.com
Citysearch
4. Have you created any personal page [such as MyYahoo, etc] in a web site?
Yes, Please specify: …………………………..
No
Task 1 - PLANNING AN EVENT
Background: It is Wednesday evening. A friend is visiting from out of town and wants you to help organize an outing with some local friends Saturday night. At SFnight, you find some events and venues where you think you’ll all have fun.
You send several options to your friends.
Task 1 Task Questionnaire
SFnight Questions
Please indicate the degree of which you DISAGREE (1) and AGREE (5) with the
following statements about this SFnight when you performed this task.
|
|
|
<- Disagree Agree -> |
||||
|
1 |
2 |
3 |
4 |
5 |
||
|
1 |
Selection process is easy. |
|
|
|
|
|
|
2 |
Selection process is
straightforward. |
|
|
|
|
|
|
3 |
The site provided adequate
information to complete the task. |
|
|
|
|
|
|
4 |
You are sure that you sent all
information you need from SFnight to your friends. |
|
|
|
|
|
|
5 |
You would use this feature again
to email selections to your friends. |
|
|
|
|
|
General Questions
1. Was any part of the steps confusing, hard to understand? Please describe.
2. Any other thoughts and comments?
Task 2 – SIGN UP FOR MYSFNIGHT
Background:You just heard from a friend that you can get personalized information about San Francisco nightlife by becoming a SFnight member. You are curious and go to the website to signup.
Task 2 Task Questionnaire
SFnight Questions
Please indicate the degree of which you DISAGREE (1) and AGREE (5) with the
following statements about this SFnight when you performed this task.
|
|
|
<- Disagree Agree -> |
||||
|
1 |
2 |
3 |
4 |
5 |
||
|
1 |
Selection process is easy. |
|
|
|
|
|
|
2 |
Selection process is
straightforward. |
|
|
|
|
|
|
3 |
The site provided adequate
information to complete the task. |
|
|
|
|
|
|
4 |
In order to sign up, the site
required adequate amount of information. |
|
|
|
|
|
|
5 |
Once you completed registration,
SFnight provided adequate indication that you are now a member.. |
|
|
|
|
|
General Questions
1. Was any part of the steps confusing, hard to understand? Please describe.
2. Any other thoughts and comments?
Task 3 –USING MYSFNIGHT
Background: Two friends are visiting from XX
to XX. Each has different taste in music. You go to SFnight to plan a calendar
that can accommodate the interests of both.
Task 3 Task Questionnaire
SFnight Questions
Please indicate the degree of which you DISAGREE (1) and AGREE (5) with the
following statements about this SFnight when you performed this task.
|
|
|
<- Disagree Agree -> |
||||
|
1 |
2 |
3 |
4 |
5 |
||
|
1 |
Selection process is easy. |
|
|
|
|
|
|
2 |
Selection process is
straightforward. |
|
|
|
|
|
|
3 |
The site provided adequate
information to complete the task. |
|
|
|
|
|
|
4 |
MySFnight made it easy to plan. |
|
|
|
|
|
|
5 |
You would use this feature again
to plan future events? |
|
|
|
|
|
General Questions
1. Was any part of the steps confusing, hard to understand? Please describe.
2. Any other thoughts and comments?
Post-test
questionnaire
1. Which aspects
did you find particularly easy in this site?
2. Which aspects
did you find particularly difficult in this site?
3. Please give
additional comments about this site. We would especially appreciate your
feedback on the following topics:
n
Which functions did you like most, which ones you think you
would not use
n
Would you in reality create your personal page to have
nightlife information at this site?
n
Anything that you think that is missing in this site
Instructions
for conducting TASK 1 - PLANNING AN EVENT
Procedure: Give only the Task description related with the site.
Time & Help: We will not count time, since the user might want to see information related to venues or events. More important to observe. If tester cannot figure out how to do after one minute, the facilitator should help, and observer register.
Expected Procedure of the User:
1. Find 2-3 venues/events that would recommend to your friends.
2. Add each event to E-MAIL List
3. View the profile of the venues he/she selected.
4. Confirm the selection and send it to friends.
What to observe
1. Did the subject understand the task?
2. Did the subject need help to find information after one minute?
3. Did the subject follow correct procedure? Did she get disoriented?
4. Did the subject come up with other needs during the process?
5. Did the subject use the E-mail list, or went to other places
6. Was it easy for the subject to add their choices to the list? And send to friends?
Measurement:
· Questionnaire satisfaction.
·
Number of clicks in the wrong place.
Instructions
for conducting TASK 2 – SIGN UP FOR MYSFNIGHT
Procedure: Give only the
Task description. If user starts to
customize MySFnight, stop her.
Expected Procedure of the
User:
1. Find sign up MySFnight button, click and follow the instructions
2. Select favorite venues and music types would like to include in the personal area.
3. View welcome and confirmation page and go back to homepage
What to observe:
1. Did the subject understand the task?
2. Did the subject noticed the personal area in the homepage?
3. Did the subject need help to find information after 1 minutes?
4.
Which procedure did the
user follow in the beginning?
5. Did the subject come up with other needs during the process?
6. Does the user see the two kinds of newsletter/registration; How confusing is this?
7. What type of facial expressions does the user make during the process: bored, smiling, uncertain, frowning, etc
8. What type of body language does the user make: shuffling, scratching their heads, putting hand on chin…
Measurement:
· Time to perform the task
· Questionnaire satisfaction.
·
Number of clicks in the wrong place.
Instructions
for conducting TASK 3 – USING MYSFNIGHT
Procedure:
Give only the Task description. Give the tester overall site evaluation
form after the task.
Expected Procedure of the User:
1. Discover that the task can be completed at MySFnight
2. Move Editor’s Picks to Your Picks
3. Add to MySFnight from venue/event profiles
4. Add to MySFnight from Search Results
5. Perhaps delete something from MySFnight
What to observe:
1. If the user uses the personal area, or would tend to go to the other areas
2. How easy is it for the user to add their own choices, and do other tasks such as sending to friends
3. Other needs that may come up when performing the task.
4. What type of facial expressions does the user make during the process: bored, smiling, uncertain, frowning, etc
5. What type of body language does the user make: shuffling, scratching their heads, putting hand on chin…
Measurement:
· Questionnaire satisfaction
· We will not measure time here, since users might take their time to explore and make decisions.
· Number of wrong clicks is the most important thing to track.