1. Administer pre-test questionnaire

• Have you heard of the term "Knowledge Management"? If so, where?
• What do you think KM is? What is the difference between KM and information systems, KM & CS?
• Have you ever used any KM site? Which ones?

2. Describe the purpose of the observation.

• Thank you for giving us an hour of your time.
• You’re helping us by trying out this user interface in its early stages.
• We’re testing this products; we’re not testing you.
• We’re looking for places where the user interface may be difficult to use.
• If you have trouble with some of the tasks; it’s the interface’s fault, not yours. Don’t feel bad; that’s exactly what we’re looking for.
• If we can locate the trouble spots, then we can go back and improve the interface.

• Remember, this is totally voluntary. Although we don’t know of any reason for this to happen, if you become uncomfortable or find this objectionable in any way, feel free to quit at any time.

4. Explain how to "think aloud."

• We have found that we get a great deal of information from these observations if we ask people to think aloud as they work through the exercises.
• It may be a bit awkward at first, but it’s really easy once you get used to it.
• All you have to do is speak you thoughts as you work.
• If you forget to think aloud, I’ll remind you to keep talking.
• Would you like me to demonstrate?

• The user interfaces we’re testing are web pages to a knowledge management portal.
• We don’t expect you to have any background or understanding of knowledge management. In fact, we are working specifically with novices who have no background in this discipline.
• I will pass you written instructions on tasks we want you to perform. Tell us what action you would take with the interface to achieve the goal we give you. For instance, you could say, "I want to press the submit button" or "I want to click this hyperlink." (wizard) will pretend to be the computer and show you the next screen based on your decision. During the whole exercise (note taker) will be taking notes keeping track of features that appear to be intuitive as well as the interface’s problem areas.

• As you’re working through the exercises, we won’t be able to provide help or answer questions. This is because we want to create the most realistic situation possible.
• Even though we won’t be able to answer most of your questions, please ask them anyway. (note taker’s name) will note you questions and I’ll answer them later.

• Do you have any questions before we start?

9. Start test using these task scenarios.

Scenario 1A -- You work as a project manager in a company. You are new to the knowledge management domain and want to find some general information about the subject. How would you go about finding information within the site? Feel free to explore the site in the process.

Our guess:

• Since he knows little about KM, he decides to read information in FAQ's on KM first to get familiar with the area.
• With enough background information in hand, he decides to browse the KM thesaurus subjects and navigate down hierarchies to get a better understanding of items that were interesting from the FAQ's.

Our guess:

• User will enter Nonaka’s name in the full-text query box and receive a number of irrelevant articles which may mention Nonaka’s name in the context of the article.
• User can refine query with bibliographic options by going to the Full Search tab.

Scenario 2 – You are a graduate student in the Business school working on a class project. A book briefly describing a branch of knowledge management called intellectual capital seems likely to have a lot of information for your project. Find information about intellectual capital. Feel free to explore the site in the process.

Our guess:

• The user has some keywords in mind for these topics. So the user finds that the full-text query option provided by the system may be the fastest way to find information she needs. The user types in the keywords in the full-text query box.
• The search results present some records of interest that the user subsequently selects and emails to herself. The user iterates through more searches by clicking on the hyperlinked subject categories of interesting records to find other records that might be useful. The user keeps doing this until she either runs a new query altogether or she has enough information.

Our guess:

• Since the question is pretty vague, she has no good keywords to start with. She decides to navigate the thesaurus hierarchies to get some ideas.
• After navigating the thesaurus and checking the items in some of the categories, she may have a clearer picture on the problem and find some interesting information.
• In this stage, she could also use the full-text query option to conduct further searches on the topics that are particularly interesting.
• As she continues throughout her search process she keeps adding records into her folder.
• When she’s finished researching, she emails the records to herself.

10. Conclude the observation.

• You’ve just finished you last exercise.
• We appreciate the time you’ve spent helping us understand how novices might search for information on our site by either browsing or employing the full-search interface. We also wanted to see whether novices might take advantage of the folder feature to compile and save their on-line work.
• Answer any questions the participant had during or after the test.
• Discuss any interesting behavior we would like the participant to explain.