Test Raw Log

[Evaluator1| Evaluator2| Evaluator3]

Evaluator1

Action sequences

• First he looked around the interface to see if there is anything on "What's KM".
• He decided to start with "about KM".
• He read the contents in that page.
• He clicked on "What is KM" on the left frame, read the articles that appeared on the right frame.
• He Kept clicking down the list on the left frame, read the articles on the right frame.
• After reading the information in "about KM" page, went back to "Home" to see if anything else interesting
• Decided to go to "Resources" page, finding out that these are outside links, not interested.
• Decided to go to "Browse Subject", browsed the categories and the items under some categories.

• He decided to go to "Full Search".
• In "Full Search" page, typed in the author name in the "Author" field, Clicked on the "Search" button. Read the results returned by the system, might find some books or articles that he wanted to explore more.
• After searching, went back to "Resources". Checked some web sites to see if he could find some other interesting information.

• Decided to go to "Full Search".
• In the "Full Search" page, typed in the keywords "Intellectual Capital", clicked on the "Search" button.
• Read the search results.
• Turned to "Browse Subject", did not know which categories may contain the information about "Intellectual Capital".
• Went back to the "Full Search", repeated the same search query as before, checked the categories on the left frame.
• Noticed "Your Folder" at the top right corner, clicked "Your Folder". He said that it may be a nice feature. But he did not notice it before. He may need to take some time to learn it.

Task3:

• Remembered seeing something about companies in "about KM" page, so decided to check "about KM" page first, but did not find the information there.
• Decided to try "Resources", checked the "Web sites" to see if there is any company information. If no, checked out the "Organizations" to find out some company names. If still could not find some info, go to "Full Search". In "Full Search" page, typed in keywords "KM systems". Read the results. Hopefully he could find the company information from the articles or books returned in the search results. Then he would go to the company's web sites and check its competitor's web sites.

Problems & questions

• Got lost in "Browse Subject", did not know which categories to go to find the information he needed.
• "Your Folder". Did not notice it. He said that It might be a nice feature but he does not use "Save" very often and it may take some time to learn how to use it.
• Asked if the Press releases on the "Home" page are news articles?
• Had problem in understanding the categories in the left frame on "Search Results" page.

Comments & Suggestions

• on the "Home" page, states what the users can find in this site, especially useful for the first-time users.
• Sitemap: brief description of the pages.
• Add company information in the "About KM" page or "Resources" page.

Evaluator2

Action Sequences

• He first examined the "Home" page to see if it is the right site to get information on KM.

• The title is not catching, has too many words.
• Did not understand what is "KM" in "About KM" tab, finally realized it is the abbreviation of "Knowledge Management".
• Did not like the tabs. Icons are easier for him.
• From the "Home" page, he wants to know how to use this site and whether it is useful for him to get information on KM, he does not care about our project or the process of developing the portal.
• "Establish authority" is ok for him, but he would expect it appears on the left frame.
• "Press Releases" is good.
• From the "Home" page, he figured out that this site provides a search engine that can be used to  search the information he needs.
• After browsing the "Home" page, he decided to go to "about KM" first.
• When presented with the "about KM" page, he said that "Oh, this is helpful, very helpful!".
• For the questions listed on the "about KM" page, he commented that from his industry background, he is more interested in the practical issues than the academic questions, such as the hot topics in KM, the most famous authors in KM.
• He would be happy to see some of the introduction information on KM appears on the "Home" page.
• After reading the "about KM" page, he turned to "Resources".
• When presented with the "Resources" page, he said that the information is very useful, but is not what he expected. He thought that "Resources" contains something more generic, like reference books. He suggested to use "References & Resources" instead of "Resources" or add a brief description of the "Resources" somewhere on the "Home" page. Otherwise, he might have difficulty to understand what is under "Resources" because the "Home" page did not tell him what these tabs are about.
• Then he clicked on the "Full Search" .
• When seeing the "Full Search", he figured out that it is a search with more options than the quick search on the "Home" page. He inferred that it is mainly for searching the literature. The interface is very typical as those used in the library.
• Then he decided to explore the "Browse Subject".
• When presented with "Browse Subject", he was confused. He think it is redundant with the "Full Search". He was confused with the terminology, "Subject" and "Topics", "Subject" and "Categories". He did not know what is the difference between them.
• He noticed "Your Folder" on the right top of the screen. He did not understand what it is and how to use it. He expects that if he clicks on the Help, it can explain the purpose and use of "Your Folder".

• Looking at the task description, he said that it is easy.
• He decided to go to "Full Search". He had seen the "Full Search" page when doing Task1A, so he knew he can search Author there. If he had not seen it before, he said that he may need some search tips.
• He typed in the Author name in the "Author" field, clicked on "Search" button.
• In the "Search Results" page, he would like to click on the "subjects" links in the listed results to verify that this item is what he wanted.
• He was puzzled by the two terms "Subject" and "Category" used in this page.

• He took the conventional way: go to "Full Search".
• He typed in "Intellectual Capital" in the "Topics" field, clicked on the "Search" button. He browsed the search results, then might go back to the "Full Search". He might try to search  "Intellectual Capital" as the subject. If he finds some interesting articles, he saves them in his folder.
• He went back to "Home" to see if there are other functions. Clicked on the "Search Tips" to see what "Quick Search" is. He decided to quit after finding it is not as powerful as "Full Search".
• He turned to "Browse Subject". He guessed that he might get some information from "Asset Utilization" category. But found out that it was not the right place. He tried other categories hoping to find something related to "Intellectual Capital". Frustrated by "Browse Subject", he changed his mind.
• He went back to "Your Folder", which contained the records he saved from the "Full Search". He would look at the search results and hope that he can find the subjects that relate to "Intellectual Capital" from the results. Then he would go back to "Browse Subject". Hopefully he had got some hints on which categories to go to from the search results in his folder.
• He felt frustrated when using "Browse Subject". He did not see any top level categories is related to "Intellectual Capital". He was also afraid that there may be several levels of sub-categories under the top level categories. He has to take time to browse down the subject hierarchy.

• He decided to go to "Resources" right away. He would check the web sites and pick up some web sites ranked as "Excellent" by the system.
• He turned to "Full Search".
• He had trouble with "Topics" and "Subjects". He did not know the difference between the two.
• He typed in the "knowledge management systems" in the "Topics" field and clicked on the "Search" button.
• He hopes that the the search results is no more than 10. If it is more than 10, he wanted to refine the query. He thought to use "Quick Search" because he thought the search results from the "Quick Search" and "Full Search" would be different.
• He went back to "Resources"
• He clicked on the "Publications". He might pick up a few well-recognized journals from there. They may have some information about companies. He would also check the  "Organizations" hoping that he can find some companies' names there.
• After finding the names of some companies, he would go back to "Full Search", and try search on the articles using "knowledge management systems" and the company name as the keywords.

Comments & suggestions

• "Browse Subject" is the last resort. It is not very useful. He will only use it when he exhausts all other resources.
• He was not very clear of  the difference between "Quick Search" and  "Full Search". He thinks that "Quick Search" is not very helpful and confusing. So it is better to merge it with the "Full Search".
• In "Full Search", he was confused with "Topics" and "Subject". He suggested using "Keywords" instead of "Topics".
• He thinks that "Subject" and "Category" are the same. we should choose one term in our interface. He thinks that "Subject" is more intuitive.
• When asked why he typed in the subject instead of clicking on the "Select Subjects", he answered that he did not realize that subjects are different from keywords at that time. He did not read the instruction notes on the top of the frame because he thinks that the search interface is very common, he won't bother reading the notes on the page.
• He suggested if the user did the wrong way, the system should give feedback indicating what was wrong. For instance, if he typed in the subject which is not a valid subject term, the system should tell him and guide him to click on "Select Subjects" to choose subject terms.
• He suggested to add "save all to the folder" in the "Search Result" page.
• When asked if he likes to email the contents of "Your Folder" to himself or other people, he said, most time, he would rather digest the information by himself first before sending it to others.
• He does not like the tabs on  "Home" page. They are too formal. And they are one-dimensional. He was forced to read from left to right. He prefers icons, some humorous, interesting icons.

Evaluator3

Action Sequences

• He looked around the "Home" page, then said,  "obviously go to about KM".
• In "about KM" page, he would click the questions from the top down to the bottom, and read the articles one by one.
• He said that when using a software, he usually start with documentation. This is why he chose "about KM" first.

• After reading the scenario for task1B, he said, "obviously Resources". After browsing the index of resources, he thought the information he wants most likely appears in "Publications". Then he realized he was wrong because "Publications" only contains information on KM journals.
• He decided to go to "Full Search". He said that he generally avoids using "Search" because "Search" often gives him a lot of irrelevant information. Some relevant information is left out or ranked very low.
• He typed in the author name in "Author" field, and clicked "Search" button.
• Then he would read the search results.
• The term "Full Search" was confusing for him. When he saw "Full Search", he expected it was "Full text search".

• He chose "about KM" first. Then click on "What are the hot topics". There may be something about "Intellectual Capital" under this question.
• Then he would use the information he got as a starting point for further search.

• When reading the scenario for task3, he looked around the interface, seemed hesitating for a while.
• He decided to go to "Browse Subject".
• In "Browse Subject" page, after browsing the top-level categories, he said, "obviously it is not here".
• Then he went to "about KM", "it is not here either".
• He turned to "Resources", tried "Organization", but did not find the information he needed.
• As a last resort, he went to "Full Search". He tried to type in some words in "Selected subjects" field, got an error message.
• Then he clicked on "Select Subject". If the subjects span on too many pages, he has to give up.

Comments and suggestions

• .For "Home" page, he suggested to change "Browse Subjects" to "Browse Contents", "Full Search" to "Search".
• The tabs are ok for him.
• For "Full Search" page, he suggested to use "keyword" instead of "topics". He preferred the "Title" appearing before "Keyword". He commented that the help information on the left frame would be helpful but he usually won't bother reading it.
• When asked why he did not click on the top level categories to continue browsing sub-categories, he answered that the "top-level categories on Knowledge management process" was misleading. He thought these categories were about KM process.
• When asked why he did not use "Your Folder", he said that he did not encounter it. it would be helpful.
• When asked if he prefers to email the search results to himself or to other people, he said he would rather digest the result by himself first.