1. Introduction
2. Method
3. Test Measures
4. Results
5. Discussion
6. Formal Experiment Design
7. Appendices 

1. Introduction

2. Method

We have selected test users who have basic knowledge of web browsing and knows the concept of bookmarking. All of our test users use web barowsers everyday and have good exeprience with the bookmark function of the browser. They also create folders to organize their bookmarks. The history function is not very popular among the test uers. The following table shows the test users' answers to the demographic questions. 
 

Gender	female 2  male 1
Education	SIMS Masters 2  SIMS Ph.D. 1
How often do you often use web browsers?	everyday 3
Do you bookmark the web sites you like or often visit?	often 1  sometimes 2  (1 - mostly use customized web page using CGI to save and access bookmarks)
Do you organize your bookmarks using folders?	Yes. I have created many folders 2  other 1  (customized web page organized by pre-set category)
Do you use the history function of the web browser?	sometimes 2  I didn't know the history function 1

Apparatus

User tests are conducted using the NT machine in the SIMS lab. The NT machine runs Exceed to access the UNIX server to run the the application coded by Java.

Tasks

We are using the same scenarios from the previous tests. They are repeated in the following;

A 
You organize your bookmarks using folders. Recently you have visited several newspaper web sites and saved these sites as bookmarks. Now you are viewing The New York Times homepage. 

1. Add this page to your bookmark using the Webmarking interface 
2. Create a new folder to store several newspaper web sites. Name the folder "News". 
3. Put the New York Times bookmark to the "News" folder. 

1. Open a history folder for yesterday. 
2. Find The Financial Times site in the folder. 
3. Move the site to the "News" folder. 

1. Go to the "Map" view. 
2. Find and select The New York Times home page as the root document in the "Map" view. 
3. Find the article of The New York Times you read which is titled "How to Separate Good Data from Bad". 

1. Open Search dialog box. 
2. Search on all pages from The New York Times. Select Calendar view as a result view. 
3. Change the view to "Week" and determine which day (not Friday and not Wednesday) it was when you visited The New York Times. 

1. Ask test user to sign the informed consent 
2. Explain the purpose of the test 
3. Ask test user to fill out the demographic questions 
4. Introduce briefly about the application 
• Elements of the interface (menu bar, tool bar, folders, views) 
• Main, Map and Calendar views
• Limitations of the current interface 
• Java is slow
• drag & drop is not implemented 
5. Introduce the user test procedure
• How to do the test - think aloud 
6. Conduct test
• A. Simple adding, saving and sorting bookmarks and folders 
• B. Browsing with History Map view 
• C. Power search history records in calendar view 
7. Follow-up interview (questions to fill out + brief interview) 
8. Thank the test user! 

3. Test Measures

Scenario A tests the general interface, measuring how effectively common tasks such as creating bookmarks and moving them around are supported. To create a useful system, it is extremely important that the users have no difficulties in performing their most common tasks. Consequently, the task time and number of errors measured in this part are of particular importance. 

Scenario B examines the usefulness of the Map View. The most significant observations here are to which extent the users understand the 'Map' concept, and whether or not they are able to effectively navigate through the Map interface. As the Map View is as of yet less than perfectly implemented (due to time constraints), it is probable that the users will encounter certain difficulties in its use. Still, we believe that the users' input will give us an indication of how to improve this part of the system. 

Scenario C gives us an evaluation of the Calendar View. As with the previous scenario, the most important observations are the users' understanding of the concept and the navigation of the interface. In addition, we examine how useful the users find our Search feature - this is another feature that will probably see rather extensive use, and it is therefore important that it runs as smoothly as possible. 

4. Results

The purpose of Scenario A was to test the Webmarking interface against simple and common user tasks. Naturally, it is critical in this scenario that users perform these tasks in the least amount of time with the least amount of errors. In this scenario 2 out of 3 users performed the tasks without any errors at all and one user made two errors. Scenario A consisted of four main tasks in which users successively progressed in task completion time. The first three tasks show that users were divided across quick and moderate, with the exception for one long value in the first task. Finally, all three users performed the last task, moving files to the new folder, in a relatively quick amount of time. 

The purpose of Scenario B was to test the intuitiveness and usefulness of Webmarking's Map view. The number of error results seem to be rather skewed for this scenario: 1/3 made no errors, 1/3 made two errors and 1/3 made one error when performing the Map view tasks. Again, like the number of errors, the first task completion times for this scenario show that each user had either a quick (1/3), moderate (1/3) or long (1/3) completion time. However, unlike Scenario A, user gradually had a more difficult time in completing the tasks. Response time increased gradually to the point that all users performed the last task, find a particular page in the map, in a rather long time. 

The purpose of Scenario C was to test the effectiveness of the Calendar view in searching and retrieving history records. The number of errors in this scenario was equal to those in the first scenario; 2/3 had no errors and 1/3 committed two errors. None of the users took a long time in completing the tasks for Scenario C. In fact tasks were completed in a relatively quick amount of time. This excludes the second task in this scenario in which all users required a moderate amount of time for completion. Nonetheless, the task completion time results seem to have leveled off in Scenario C. 

After performing the scenarios the users answered a follow-up survey about their impressions of the Webmarking user interface. Based on the survey results (for details see "Follow-up Survey Results" in the Appendix) 2/3 felt the interface was relatively easy. 2/3 rated the intuitiveness of the icons a 4 out of 5 (5 being very intuitive). In general a majority (2 out of 3) of the users felt that the features provided by the Wemarking interface, however, the Map View seemed to result in the lowest average score. It was not rated as high as the other features. And finally, all of the users liked the fact that search results could be viewed in either the map, calendar and tree views. 

5. Discussion

The general interface seemed to function adequately - most tasks were easily and quickly completed. One of the issues that did arise was the lack of redundancy in our interface widgets - it should be possible to initiate a certain function in several ways (pull-down menu, icon, keyboard shortcut etc.). Some testers seemed to immediately jump to the pull-down menus when looking for a function, while others started by browsing the menu bar for appropriate icons. Preferably, both kinds of users should be able to find the required function through their widget of choice. Similarly, keyboard shortcuts should be available, as power users sometimes tire of the 'slow, inefficient' icons and menus.

Another issue is the extent to which dialog boxes should be utilized. One test user indicated that a dialog box should appear when she clicked 'New Bookmark', prompting for a title and a description. Others, though, preferred the current interface (with no dialog box); they would rather set the title and description at a later point than be bothered with a dialog box every time they added a bookmark. Obviously, it should be possible to set the personal preferences to enable or disable such dialog boxes - still, however, most users would probably go with whatever is the system's default choice. As the pilot test is inconclusive, only a broader study would be able to tell us which way to go in this matter.

As we might have expected, the users found our 'Map View' to be somewhat confusing (as the follow-up questionnaire reveals, it was their least favorite feature). First of all, they found it rather difficult to navigate, even though we only have a minimum of icons to navigate through (<10, as compared to a more realistic >100). We do believe that this is largely due to the limitations of our prototype - designing a functional and aesthetic Map View has proven to be quite complicated and time-consuming. Still, part of the blame must be given to the fact that the icons clearly do not display enough information about their underlying Web pages. Obviously, this problem will need to be addressed.

Beyond the navigation difficulties, some users also had problems with understanding the basic concept behind the Map View. Arguing that "the way people navigate the web is not really sequential", they wonder how the system will interpret people moving back and forth between Web pages, or 'teleporting' to a page using a bookmark instead of a in-page link. Though we believe our Map system has no conceptual flaws of this kind (jumping back and forth is no problem as the system will always link a new page from the referring page; 'unrelated' pages will simply show up as root nodes in the map), it is clear that our testers found it confusing. We think this problem is to some extent related to the navigation problem - if the users were able to grasp the navigation of the Map View more firmly, we believe they would also get a better understanding of the way this feature works behind the scenes. For a real user test, we would have to improve the Map View functionality by displaying more information and providing more obvious links between the icons. Provided that our hypothesis is a correct one, we would then see a broader understanding of this feature among our users.

Finally, our Calendar View was arguably our most popular feature. The main problem with this function was the navigation between month, week and day view - our current prototype cheats by automatically zooming in on the correct week or day. Clearly, we need a way of letting the users themselves specify which week or day to zoom to. Also, we lack a feature for navigating between months (usually, a history file will be deleted well before a month has passed, but for completeness' sake month-to-month navigation should still be possible).

6. Formal Experiment Design 

Hypotheses

We believe the best way to exploit the properties common to both bookmark and history files is to display all results and information using one window. This approach makes it possible to use bookmarks and history as one large body of uniform data. The one window interface will make the application more efficient, as a larger part of the workload involved in combining the sources can be done by the application instead of the user.
 

Independent variables

The experiment would involve changing how distinct the interface should present bookmarks and history files. The current design presents bookmarks and history uniformly, alternative interfaces would present the  two types of data with a varying degree of independence. Using only two levels, a one-window and a two-window interface, would probably provide most of the information needed to test the hypothesis.
 

Response variables

Webmarking is meant as a tool for locating information. Hence, what's important is how efficient the users are able to locate whatever they're looking for. The most important response variable would be the time it takes to find a specific web page. Other would include how fast the user learns to use the interface or simply the degree of satisfaction involved.
 

Blocking and Repetitions

We would have all test users test both the current interface and a two-window interface.  Several tasks would be performed using each interface, to ensure that we test as many aspects of the interface as possible. The users would perform similar tasks using both interfaces, but to avoid users remembering how to do a task from testing the other interface, the tasks should not be identical. Similarly, we would use data sources that are slightly different from one interface to another. Fifty percent of the users should test the current interface first, the other half the two-window interface.

7. Appendices

• Test Procedure
• Task Instructions
• Demographic Questions
• Follow-up Survey
• Informed Consent
• Test Logs
• Test Measure Results
• Follow-up Survey Results