Formal Experiment Design

Title: Effects of Text and Icons on Calendar Manipulation

 ..................................................................................................................................

Problem Statement

In our pilot usability testing, we indentified that users perceive and interact with interfaces differently depending on their cognitive style. For example, one of our users (who has a architecture background) not only immediately recognized the icons of MySFnight without looking at the legend, but also mentioned how comfortable he was with that icons. However, at the same time he could not find where to sign up, probably because the text associated with the task was not associated with an icon. He felt completely confused looking for a clue leading to sign up. Another tester needed to read each line to perform the tasks and the icons did not have an immediate impact.

Research questions: These different behaviors introduced an interesting problem: Does MySFnight provide enough feedback for different cognitive styles? Is it becoming too visually centered, to a point of making it difficult for text oriented users to understand immediately its use and potential value or to learn the use easily? Would a more text-oriented calendar result in a better performance for more users or would it decrease the level of usability for visual users that the same percentage of users would still find the site difficult to use?

In this study we are interested in measuring user satisfaction with two kinds of calendars as the user interface: MySfnight as presented in the third interactive prototype, and another text-oriented design.

For this experiment, users satisfaction can be measured by (a) task completion time and (b) usefullness of the information provided as rated by users. The user interacts with visible objects like icons, text, and other elements of information architeture. If designed effectively these objects can help users to achieve better performance time when completing specific tasks involving actions like browsing and manipulation of information. The speed of task completion can be interpreted as the understandability and usability of the user interface. The usefulness of the information presented is determined by how relevant and adequate the information is when helping users achieve their goals.

A good design, according to A. Norman in his evaluation of everyday things, is a result of architecture visibility (presenting alternative to users), a good conceptual model (consistency in the presentation), natural mapping (users don't need to decode the interface elements) and feedback (users know where they are, where they go, how to go, what they got) and could minimize the differences between individuals.

By mapping and understanding users interaction with two different user interfaces we might be able to accomplish the goal of any good design that is to be cognively comprehensible and effectively acceptable by all users.

The Experimental Testing

This formal testing is a between-subjects design: two group of subjects will each perform three tasks in one interface, not both. We will assign them randomly.

Method: two groups with different cognitive style will come to the lab to use two interfaces and perform tasks that are assigned randomly to groups. Some will use MySfnight- 'visual', and some MySfnight-'text'. To guarantee statistically significant results we will need 60 subjects to perform this experimental design,

Hypotheses

1. Users with a more visual cognitive style might find MySfnight-visual centered (Prototype 3) more comfortable when performing the tasks.

2. Users with a more text-oriented cognitive style might find MySFnight-text centered (a new design to be tested) more comfortable when performing the tasks.

3. Users will not be comfortable with either of the interfaces, because the problem of manipulating two calendars is more significant than the two styles of communication.

Factors and Levels (Independent Variables)

Two levels: two user-interfaces for events calendar of MySFnight: MySFnight-visual oriented (Interactive Prototype 3) and MySFnight-text oriented (another prototype to be created)

2nd Independent variable: visual cognitive style.

Factors: 2 x 2 design (two interfaces + two groups: a group with visual cognitive, a group more text oriented)

Response Variables (Dependent Variables)

Dependent variables:

• The manipulation of information (objects)
• Readability of the information
• Information organization
• Information usefulness
  

Measures:

• Time to create "Your Picks Calendar" [users will be asked to find specific events at Editors's Picks and add to their calendar].
• Time to remove events of the calendar.
• Completion of the task
• User satisfaction questionnaire (pos-task & pos-test)
• Log data (number of errors and style)

Possible Counfounding Variables:

• Font and size of text: we will consider them constant.
• Constrast/size/ color of screen and hardware configuration, may interfere with icon and text resolutio. We will control this by using the same equipments.
• The cognitive style: we will control the two groups by first applying a psicological test to have an approximation of their style. We will then use this difference as independent variable.

^

 

Blocking and Repetitions

30 people x 3 tasks = 90 per interface

subjects	tasks by interface
1	T1	T2	T3
2	T2	T3	T1
3	T3	T1	T2
4	T1	T2	T3
5	T2	T3	T1
6	T3	T1	T2
...	...	...	...
28	T2	T3	T1
29	T3	T1	T2
30	T1	T2	T3

 

Variance Control

In order to achieve valid inferences and to minimize the effect of systematic variance, we considered the following methods:

control of extraneous source of variance:

• randomization of tasks: to statistically equalize the learning effect.
• elimination of secondary sources of variance, such as familiarity with the site tested, we will choose users who have no previous experience with the interfaces.
• statistic control: related to experience in using the Internet, nighlife events, and their cognitive style. To address the difference in cognitive style, we will perform a test with a psychologist's assistance to identify two different groups according to their cognitive style

validity: To minimize the possibility of error variance affecting the satisfaction results, regarding information organization, we plan not to include subjects with human-computer interaction design background. Their ability to judge would be biased by their skills and experience. We will also do the testing in a computer lab to control Internet conection speed and computer settings. We will use a log browser, such as ergoBrowser, to log clicking and navigation activities performed by users.

^