|
Course
Description
|
| Note.
In describing the course below, at the end of each section, I have
posed questions which students should be able to answer at the end
of the section. The questions are often posed as specific situations
that students might face in the course of research or while working
in industry. |
Part I:
An introduction to various quantitative methods and the appropriate
statistical analyses
|
|
Overview
of design and analysis:
- Posing a
usability question
- Conceptualizing
the question
- Operationalizing
the related concepts
- Identifying
the Independent, Dependent, and Controlled Variables
- Developing
the Hypothesis
Question:
I have been asked to test the usability of web retail site. How
should I begin to test it? What aspects of the site should I test?
|
| |
|
Choosing
the testing method (experiment, observation, surveys etc.)
- What kind
of method is appropriate for the current situation?
- Choice of
testing method as a tradeoff between control and realism (often,
the more controlled the study is, the less realistic it is)
- Experimental,
Quasi-Experimental and Non-Experimental Methods.
- Other aspect
of testing: Ethical issues in using human subjects.
- Informed
Consent and Debriefing.
Question:
What testing method should I use? What are the pros and cons
|
| |
|
Collecting
data
- The art
of finding and recruiting participants (taking into account Random
Selection and Random Assignment)
- A practical
view of randomization: Randomization and Pseudo Randomization.
- Practical
issues about sample size and statistical power.
- Developing
a participant database.
Question:
How will I find participants for my studies who are representative
of the product's typical users? How should I recruit them and assign
them to various experimental conditions? How many people should
I test?
|
| |
|
Preliminary
Analysis of data
- Basic statistics
- Levels of
measurement: nominal, ordinal, interval, and ratio
- Mean, median,
standard deviation
- Parametric
and non parametric statistics
- Testing
mean differences, significance levels and what they mean
- Type 1 and
Type II errors
- Graphical
representation of data
|
| |
|
Analysis
of experimental designs
- Single Factor
Experiments:
- Statistical
Hypothesis Testing
- Estimates
of Experimental Error
- Estimates
of Treatment Effects
- Evaluation
of the Null Hypothesis
- Various
ANOVA models
|
|
Multi-Factor
Experiments
- Advantages
of the factorial design
- Interaction
Effects
- The two
factorial experiment
- Higher Order
Factorial Designs
- Standard
Analysis for Higher Order Factorial Designs
- Simple Effects
- Interpreting
Interactions
- Other possible
designs: e.g., Latin Square Designs
Question: My
company is redesigning their web-site to support faster navigation.
I have been asked to compare the old and new designs to find out
which supports faster navigation. What experiments can I run to
find this out?
|
| |
|
Analysis
of Non Experimental Studies
- Statistical
methods for analyzing correlational data
- Correlations,
Scatter Plots
- Multiple
Regression
- Brief Introduction
to Factor Analysis, Cluster
Analysis and Multidimensional Scaling
|
| |
|
Surveys
and Questionnaires
- The design
of surveys and questionnaires
- How to frame
questions
- Kinds of
scales: Likert, Semantic Differential etc.
- Analyzing
survey data
Question: I
need to design a questionnaire to test users' satisfaction with
the personalization features for our website. How should I go about
doing this?
|
| |
| Part II:
A closer look at some usability testing methods and Group |
| |
|
GOMS analysis
- What is
GOMS analysis,
- The different
flavors of GOMS analysis.
- A keystroke
level GOMS analysis.
Question:
Registered users can personalize their stock portfolios on my company's
website. However, the method to create the personalized portfolio
takes too long and is prone to errors. I have been asked to try
to reduce the average time and errors in creating a portfolio. How
can I achieve this goal?
|
| |
|
Information
Architecture
- Inferring
the mental models of users for designing large-scale websites.
- Methods:
Card Sorting and Similarity Ratings.
- Statistical
Analysis: Cluster Analysis and Multidimensional Scaling.
Question:
I have been asked to restructure a large consumer website. How should
I develop the product categories so that it will be easy for users
to find items on the site?
|
| |
|
Measuring
Individual Differences
- How to find
out if there are significant individual differences between groups
of users.
- What kind
of individual differences variables might exist: demographic variables
such a age, sex etc. situational variables such as motivation,
level of interest, fatigue etc., cognitive variables such as memory,
cognitive style etc.
- How to analyze
existing data to identify individual differences?
- How to design
studies to test for individual differences?
- Item Response
Theory
Question:
I suspect that my company's product is heavily biased towards expert
users, How should I demonstrate this? I think that the women visitors
do not like a particular website. How can I find out if this hypothesis
is true?
|
| |
|
Automated
Web Usability Evaluation
- What are
the various kinds of automated methods in web usability evaluation
(e.g., automatic capture of usage data, Log file analysis)?
- What kind
of usability problems can such methods help identify?
|
| |
|
Steps in
the Group Project
- Identify
an interesting usability problem
- Conceptualizing
the problem in terms of more specific usability questions
- Operationalizing
the concepts
- Choose
at least two method of testing, develop the testing method
- Recruiting
and testing subjects
- Analyze
data, write research reports.
|
| During each
stage of the project, the groups will give short presentations. This
allows other groups to get exposure to the complete research process
for each testing method. |
| |
| |
| |
| |
