Main Course Page: Detailed Course Description

Title: Quantitative Research Methods for Information Management (Focus on web usability design and testing)

Number: IS247, 3 units, CCN No: 42723 (Catalog description) (sign up for course)
Instructor: Dr. Rashmi Sinha Email:,
Time and Venue: WF 2-3:30 202 South Hall

The course is divided into two parts. Part I consists of basics of research design and statistics. Part II examines specific usability testing methods in greater depth; students will be required to do a major project. The focus of this course will be to provide students with a background in statistics and research design, giving them the basic tools to (a) choose the appropriate testing method for the current problem, and (b) implement the chosen method.

Purpose of the course: Academic quantitative methods are not used very often to solve usability problems in industry settings because (a) they tend to be time-consuming, (b) require an understanding of research design and statistics, and (c) are often difficult to implement. Therefore discount usability methods (which are mostly qualitative) have become popular. However, there are a few problems with discount methods. Discount methods are mostly ad hoc; there are no well-established standards. Also, they lead to very low agreement between different usability experts. In this course, I will try to focus on quantitative methods that retain the rigor and quantitative nature of more academic research methods, but which can implemented easily and in internet time.

Format of the course: In general it will be a fast paced course, you will be learning and doing statistical exercises. The class will also be project based. For example, when you learn about surveys, you will be asked to design, conduct and analyze data from your own surveys. You will be conducting at least four projects (one survey, one experiment/quasi-experiment, one armchair experiment, and a major final project with the problem and method of your choice)

Students will work in groups of two and more for the projects (depends on the total number of students in the class). I want to be able to give individual input to each of the groups, so the effort will be to have fewer groups.

The focus will be learning research methods and statistics by practice, rather than by reading about it.


  • Projects: Grading will primarily be based on project performance (50%).
  • Assignments: The assignments will constitute 20% of the grade.
  • Midterm: The midterm will constitute 20% of the grade. The midterm will be a take home statistics (you will get two days) exam. I will give you two data sets. You will be asked to analyze it in as many ways as possible (using SPSS) and write a short report describing your findings. The emphasis will be on your ability to use multiple methods to analyze a dataset, and know which methods are the most appropriate in the current situation.
  • Research participation: You will be required to participate (be a subject) in at least three projects of your fellow classmates. An important part of research methods is understanding the study from the participants perspective. This requirement will constitute 5% of your grade.
(If you noticed, the above adds up to 95%, I am still deciding about the remaining 5%)

Textbooks and Readings: I am deciding between two statistics textbooks:

You will also need some references for SPSS. One choice is to buy the SPSS graduate pack (comes with a student version of the software, and a SPSS help book). The second option is to only buy a book for SPSS. One book I like is Using SPSS for Windows: Analyzing and understanding data, by Green, Salkino and Akey.

And you will have some readings through the course.

Statistical background: I have designed the course assuming little or no statistical background. Students who have some background can come and speak to me individually. They might be excused from the first few lectures.
Tools one needs for usability testing: (a) software to analyze data (SPSS) (b) software to run experiments (e.g., to record keystrokes during task) (e.g., WebVIP) (c) software to conduct surveys (e.g., NetRaker or Epoll)
Course Description
Note. In describing the course below, at the end of each section, I have posed questions which students should be able to answer at the end of the section. The questions are often posed as specific situations that students might face in the course of research or while working in industry.

Part I: An introduction to various quantitative methods and the appropriate statistical analyses

Overview of design and analysis:

  • Posing a usability question
  • Conceptualizing the question
  • Operationalizing the related concepts
  • Identifying the Independent, Dependent, and Controlled Variables
  • Developing the Hypothesis

Question: I have been asked to test the usability of web retail site. How should I begin to test it? What aspects of the site should I test?


Choosing the testing method (experiment, observation, surveys etc.)

  • What kind of method is appropriate for the current situation?
  • Choice of testing method as a tradeoff between control and realism (often, the more controlled the study is, the less realistic it is)
  • Experimental, Quasi-Experimental and Non-Experimental Methods.
  • Other aspect of testing: Ethical issues in using human subjects.
  • Informed Consent and Debriefing.

Question: What testing method should I use? What are the pros and cons


Collecting data

  • The art of finding and recruiting participants (taking into account Random Selection and Random Assignment)
  • A practical view of randomization: Randomization and Pseudo Randomization.
  • Practical issues about sample size and statistical power.
  • Developing a participant database.

Question: How will I find participants for my studies who are representative of the product's typical users? How should I recruit them and assign them to various experimental conditions? How many people should I test?


Preliminary Analysis of data

  • Basic statistics
  • Levels of measurement: nominal, ordinal, interval, and ratio
  • Mean, median, standard deviation
  • Parametric and non parametric statistics
  • Testing mean differences, significance levels and what they mean
  • Type 1 and Type II errors
  • Graphical representation of data

Analysis of experimental designs

  • Single Factor Experiments:
  • Statistical Hypothesis Testing
  • Estimates of Experimental Error
  • Estimates of Treatment Effects
  • Evaluation of the Null Hypothesis
  • Various ANOVA models

Multi-Factor Experiments

  • Advantages of the factorial design
  • Interaction Effects
  • The two factorial experiment
  • Higher Order Factorial Designs
  • Standard Analysis for Higher Order Factorial Designs
  • Simple Effects
  • Interpreting Interactions
  • Other possible designs: e.g., Latin Square Designs

Question: My company is redesigning their web-site to support faster navigation. I have been asked to compare the old and new designs to find out which supports faster navigation. What experiments can I run to find this out?


Analysis of Non Experimental Studies

  • Statistical methods for analyzing correlational data
  • Correlations, Scatter Plots
  • Multiple Regression
  • Brief Introduction to Factor Analysis, Cluster Analysis and Multidimensional Scaling

Surveys and Questionnaires

  • The design of surveys and questionnaires
  • How to frame questions
  • Kinds of scales: Likert, Semantic Differential etc.
  • Analyzing survey data

Question: I need to design a questionnaire to test users' satisfaction with the personalization features for our website. How should I go about doing this?

Part II: A closer look at some usability testing methods and Group

GOMS analysis

  • What is GOMS analysis,
  • The different flavors of GOMS analysis.
  • A keystroke level GOMS analysis.

Question: Registered users can personalize their stock portfolios on my company's website. However, the method to create the personalized portfolio takes too long and is prone to errors. I have been asked to try to reduce the average time and errors in creating a portfolio. How can I achieve this goal?


Information Architecture

  • Inferring the mental models of users for designing large-scale websites.
  • Methods: Card Sorting and Similarity Ratings.
  • Statistical Analysis: Cluster Analysis and Multidimensional Scaling.

Question: I have been asked to restructure a large consumer website. How should I develop the product categories so that it will be easy for users to find items on the site?


Measuring Individual Differences

  • How to find out if there are significant individual differences between groups of users.
  • What kind of individual differences variables might exist: demographic variables such a age, sex etc. situational variables such as motivation, level of interest, fatigue etc., cognitive variables such as memory, cognitive style etc.
  • How to analyze existing data to identify individual differences?
  • How to design studies to test for individual differences?
  • Item Response Theory

Question: I suspect that my company's product is heavily biased towards expert users, How should I demonstrate this? I think that the women visitors do not like a particular website. How can I find out if this hypothesis is true?


Automated Web Usability Evaluation

  • What are the various kinds of automated methods in web usability evaluation (e.g., automatic capture of usage data, Log file analysis)?
  • What kind of usability problems can such methods help identify?

Steps in the Group Project

  • Identify an interesting usability problem
  • Conceptualizing the problem in terms of more specific usability questions
  • Operationalizing the concepts
  • Choose at least two method of testing, develop the testing method
  • Recruiting and testing subjects
  • Analyze data, write research reports.
During each stage of the project, the groups will give short presentations. This allows other groups to get exposure to the complete research process for each testing method.