IS214. Needs and Usability Assessment

School of Information Management & Systems, University of California Berkeley

Updated 7/30/03


Prof. NANCY VAN HOUSE
vanhouse@sims.berkeley.edu

TuTh 12:30-2:00 PM

202 South Hall
Office Hrs: Tues 2:45-4 and by arrangement.

ASSIGNMENTS

PROJECTS

RESOURCES

***

WEEK BY WEEK

Requirements: Collecting Data from and About Users
Week 2 Ethnographic Methods
Week 3 User and Task Analysis
Week 4 Measurement and Evaluation
Week 4b Interviews and Focus Groups (I)
Week 5a
Interviews and Focus Groups (II)
Week 5b
Qualitative Data Analysis and Presentations
Week 6 Surveys
Week 7 Quantitative Data Analysis and Presentation

Prototype Evaluation
Week 8 Design Guidelines, Heuristics, and Inspection Methods
Week 9 Usability Testing

Spring Break

Week 10a
Week 10b NO CLASS

Post-release Evaluation
Week 11a Writing Usability
Reports

Week 11b
Credibility

future

Monitoring, User Satisfaction Surveys, User Feedback
Universal Usability
Ethics
, Offshore Usability
Project Presentations

 

***

added 4/8/03 RE FINAL PROJECTS: Due date for final project: Monday, May 12.
I will review drafts of final reports if you wish. When I need to receive them depends on how many I get; May 2 is target date for turning in drafts.

Writing is easy. All you do is stare at a blank sheet of paper until drops of blood form on your forehead.-- Gene Fowler as quoted by Blade Kotelly in The Art and Business of Speech Recognition, in Ubiquity, Volume 4, Issue 7, April 9 - 15, 2003.

 

Class Schedule

Go directly to the current week

Note: links that are under "Readings" are required; those under "Links and Recommended Readings" are recommended. Documents from required links may or may not be in reader; some are not suitable for printing and putting in reader.
INTRODUCTION AND OVERVIEW

Week 1.
Jan 21 & 23

 

Readings

copies are outside room 305A South Hall

Vredenburg, K., Isensee, S., & Righi, C. (2002). User-centered design: an integrated approach. Upper Saddle River, NJ: Prentice Hall PTR: ch 1, Taking Stock, Ch. 2, The Integrated Approach.

Preece, J., Rogers, Y., & Paine, L. (2002). Interaction design: beyond human-computer interaction. New York: J. Wiley & Sons. ch. 1: What is interaction design?

Preece, J. & Benyon, D. (1993). A Guide to usability: human factors in computing. Wokingham: Addison-Wesley. ch. 6: Evaluation.

Read before Thursday (handed out in class):

Assignments & activities

Begin identifying final project and project team

Assn 1 handed out; due Tues 28th (not Thurs 23rd)

Look at interactive methods table from UsabilityNet

Scan some of the key usability web sites to get a sense of how they define the field: usability.gov, http://www.upassoc.org/, http://usableweb.com/ (old, but still pretty good), http://www.stcsig.org/usability/

Links, Recommended readings

IBM's Web Design Guidelines -
-planning
- short, concise, useful overview on starting a project, looking at users, competitive and market analysis, strategy, content, resources.
-phases - lays out steps of user-centered design project.

STC Usability SIG Topics in Usability

Class slides added 1/25

COMPETITOR ANALYSIS
Readings:

Bunnyfoot - Competitor analysis: an alternative approach.

IBM's Web Design Guidelines - Competitive and Market Analysis

Note: This segment will not discussed in class, but you need to do these readings sooner or later for your project.

SEGMENT I: REQUIREMENTS: COLLECTING DATA FROM AND ABOUT USERS

Week 2.
Tues. Jan 28: Speaker
Thurs. Jan. 30: Ethnographic Methods

 

Guest Speaker
Tuesday, Jan 28

 

Anna M. Wichansky
Senior Director, Advanced User Interfaces,
Usability and Interface Design, Oracle Corporation

 
ETHNOGRAPHIC METHODS

Readings

Blomberg, Jeanette, & others, "Ethnographic Field Methods and their Relation to Design." Douglas Schuler and Aki Namioka, eds. Participatory Design: Principles and Practices. Hillsdale, NJ: Lawrence Erlbaum Associates, 1993; p. 123-155. Read pp. 123-147; the rest is recommended.

Blomberg, Jeanette L. "Ethnography: aligning field studies of work and system design." in Monk, Andrew F., and G. Nigel Gilbert, eds., Perspectives on HCI: Diverse Approaches. ch 8, p. 173-197. Academic Press, 1995. Read sections 8.1-8.25; the rest is recommended.


David R. Millen. Rapid ethnography: time deepening strategies for HCI field research. Conference proceedings on Designing interactive systems : processes, practices, methods, and techniques: processes, practices, methods, and techniques. ACM, 2000. From campus IP addresses: http://doi.acm.org/10.1145/347642.347763
Rapid ethnography is getting a lot of attention in HCI and design; 'real' ethnographers (like Blomberg) find it a travesty. We'll talk about how ethnography can be useful and about this controversy.

 

Assignments & activities

Assignment 1 due Tues, Jan. 28: naive usability assessment

Links, recommended readings

CHARM (Choosing Human-Computer Interaction (HCI) Appropriate Research Methods): Ethnographic Methods

Week 3.
Feb 4 & 6

 

USER AND TASK ANALYSIS
Readings JoAnn T. Hackos and Janice C.Redish. User and Task Analysis for Interface Design, New York: Wiley, c1998. $60 (H&R) NOT IN READER
ch 1, Intro to User and Task Analysis for Interface Design:
ch 2, Thinking about Users.
ch 3, Thinking about Tasks. 
ch 4, Thinking about the Users' Environment
recommended: ch 5. Making the Business Case for Site Visits

Rubin, Jeffrey, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley, 1994; ch 6, Selecting and acquiring participants. NOT IN READER.

Contextual Inquiry and Design is a structured approach to ethnography for system design. We don't have time to do this method justice, but you should know that it exists. Very brief overview: http://www.incent.com/cd/cdhow.html

Assigments & activities

Due Feb 6 (tentative): Description of proposed project

Links, Recommended readings

sample forms for site visits and user-task matrices from STC usability SIG

The key book on Contextual inquiry: H. Beyer and K. Holtzblatt. Contextual Design: Defining Customer-Centered Systems, 1998, Morgan Kaufmann Publishers. More info on their site :http://www.incent.com/cd/cdhow.html

Class slides added 2/06

Week 4a.
Tues. Feb 11
Interlude: MEASUREMENT & EVALUATION

Readings

Babbie, Earl R. (2001) The Practice of Social Research, 9th ed., Wadsworth. Chapter 5: Conceptualization, operationalization, and measurement (or equivalent chapter from another social research methods text).

Robert Opaluch, "Usability Metrics". In Ratner, Julie, ed. Human Factors and Web Development, 2nd ed. Lawrence Erlbaum, 2003. p. 101-144.

Weeks 4b & 5a.
Thurs. Feb. 13
Tues. Feb. 18
INTERVIEWS AND FOCUS GROUPS

Readings

H&R ch. 7-10

David L. Morgan, Focus Groups as Qualitative Research, Sage, 1988; pp. 53-71, Conducting and Analyzing Focus Groups.

Krueger, Richard A., Focus Groups, 2nd ed. Sage, 1994; "Asking questions in a focus group." pp. 53-69
.

Assigments & activities Interviewing assignment
Links, Recommended readings

IS208 readings on these topics - many IS214 students are also taking, or have taken, IS208.

UC Berkeley Committee for the Protection of Human Subjects: http://cphs.berkeley.edu:7006/

Class slides interviews and focus groups

Week 5b.
Thurs. Feb 20

QUALITATIVE DATA ANALYSIS AND PRESENTATION, INCLUDING
SCENARIOS AND PERSONAS

Readings

H&R ch. 11 - you don't need to master all the methods they discuss, but be familiar enough with them to know what's possible.

Rosson & Carroll, Usability Engineering, ch. 1 - some of this is review, but pay attention to how they define scenarios.

Cooper, A. (1999). The inmates are running the asylum. Indianapolis: Sams. ch 11: Designing for people.

Grudin, Jonathan and Pruitt, John. Personas: Practice and Theory. 2002.

Assigments & activities  
Links, Recommended readings

Carroll, Making Use, ch. 3, Scenario-based Design. This book is more reflective and abstract than Rosson & Carroll; Rosson & Carroll is more of a how-to.

What is a scenario? from Information & Design.

Personas: articles from Cooper Interaction Design: Kim Goodwin, Perfecting Your Personas; Elan Freydenson, Bringing Your Personas to Life in Real Life

Class slides added 2/19

Where We Are added 2/18

Week 6.
Feb 25 & 27

SURVEYS
Readings

Surveys are used for a wide variety of purposes. User surveys are probably most useful as post-implementation evaluation, but they are also used for collecting pre-design information from users. We'll talk about principles of survey design and administration here, then revisit the topic periodically as we talk about other usability topics.

Dillman, Don A. (2000) Mail and Internet Surveys: the Tailored Design Method, 2nd ed. Wiley. chs. 2-3; 5. The rest of the book is recommended.

Web-based surveys:
Dillman, Don, and Dennis K. Bokwer. The web questionnaire challenge to survey methodologists.

Using surveys in usablity:
Good overview
of user surveys in pre-design stages -- Jarrett and Bachmann STC workshop. (not in reader)

Assigments & activities Survey creation
Links, Recommended readings

A variety of survey resources can be found here

slides

Week 7a.
Tues. Mar 4

 

 

DATA ANALYSIS AND PRESENTATION #2: QUANTITATIVE DATA
Readings

TBA

Assigments & activities Survey data analysis
Links, Recommended readings


SEGMENT II: PROTOTYPE EVALUATION

Week 8.
Tues. Mar 11,
Thurs. Mar 13

DESIGN GUIDELINES, HEURISTICS, AND INSPECTION METHODS

Readings

Nielsen, Jakob, "Heuristic Evaluation," in Nielsen and Mack, Usability Inspection Methods, Wiley and Sons, 1994: chs 1 &r 2

Nielsen's 10 Usability Heuristics

Nielsen & Tahir, Homepage Usability: 50 Websites Deconstructed, New Riders Publishing, 2002.:
ch 1, p. 7-33 homepage design guidelines
ch 2 - p. 37-53 homepage des
ign stats

Standards and guidelines - a set of link from STC. Excellent source.
IBM Web Design Guidelines
Web Credbility Guidelines, Stanford Web Credbility Project

W3C Web Content Accessibility Guidelines 1.0 (in reader in Universal Usability section)

Assigments & activities Create and administer heuristics
Links Sample heuristics report from Barnum
Sample checklist from InfoDesign
Sample test materials from STC usability SIG: scroll down to Usability Checklists and Heuristic Reviews; include process for heuristics review and sample checklists 1 2

Instone's Web usability heuristics

slides

Week 9.
Mar 18 & 20

 

USABILITY TESTING
Readings

Rubin, Jeffrey. Handbook of Usability Testing (not in reader)
ch. 2, Overview of usability testing
ch. 5, Developing the Test Plan
ch. 7, Preparing the Test Materials
ch. 8, Conducting the Test
The rest of the book is recommended.

Dumas, Joseph S. and Janice C. Redish. A Practical Guide to Usability Testing:
ch. 11, Selecting and Organizing Tasks to Test
ch. 13, Deciding How to Measure Usability

Assigments & activities perform a usability text (added 3/13/03)
Links, added readings

Carol Barnum, Usability Testing and Research, , $51.33
not as thorough as Rubin but covers more ground concerning usability in general.

Good summary based on Rubin
Common Industry Format for Usability Test Reports
Sample reports: DailogDesign; from Barnum

Sample test materials from STC usability SIG
usability labs from STG usability SIG

REMOTE usability testing:
OCLC, How we do it: remote usability testing
Tom Tullis and others. An Empirical Comparison of Lab and Remote Usability Testing of Web Sites.
Commercial solutions recommended by a group of practitioners: Webex, Sametime, and GoToMyPC

SPRING BREAK March 24-28
POST-RELEASE EVALUATION

Week 10a.
Tues. April 1

Catch-up

Readings  
Assigments & activities  
Links

 

Week 10b.
Thurs. April 3

NO CLASS

Readings  
Assigments & activities  
Links

 

Week 11a.
Tues. April 8

WRITING USABILITY REPORTS

Readings Common Industry Format for Usability Test Reports

AskTog, 2001: How to Deliver a Report Without Getting Lynched

Sample reports

Assigments & activities We will review a number of sample reports from outside sources for their strengths and weaknesses.
Links  

Week 11b.
Thurs. April 10

CREDIBILITY

Readings

Evaluating Web Pages, UC Berkeley Library (not in reader)

Web Credbility Guidelines, Stanford Web Credbility Project

How Do People Evaluate a Web Site's Credibility? Results from a Large Study Skim the full report and then read all of Part 1: Results & Discussion — Overall Analysis of Credibility Comments

Class Slides (TRUST)

Assigments & activities  
Links

 

 

MONITORING, USER-SATISFACTION SURVEYS, USER FEEDBACK

Readings

Look at http://usableweb.com/topics/000649-0-0.html and http://usability.gov/serverlog/index.html. Review their content and several of the papers linked from each site to get a sense of what can be done with server logs.

Assigments & activities  
Links Look at some user satisfaction surveys (note that many are proprietary and only provide glimpses of what they ask):
WAMMI
SUS — "A 'quick and dirty' usability scale" (1986)
Week 12.

Data Analysis

Tue. April 17

Thurs April 19
Readings

TBA

Assigments & activities  
Links

slides

MANAGEMENT ISSUES, OTHER TOPICS
Week 13.

UNIVERSAL USABILITY

Tue. April 22

Guest Speakers

Ward Newmeyer
Campus compliance officer for the Americans With Disabilities Act
UC Berkeley

Adam Tanners, Disabled Students Program, UCB

Readings

Ratner, Julie, ed. Human Factors and Web Development, 2nd ed. Lawrence Erlbaum, 2003:
-Andrew Sears, "Universal Usability and the WWW." In p. 21-45.
-Masaaki Kurosu, "A Cultural Comparison of Website Design from a Usability Engineering Perspective."

Nielsen, J. (2000). Designing Web usability. Indianapolis: New Riders. ch 6: Accessibility for users with disabilities. ch. 7: international usability.

W3C Web Content Accessibility Guidelines 1.0 (in reader)

Web Accessibility Links

Thurs April 24

Readings on international usability issues:
-doing international testing
-designing for international usability

Jacob Neilsen, International Web Usability; International Web Usability Testing; Offshore Usability

Tips on international usability testing

Cross-cultural Web Site Design

Cross-cultural applicability of user evaluation methods: a case study amongst Japanese, North-American, English and Dutch users, CHI '02
requires ACM DL access -- accessibility from campus IP addresses. (Short paper)

slides

  Assigments & activities reports on projects
  Links

Section 508 standards
Bobby for evaluating web sites
Web accessibility for kids - and for seniors -- Nielsen's Alertbox
More comprehensive info about usability for the old and young from the STC usability SIG
Good set of links to accessibility info from usability.gov
Conference on Universal Usability

Accessibility resources:
Center for Accessible Technology, Berkeley
CFAT's resources link
How to design accessible sites

Week 14.

Tues. April 29

ETHICS

Readings Oliver K. Burmeister, "HCI Professionalism: Ethical concerns in Usability Engineering" (PDF) & "Usability Testing: Revisiting Informed Consent procedures for testing Internet sites" (PDF), Australian Institute of Computer Ethics Conference, 11/12/2000.

TRUSTe Model Privacy Statement. (in reader)

Look at http://usability.gov/about.html for their statements about the site and their privacy statement.
Assigments & activities  
Links slides

Thurs May 9

 

Server Log Analysis

Readings

First read http://usability.gov/serverlog/index.html, with special attention to the content of web logs.

Then review several available web log analysis tools on the web. Some sources:
http://webdesign.about.com/cs/loganalysistools/index.htm?terms=analysis+tools
http://www.analog.cx/
http://www.mrunix.net/webalizer/
http://www.netiq.com/webtrends/default.asp

several are listed at
http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/

 

Assigments & activities Take one of these tools and go through the kinds of statistics that they present to see how they are using the weblog data. Remember that the weblog is their primary source of data.
Links slides

Summary

Readings  
Assigments & activities  
Links slides

Week 15.
May 6 & 8

PROJECT PRESENTATIONS

Tues. May 6

Google News
Team: Mary Hodder, Zhanna Shamis, Diana Stepner

Wireless Networks for Lighting Control
Researcher: Rebekah Yosell-Epstein

Lin, Phoebe

Justin Makeig

 

Thurs. May 8 Internet Use in University Admissions Applications
Team: Mukesh Darke , Michelle Kim , Joyojeet Pal

CampusConnect
Team: Catherine Lai, Yin Lau

Keasha Martindill

 


May 13 - Offiical last day of instruction - Class will not meet.