go to UC Berkeley home page go to SIMS home page
SIMS 214 - Needs Assessment and Evaluation of Information Systems

Last updated April 04, 2000 12:30 am
Jan 17 & 19 No class - MLK Birthday  Introduction/overview
Jan 24 & 26 Overview - Needs & Evaluation Evaluation of content: Trust
Jan 31 & Feb 1  Trust (Continued) California Digital Library Presentation
Feb 7 & 9 CDL Heuristics
Feb 14 & 16  Heuristics; Disabilities Task analysis; Scenarios
Feb 21 & 23  No class - Presidents' Day Walk-thrus
Feb 28 & 
Mar 1 
Usability testing, Principles of measurement & evaluation, Experimental design 
Mar 6 & 8  Usability testing (Continued) Surveys and Interviews
Mar 13 & 15  Focus groups Guest speaker: Kevin McBride / IMS User Centered Design / IBM Santa Teresa Laboratory
Mar 20 & 22  Field observation and Ethnography
March 27 & 29  SPRING BREAK 
Apr 3 & 5  Contextual Inquiry
Apr 10 & 12 Contextual Design
Apr 17 & 19  Logs & Monitoring Ethics of Representation.
Collecting data in cyberspace.
Apr 24 & 26 Wrap-up & Synthesis NO CLASS
May 1 & 3 Final project reports
May 8 Scheduled last class but we will not meet   


1b. Jan 19 -- Introduction to the Course


2a. Jan 24 --Needs  Assessment and Evaluation - Overview

  • Nielsen, Usability Engineering -- NOT IN READER 

  • ch 1, Exec summary
    ch 2, What is usability
    ch 4, Usability engineering life cycle
    also look at p. 224
  • Preece, Jenny, ed. Guide to Usability: Human Factors in Computing, Addison-Wesley, 1993; ch 6, "Evaluation."  
 2b. Jan 26 & 3a. Jan 31 --  Evaluation Criteria for Information Systems and Web Sites: Relevance and Trust
  • Articles about why trust and credibility are problems on the Web:  

  • NY Times 3/4/99 - Whales in the Minnesota River?
    NY Times 2/26/99 - Caveat Emptor on the Web: Ad and Editorial Lines Blur.
About how to evaluate web resources  3b. Feb 2 & 4a. Feb 7--  California Digital Library

4b. Feb 9-- Heuristics & Standards

Browse: Nielsen's Alertbox, http://www.useit.com/alertbox/
Browse: webreview.com on Usability. http://webreview.com/wr/pub/Usability.

5a. Feb 14- Disabled users

Resources for Web Accessibility: 

Check our links page for more resources on Web accesibility.


5b. Feb. 16 - Task Analysis & Scenarios
  • Fuccella, Jeanette and Jack Pizzolato, Creating Web Site Designs Based on User Expectations and Feedback. ITG  

  • Newsletter 1.1 June 1998 http://www.sandia.gov/itg/newsletter/june98/web_design.html
    Simple but useful overview of many methods.
  • Hackos, JoAnn T. and Janice C.Redish. User and Task Analysis for Interface Design. Wiley, 1998.  chs. 1-3 and p. 322-9. 
  • The following readings are now OPTIONAL instead of required. 2/14/00
    • Carroll, John, ed. Scenario-Based Design. 1995 John Wiley & Sons 
    • Kuutti, ch 1  
    • Carey and Rusli, ch 7

6b. Feb. 23 - Walk-throughs
  • Nielsen & Mack, chs 1, 3, 5, 6 Not in reader 



    Reminder: Assignment 2 is due today.

7. Feb 28, Mar 1 & 8a. Mar 6- Usability Testing / Principles of Measurement and Evaluation /Experimental design
  • Nielsen, Usability Engineering, ch 6, "Usability Testing"   Not in reader 
  • Rubin, Handbook of Usability Testing, Table of Contents (a good summary of steps of usability testing); and Ch. 2, "Overview of Usability Testing."

  • (This is an excellent resource and well worth buying if you are serious about usability testing. $45 in paper.)
  • Usability Testing" From Behavior and Information Technology 13: 1-2, Jan-April, 1994, Special issue on usability labs:  

  •           Blatt, Louis, and others, "Designing and equipping a usability laboratory." 
              Faith, Janet L. and others, "A practical guide to using software usability labs." 
              Bevan, Nigel, and Miles MacLeod, "Usability measurement in context." 
If you are not familiar with measurement (especially reliability and validity), sampling, and/or social science research methods, read about them in any quantitative social science research methods text. We will put on informal reserve within South Hall the following:  (Not in reader -- on informal reserve in the Computer Lab)
Singleton, Straits, Straits, and McAllister. Approaches to Social Research. Oxford Univ Press, 1988:
CH. 4, Elements of Research Design

Ch. 5, Measurement
Ch 6, Sampling
Also recommended: Joseph S. Dumas, Janice C. Redish. A Practical Guide to Usability Testing, Ablex, 1993. 
If you are unfamiliar with experimental design, read -- not in reader:
Singleton et al -- or another social science research methods text  Not in reader
ch 7 - Experimentation
ch 8 - Experimental design
Other useful resources:
Microsoft - Usability Home Page - Information about facilities and examples of their work.
Sun Usability Lab - Information on sevices including equipment and staff cost and a very good summary on usability engineering general -- according to last year's students. 

Society for Technical Communication - Their usability toolkit has guidelines and sample forms for usability testing.
8b. Mar 8 & 9a. Mar 13 - Surveys and Interviews 
  • Singleton, Straits, Straits, and McAllister. Approaches to Social Research. Oxford Univ Press, 1988. (on informal reserve within South Hall; not in reader)  ch 9, survey research; ch 10, survey instrumentation.  If you plan to do surveys and/or interviews, you need to learn the basics about survey research and interviewing, and then how it can be adapted to your situation.  Most social science research texts cover these topics, and there are many books out that treat them in depth.  Nielsen on web surveys -- "Collecting Feedback from Users of an Archive."   http://www.useit.com/alertbox/990110.html

  • Schmidt, WIlliam C.  "World Wide Web Survey  Research: Benefits, Potential Problems, Solutions." Behavior Research Methods, Instruments, and Computers 29 (2) 1997 p. 274-9. 
Also check these Online Resources for Survey Design and samples.
9b. Mar 15 - Focus Groups
  • David L. Morgan, Focus Groups as Qualitative Research, Sage, 1988; p. 53-71.  
  • Krueger, Richard A. Focus Groups, 2nd ed. Sage, 1994; 

  • " Asking questions" p. 53 -69. 
If you plan to use focus groups, you need to read one or both of Morgan and Krueger in or something similar - Sage publishes a number of useful books which you can get from the library or order from www.sagepub.com
  • Sullivan, Patricia (1991) Multiple Methods and the Usability of Interface Prototypes: The Complementarity of Laboratory Observation and Focus Groups (requires ACM account). ACM Ninth International Conference on Systems Documentation, p.106-112   (not in reader) 
  • Greenbaum, Thomas L. (1997) The Handbook For Focus Group Research (2nd ed.), Thousand Oaks, CA: Sage. 
10. March 20 & 22- Field observation & Ethnography

These readings overlap somewhat but each offers something that the others don't. 

  • Blomberg, Jeanette L. "Ethnography: aligning field studies of work and system design." in Monk, Andrew F., and G. Nigel Gilbert, eds., Perspectives on HCI: Diverse Approaches. ch 8, p. 173-197. Academic Press, 1995.  
  • Blomberg, Jeanette, Jean Giacomi, Andrea Mosher, Pat Swenton-Wall, "Ethnographic Field Methods and their Relation to Design." Douglas Schuler and Aki Namioka, eds.. Participatory Design: Principles and Practices. Hillsdale, NJ: Lawrence Erlbaum Associates, 1993; p. 123-155.  
  • Brun-Cottan, Francoise and Patricia Wall (1995). "Using Video to Re-Present the User." Communications of the ACM 38 (5): 61-71.  
  • Hughes, John, and others. "Ethnography in Interactive Systems Design." Interactions II.2 April 1995 p. 58-65.  
  • Jordan, Brigitte, "Ethnographic Workplace Studies and CSCW," Shapiro, Dan, and others, eds., The Design of Computer-Supported Cooperative Work and Groupware Systems, Elsevier, 1996, ch 3, p. 17-42.  
  • Recommended: not in reader: Paul Dourish and Graham Button. 1996. "Technomethodology: Paradoxes and Possibilities." In Proc. ACM Conference on Human Factors in Computing Systems CHI'96 (Vancouver, Canada). New York: ACM. http://www.acm.org/sigchi/chi96/proceedings/papers/Button/jpd_txt.htm
11 & 12. Apr 3, 5, 10, 12 - Contextual Inquiry & Design
  • H. Beyer and K. Holtzblatt. Contextual Design.

  • Interactions  vi.1 (Jan-Feb 1999) p. 32-42.
  • H. Beyer and K. Holtzblatt. 1998. Contextual Design: Defining Customer-Centered Systems. San Francisco, CA:Morgan Kaufmann Publishers, Inc.  

  • ch. 3, "Principles of Contextual Inquiry"
    ch. 6, "Work Models"

    ALSO READ (not in reader):

    ch. 8, "Consolidation"
    ch. 9, "Creating One View of the Customer"
    ch. 11-15, Contextual Design

For more info, see their website  http://www.incent.com

13a. Apr 17 - Logs and Monitoring

13b. Apr 19 - Ethics of Representation 
  • Suchman, Lucy. Making work visible. CACM 38:9 (Sept. 1995) p. 56-64.  
  • Muller, Michael J. Ethnocritical questions for working with translations, interpretation, and their stakeholders. CACM 38:9 (Sept. 1995) p. 64-65.  
  • Bannon, Liam J. The politics of design: representing work. CACM 38:9 (Sept. 1995) p. 66-68.  
- Collecting data in cyberspace
  • Thomas, Jim. Introduction: a debate about ethics of fair practices for collecting social science data in cyberspace. The Information Society 12:2 (1996) p. 107-118. 
  • King, Storm A. Researching internet communities: proposed ethical guidelines for the reporting of results. The Information Society 12:2 (1996) p. 119-128.  
  • Max, Gary T.  Ethics for the New Surveillance.  The Information Society 14: 171-185, 1998. 
14a. Apr 24 - Recap and summary of the course

14b. Apr 26 - NO CLASS

May 1st Deadline to turn in project drafts for review

5. May 1, 3- Final project presentations.

  • 15-20 min. presentation per project.
16. May 8- Last scheduled class but we will not meet.


 back home
send feedback ~ updated March-23-2000