go to UC Berkeley home pagego to SIMS home page

Needs Assessment and Evaluation of Information Systems
Mondays, 3:30-6, 202 South Hall
Admission: SIMS students, or permission of instructor

Prof. Nancy Van House
307A South Hall
Updated   5/11/99.


 Overview/wrap-up presentation  (currently not working)


BUY -- Not in campus bookstores

Jakob Nielsen, Usability Engineering, Academic Press, 1993.
Amazon.com: Ships in 2-3 days. Price: $25.60
Jakob Nielsen and Robert L. Mack, eds. Usability Inspection Methods, Wiley, 1994
amazon.com: Usually ships in 24 hours; $55.00
READER  will be available soon at Copy Central on Bancroft above Telegraph.

SCHEDULE  (subject to change)

Jan 25       Intro to the course

Feb 1        Needs and usability assessment
Feb 8        Usability testing     Barbara, Chaitee, Masako
Feb 15      President's day - NO CLASS
                    In-depth investigation web pages to be available for others.
Feb 22      Usability inspection, heuristic evaluation, cognitive walk-through    Wang, Kirch, Hernandez, Timms
Preliminary project description due.   Link
March 1    Surveys and interviews     Henke, Kada, Stetson
                    Assignment 1 due
March 8    Focus groups   Thomas,
             Assignment 2 due
March 15  Ethnography       Lussier, Mark
March 29  Task analysis; contextual inquiry  Wang, Carol, Jamie
April 5       Logs and monitoring; scenarios   Huney, Arti; Ana, Masako, Jennifer
April 12     Information needs.  Note change of topic and readings.
 April 19      Large-scale social impacts.Assignment 3 due   -- note change of date.
April 26       NO CLASS MEETING.
                    2-page synopsis of major issues in area of your in-depth investigation due  Tuesday April 27, 5 pm.
May 3       Ethics
May 10     Synthesis of course
May 12    FINAL DATE TO TURN IN FINAL PROJECTS.  Projects that are turned in earlier than this are much appreciated, and a happy instructor...

ASSIGNMENTS (subject to change)
Assignment 1   Assignment 2    Assignment 3   In-depth investigation   Final project

USEFUL WEB SITES  (we'll keep updating this list)

Usability Professionals' Association.  "A forum to promote usability concepts and techniques worldwide."

"Suggested Readings for Usability Testing"
I don't know who the author is, but this was recommended by a knowledgeable usability professional, and it references useful sources.

IBM on user-centered design.

Jakob Nielsen's Alertbox: http://www.useit.com/alertbox
One-stop shopping for Nielsen's view of the (web) world.

UTEST Home Page: http://tigger.clemson.edu/utest/index.html

 Society for Technical Communication Usability SIG, The Usability Bookshelf.  Lots of useful references.

The order of topics is structured to introduce early the methods that students are most likely to need to use on final projects. The result is not conceptually clean, but practical.

Please note that the order of the readings was changed after they were sent to Copy Central, so that the order of the reader does not match the order of the syllabus. To find readings more easily, check the reader page numbers, listed in square brackets after the articles and chapters below.

 1. January 25 - Introduction to the course

If you are not familiar with measurement, sampling, and/or social science survey research methods, read about them, early in the course, in any quantitative social science research methods text. We will put on informal reserve within South Hall the following:

Singleton, Straits, Straits, and McAllister. Approaches to Social Research. Oxford Univ Press, 1988.
Chs. 5, p. 110-end, on reliability and vailidity
ch 6, sampling
 2. February 1 - Needs assessment and usability assessment - overview

Buttenfield, B.  "Usability Evaluation of Digital Libraries."  Science and Technology Libraries (forthcoming).  Will be distributed in class.

Marchionini, Information Seeking in Electronic Environments, ch 3 [Reader pp. 5-38]

Nielsen, Usability Engineering
ch 1, Exec summary
ch 2, What is usability
ch 4, Usability engineering life cycle
also look at p. 224
Preece, Jenny, ed. Guide to Usability: Human Factors in Computing, Addison-Wesley, 1993; ch 6, "Evaluation." [Reader pp. 63-78]

Optional, but highly recommended -- in the reader:  Schamber, "Relevance and Information Behavior," Annual Review of Information Science and Technology 29 (1994) p. 3-48. [Reader pp. 39-62]

3. February 8 -  Usability Testing

Nielsen, Usability Engineering, ch 6, "Usability Testing"

From Behavior and Information Technology 13: 1-2, Jan-April, 1994, Special issue on usability labs: [Reader pp. 389-427]

    Blatt, Louis, and others, "Designing and equipping a usability laboratory."
    Faith, Janet L. and others, "A practical guide to using software usability labs."
    Bevan, Nigel, and Miles MacLeod, "Usability measurement in context."

Rubin, Handbook of Usability Testing, Table of Contents (a good summary of steps of usability testing); ch. 2, "Overview of Usability Testing." [Reader pp. 363-388] This is an excellent resource and well worth buying if you are serious about usability testing. $45 in paper.

Also recommended: Joseph S. Dumas, Janice C. Redish. A Practical Guide to Usability Testing, Ablex, 1993.

Rubin and Dumas & Redish are on reserve in Main Library.

4. no class February 15 - President's Day -- however, rather than lose a week's work due to one day holiday, we'll cover more ground for our next class

5. February 22 -- Usability Inspection; Heuristic Evaluation; Cognitive Walk-Throughs

Nielsen & Mack, chs 1, 3, 5, 6

Nielsen UE ch 5

Nielsen & Mack UIM ch 2

Instone, Keith, "Site Usability Evaluation," [Reader pp. 429-432] http://webreview.com/wr/pub/97/10/10/usability/index.html

Instone, Keith. "Usability Heuristics for the Web." [Reader pp. 433-437] http://webreview.com/wr/pub/97/10/10/usability/sidebar.html

Browse: Nielsen's Alertbox, http://www.useit.com/alertbox/

Browse: webreview.com on usability. http://webreview.com/wr/pub/Usability.

Assignment 1, due March 1.

6. March 1  - Surveys and Interviews 

Singleton, Straits, Straits, and McAllister. Approaches to Social Research. Oxford Univ Press, 1988. (on informal reserve within South Hall)    ch 9, survey research; ch 10, survey instrumentation.

Fuccella, Jeanette and Jack Pizzolato, Creating Web Site Designs Based on User Expectations and Feedback. ITG Newsletter 1.1 June 1998 [Reader pp. 93-102] http://www.sandia.gov/itg/newsletter/june98/web_design.html

Marchionini, G., Plaisant, C., Komlodi, A. (May 1996). User needs assessment for the Library of Congress National Digital Library.  CS-TR-3640, CAR-TR-829, CLIS-96-01. [Reader pp. 103-134] ftp://ftp.cs.umd.edu/pub/hcil/Reports-Abstracts-Bibliography/3640html/3640.html. Look also at the subject of their work: http://www.loc.gov/ammem/ - which also has a survey.

Nielsen on web surveys:  http://www.useit.com/alertbox/990110.html
Assignment 1 is due today.
Assignment 2 is about today's content.

7. March 8 -- Focus Groups

David L. Morgan, Focus Groups as Qualitative Research, Sage, 1988; p. 53-71.

Krueger, Richard A. Focus Groups, 2nd ed. Sage, 1994; "Asking questions," p. 53 -69. [Reader pp. 145-154]

Nielsen, Usability Engineering, p. 214-217.

Bishop, Ann Peterson. "Digital Libraries and Knowledge Disaggregation: The Use of Journal Article Components." DL98: Proceedings of the 3rd ACM International Conference on Digital Libraries. New York, NY: ACM, 1998. [Reader pp. 155-167] http://anshar.grainger.uiuc.edu/dlisoc/socsci_site/conf-dl98-ann-knowl-disag.html
Optional: in the reader: Hufnagel, Ellen M. and Christoper Conca. "User Response Data: the Potential for Errors and Biases." Information Systems Research 5:1 (March 1994) p. 48-73. [Reader pp. 79-91]

Assignment 2 is due today.

8. March 15 - Ethnography

These readings overlap somewhat but each offers something that the others don't.

Blomberg, Jeanette L. "Ethnography: aligning field studies of work and system design." in Monk, Andrew F., and G. Nigel Gilbert, eds., Perspectives on HCI: Diverse Approaches. ch 8, p. 173-197. Academic Press, 1995. [Reader pp. 207-218]

Blomberg, Jeanette, Jean Giacomi, Andrea Mosher, Pat Swenton-Wall, "Ethnographic Field Methods and their Relation to Design." Douglas Schuler and Aki Namioka, eds.. Participatory Design: Principles and Practices. Hillsdale, NJ: Lawrence Erlbaum Associates, 1993; p. 123-155. [Reader pp. 219-251]

Brun-Cottan, Francoise and Patricia Wall (1995). "Using Video to Re-Present the User." Communications of the ACM 38 (5): 61-71. [Reader pp. 263-273]

Paul Dourish and Graham Button. 1996. "Technomethodology: Paradoxes and Possibilities." In Proc. ACM Conference on Human Factors in Computing Systems CHI'96 (Vancouver, Canada). New York: ACM. [not in reader]  http://www.acm.org/sigchi/chi96/proceedings/papers/Button/jpd_txt.htm

Hughes, John, and others. "Ethnography in Interactive Systems Design." Interactions II.2 April 1995 p. 58-65. [Reader pp. 253-262]

Jordan, Brigitte, "Ethnographic Workplace Studies and CSCW," Shapiro, Dan, and others, eds., The Design of Computer-Supported Cooperative Work and Groupware Systems, Elsevier, 1996, ch 3, p. 17-42. [Reader pp. 275-300]
no class March 22 - Spring Break

9. March 29 -- Task Analysis: Contextual Inquiry

K. Holtzblatt and S. Jones, "Contextual Inquiry: A Participatory Technique for System Design," Participatory Design: Principles and Practice. Aki Namioka and Doug Schuler (Eds.), Hillsdale, NJ: Lawrence Earlbaum Pub. 1993. [Reader pp. 169-186]

K. Holtzblatt and H. Beyer, "Representing Work for the Purpose of Design," Representations of Work, HICSS Monograph (Hawaii International Conference on System Sciences), January 1994. Lucy Suchman, Editor. [Reader pp. 187-205] http://www.incent.com/papers.indx/WorkModeling.html

For list of papers by these authors on Contextual Inquiry, see http://www.incent.com/Papers.html. [Note that Holtzblatt et al.'s web papers are bad examples of web documents! The paper itself doesn't include authors and source.] Their book: H. Beyer and K. Holtzblatt. 1998. Contextual Design: Defining Customer-Centered Systems. San Francisco, CA:Morgan Kaufmann Publishers, Inc.

10. April 5 - Logs and Monitoring; Scenarios

Cooper, Michael D. "Design considerations in instrumenting and monitoring web-based information retrieval systems." JASIS 49 (10): 903-919, 1998. [Reader pp. 301-317] Can be accessed via http://www2.interscience.wiley.com/zcgi/zti

Buttenfield, Barbara, article in IJHCS, forthcoming. Will be distributed in class.

James E. Gaskin, User Monitoring Drives Site Designs, Inter@ctive Week, December 14, 1998

Nielsen, Usability Engineering, p. 217-221.

Carroll, John, ed. Scenario-Based Design 1995 John Wiley & Sons [Reader pp. 319-350]
    Kuutti, ch 1
    Kyng l, ch 4
    Carey and Rusli, ch 7

Constantine, Larry L. "Essential Modelling: Use Cases for User Interfaces." Interactions II.2, April, 1995; p. 35-46. [Reader pp. 351-362]


Assignment 3.


Marchionini, Information Seeking in Electronic Environments, ch 3 [Reader pp. 5-38]

Schamber, "Relevance and Information Behavior," Annual Review of Information Science and Technology 29 (1994) p. 3-48. [Reader pp. 39-62]  Read her pp. 23-36, reader pp. 50-56.

Dervin, Brenda.  "An Overview of Sense-making Research: Concepts, Methods, and Results To Date."    Presented at:   International Communication Association Annual Meeting, Dallas, Texas, USA, May l983.  Author's note: "This 1983 presentation of the Sense-Making approach is now out of date but still provides a foundation for interested readers."  72 pages but much of that is appendices.


Nancy A. Van House, Mark H. Butler, Lisa R. Schiff.  Cooperative Knowledge Work and Practices of Trust: Sharing Environmental Planning Data Sets.  CSCW 98: The ACM Conference On Computer Supported Cooperative Work.  Seattle, WA,  November 14-18, 1998, Proceedings; ACM, 1998,  pp. 335-343.

Articles about why trust and credibility are problems on the Web:

NY Times 3/4/99
NY Times 2/26/99

About how to evaluate web resources

Information quality on the web
Harvard's Widener Library on evaluating web resources
Practical guide to evaluating info quality, from Johns Hopkins' Eisenhower Library
Nielsen's AlertBox on  trust and web design
April 19 - Assessing larger social impacts
 trust, cont.
 Due April 19: assignment 3  -- Will be discussed in class.

Eugene Rochlin: Trapped in the Net. Princeton  Univ Press, 1997: chapters 1; ch 2 p. 46 (THe Interconnected Office to
end of chapter; 4; 7; and 12. http://pup.princeton.edu/books/
The rest of the book is recommended; some of you may find the chapters on
financial trading interesting, if you're interested in e-commerce.
(The chapters on the military may be of particular interest just now when
you consider what NATO is trying to do with international forces.)

April 26 NO CLASS.

May 3  Ethics

Representing work:

Suchman, Lucy. Making work visible. CACM 38:9 (Sept. 1995) p. 56-64. [RPN 495 - 502]

Muller, Michael J. Ethnocritical questions for working with translations, interpretation, and their stakeholders. CACM 38:9 (Sept.
1995) p. 64-65. [Reader pp. 503-504]

Bannon, Liam J. "The politics of design: representing work." CACM 38:9 (Sept. 1995) p. 66-68. [Reader pp. 505-507]

Collecting data in cyberspace:

Thomas, Jim. "Introduction: a debate about ethics of fair practices for collecting social science data in cyberspace." The Information
Society 12:2 (1996) p. 107-118. [Reader pp. 523-533]

King, Storm A. "Researching internet communities: proposed ethical guidelines for the reporting of results." The Information Society
12:2 (1996) p. 119-128. [Reader pp. 535-543]

May 10 -- Recap and synthesis.

Review of methods,  timeline of approach,  who does usability; discuss course in general and  what we learned individually..
May 12 Final deadline to turn in projects