Lo-fi prototype/testing

PILOT Usability Study

 

Introduction
Method
Test Measures
Results
Discussion
Formal Experiment Description
Appendix A: The script
Appendix B: Task List for the Monitor
Appendix C: Consent Form
Appendix D: Background Questionaire
Appendix E: CMS Information and Task Instructions for the Users
Appendix F: Post Test Questionaire
Appendix G: Coding Scheme for Raw Data
Appendix H: Raw Data for Usability tests
Appendix I: Results of Post-Test Questionaire and Background Questionare

 

 

Introduction

      We conducted an assessment usability test of our second interactive prototype for our Content Management System (CMS). The purpose of this study was to identify usability problems with CMS so that we can rectify the problems for the next interactive prototype.   Our goal is to create a Content Management System that is easy to use, satisfying to use, and provides functionality that is highly valued by the target population.  In the following sections we discuss the participants, methodology, and results.

       Method

Participants

     The ideal way to test our system is to install the system in an environment where a group of people are working on web development.  The group would consist of a webmaster, a less technical person who oversees the web site, and a person whose job involves adding text to the company web site.  Unfortunately, we do not have access to such a situation in order to test our system.

     The first participant was Steve Kafka.  We selected Steve because he is involved in web development at SIMS and he has some experience with usability.  Thus we felt he could provide an expert criticism for our system.

     The second participant was Margo Dunlap.  We selected Margo because she recognized problems with our system in the heuristic evaluation and we felt she would be able to give us feedback on how the new system works.

     The third participant was Jonathon Henke.  We selected Jonathon because he has experience with web development as well as usability issues.  We felt he could provide valuable criticism about using our system.

Apparatus

     The usability test was conducted at South Hall in the downstairs lab on the computers.

Tasks

Scenario 1: You are working for Lobotech and need to add a company announcement for the annual picnic to the company web site. You are going to use the Content Management System (CMS) to complete this task. As part of your job you know that the content for company announcements is stored under the category AboutUS in the CMS.  You also know you want to let the default style sheet and graphic apply for all three sites.

Scenario evaluates test measures 1, 2, 3, and 4

Scenario 2: One of your hobbies is drawing. You created a cool graphic of one of your drawings in your spare time. You asked your boss at Lobotech if he wanted to use it as the company logo. You boss thought it was great and asked you to add the graphic to CMS for the HTML site so it will be available when the webmaster decides to use it.  The graphic is currently on the desktop of your computer under the name logo.gif

Scenario evaluates test measures 1, 2, 3, and 4

Scenario 3: You are the webmaster for Lobotech. You decided you want to change the color for active links to white (#ffffff) in the default style sheet for the HTML site.  You know the name of the style sheet file is newwap.

Scenario evaluates test measures 1, 2, 3, and 4

Scenario 4: You are the head of Human Resources for Lobotech. You asked an underling to put a company announcement for the annual picnic on the company web site. You now want to check that addition to the company web site before it goes live. 

Scenario evaluates test measures 1, 2, 3, and 4

Procedure

     In order to conduct an assessment usability test of CMS we recruited three users. Detailed information about our users is provided in the participants section of this paper. After greeting each user, we read a script to introduce the user to the study (see part 1 of Appendix A). Then each user was asked to sign a consent form and fill out a background questionaire (see Appendix C and Appendix D). Next, the monitor read a script to introduce the user to the system (see part 2 of Appendix A).  Then the monitor provided the user with a brief demonstration of how to use the system to modify a document. Each user was then provided with a list of four tasks that included the information they needed to complete the task.  In addition, this piece of paper also included the introductory information provided about CMS so the user could refresh their memory. (see Appendix E).

     The monitor then opened up a window on the computer with the CMS log in page.  Then the monitor prompted the user to enter the log in and password they had been given. Next the monitor read task scenario 1 and prompted the user to begin the task.  As the user completed each task the monitor read the next task and prompted the user to proceed.   During this time the monitor encourage the user to speak out loud and the observers noted the users comments and interactions with the system.  After completing the tasks, the user was presented with a post-test questionaire (see Appendix F).  Then the user was asked to provide any additional feedback or criticism.  Finally, the user was thanked for participating.

      Test Measures

The following is a list of what we measured and why in our test:

1. We want to determine if the navigation in the system is adequate.  Users need to transition from one area of the interface to another (e.g. from modifying a document to changing a style sheet). Therefore, we want to know if they are able to smoothly and easily navigate through the system. We recently added a 'go back' button and a 'cancel' button.  We want to make sure we are not missing any other navigation buttons.

2. We want to determine if the terminology is clear.  We want to make sure the terms we have chosen match the user's conceptual model.  If the terminology is unclear users will have difficulty determining how to complete their tasks with the system.

3.  We want to determine if the instructions provided on each page are clear.  The instructions are designed to assist users in completing their tasks if they forget how to do something in the system or if they are using the system for the first time. We need to make sure the instructions are clear and easy to follow so that users can complete their tasks.

4. We want to determine if the flow of action on the screen follows the users conceptual model and work flow. If the flow through the interface is natural then users will be able to seamlessly complete their tasks.

5. We want to determine if the feedback in the system meets the users needs.  We added a preview button and we want to see if it provides the user with adequate feedback about the changes that have been made.

      Results

     Below are the problems we found in our interface as a result of the usability study. The raw data for each of the users is in Appendix H.  Overall, we found four major categories of problems.  First, we found several bugs in our system.  Second, we found a variety of small changes and additions that we need to make in the system. Many of these changes have to do with the instructions and navigation of the system.  Third, we found some issues with the feedback in the system.  Finally, we found a list of bigger issues.  Many of these deal with the users work flow in the system and need to be discussed in order to determine the best solution. Next to each finding is a number in parentheses that denotes how many users had this problem.  The reference number next to each finding ties the finding to the raw data in Appendix H.   In addition, for demographic information about the user and the raw data from the post-test questionnaire please see Appendix I.

Category 1: Bugs in the system

Category 2: Problems with instructions/terminology

 Category 3: Navigation problems

Category 4: Inadequate Feedback within the system

Category 5: Problems with the work flow and other issues that need to be discussed

Results of the Post-Test Questionnaire

      Discussion

Navigation

     Overall, users were able to navigate quite well within the system.  The 'go back' and 'cancel' buttons were useful.  At the same time, we learned about several different pieces of navigation that are missing in our system. Most of these are missing options are easy to add. We plan to fix as many of these as we can so that the system better meets the user's needs.  First, we didn't provide the user with the option to make changes after they preview a document.  Second, we learned that we need to replace many of our button titles with descriptions that more clearly express what the user is trying to do.  For example, instead of having a submit button for category selection we plan to have a button that says select category. Third, we learned that it is useful to the user to have titles on the top left of each page. Finally, a variety of other smaller navigation issues were brought to light by the study and our listed in the Results section. 

Terminology

     We learned that there are several instances where the terms we use in the instructions are unclear.  For example, using the word changes when talking about adding a new document is confusing to the user. We plan to revise the instructions and attempt to make them more clear. At the same time, the terminology used to refer to the main areas within the interface (style sheets, graphics, and verification console) seem to be clear.

Instructions

     As mentioned above there are several places where are instructions are unclear. We plan to revise the instructions.  In addition, we may add additional instructions about issues raised by the usability test.  For example, we may add instructions about the file format that needs to be used.  However, it is possible that the file format is something that would be specified by the business that chooses to implement the system.  Thus, we would not want to constrain the business with a specified file format.  However, we consider adding file format instructions for demonstration purposes.

Feedback

     We learned that we were missing a preview button on the style sheet page.  Users really want to view the changes they make to style sheets.  We plan to put a preview button in the style sheet section where the user could look at modified style sheet applied to each page to which the style sheet is linked.  In addition, we learned that users would like to know that the editor has been notified that something is waiting for them in the verification console.  We plan to let the user know that an e-mail has been sent to the verifier.  One thing we may have to consider is allowing the user who to have the e-mail sent to if there are multiple verifiers.

Work Flow

     We found the most sever usability issues in this area. First, we learned that having the selection of the style sheets and graphics separate from the DEM in both working with a document and in the verification console prevents a break in flow.  The user is unsure what they are suppose to do in the style sheet and graphic selection because they feel they have already submitted the document to the system.  We would like to have the style sheet and graphic selection on the same page as the DEM.  We knew that the set up of this part of our interface was not ideal in our second prototype.  One of the difficulties in combining these options on one page involves Ezedit.  The Ezedit application has a save button and we are having difficulty programming around the save option.   We need to program around the save button so that all the options can be combined on one page and the button can press one button to submit the selections to the system.

     Second, we learned that the editor needs more information about the documents waiting in the verification console.  Firstly, the editor needs to be notified perhaps via e-mail that something is waiting for them in the verification console. One issue we will have to consider with this e-mail is that the user would need to have the option to choose which person within the system the e-mail should be sent to.  Presumably, the system could be used in a situation where there are multiple editors for different people working within the system.  Second, when the editor comes to the console, they need a way to figure out which documents they should look at.  Providing an e-mail with the name of the document and the author provides the editor with this information.  In addition, having the author's name next to each waiting document would help the editor determine which documents they need to verify.  Finally, having a file format for naming documents would assist in providing the editor with information about the document without actually looking at the document.

Conclusion

     We realize that our system was not tested in the ideal setting.  More specifically, our system should be tested in a group of people working using the actual files that the people work with.  However, this was not feasible.  At the same time, we feel we gained insights from our study about components of our system that need revision.  As mentioned above, there are several places that need additional navigation as well as more informative instructions and feedback.  In addition, we learned about many larger problems with the verification console.  We plan to address as many of the problems as we can.

      Formal Experiment Description

CMS has a large quantity of documents in the database. We want to provide users with a fast and easy way to find a document. This experiment is designed to determine the easiest and fastest way to find a certain document in mCMS among three alternatives. We designed this formal experiment to investigate amoung these three design options because we were unable to test this for these options in our pilot usability test.

Hypotheses

1. Showing list of documents of a chosen category in one page (getcategory.cfm) will reduce number of click and search time.
2. Showing list of documents that were recently modified or added in one page will reduce number of click and search time.

Dependent Variables

1. Number of click: the number of a user’s click to open a certain document from the log on page to the document page (edit.cfm).
2. Search time: time a user spends to open a certain document from the log on page to the document page (edit.cfm).

Independent Variables

    #One between subjects factor

1. Ways of displaying document list

1. Separate: Display the list of document of the selected category in the next page of category selection page, when a user select a category and submit (current way). 
2. Together: Display the list of document of the selected category in the same page next to the category list, when a user select a category and submit.
3. Recent list: Display 10 documents modified or added most recently the same page with category selection box.

    # Two within-subjects factor

1. The author of the document the user searches
1. Himself/Herself
2. Other person
2. The last day a document is modified or added by the same user
1. A document that was modified 1 week ago.
2. A document that was modified 2 weeks ago.
3. A document that was modified 3 weeks ago.

Blocking and repetition

In each 12 block, the order of the document’s author (Himself vs. Other) will differ for each participant. 

      Appendix A: The Script

Part 1

     Hello, my name is______. Thank you for participating in our study.  We are here to test how easy it is to use a web based system for managing the content of website for a company called Lobotech.  We will ask you to perform some typical tasks to see how well the website helps you complete your work. Do your best, but you may run into problems that make it hard to complete the tasks.  This is actually what we are looking for.  We want to identify the places where we can improve the design of the content management system.  Feel free to ask questions, but we may not answer them during the test because we want to see how the system performs when the user does not have the help of an experienced user. We would also like to encourage you to think out loud as you work on each task.  If you start to feel uncomfortable and want to quit at any time that is okay. 

Part 2

     For the purpose of this study suppose that you are part of the web development team in an organization called Lobotech.  This system allows you to manage your style sheet files, graphic files, and text files, which we refer to as documents.  You are able to combine these files and then templates are used to create web pages from these files.

     In addition, Lobotech has three different sites within the company.  One site is for wireless devices and it is called the WAP site.  The second site is for users who have Flash enabled software and it is called the Flash site.  The third site is the basic HTML site and is called the HTML site.

     Finally, you need to know that your username is cmarin and your password is test 44.

     Now, we are going to give you a small demo of the system.  We are going to show you how to modify an existing document so that you get a feel for a task flow in the system.

      Appendix B: Task List for the Monitor

Scenario 1: You are working for Lobotech and need to add a company announcement for the annual picnic to the company web site. You are going to use the Content Management System (CMS) to complete this task. As part of your job you know that the content for company announcements is stored under the category AboutUS in the CMS.  You also know you want to let the default style sheet and graphic apply for all three sites.

-remind user to think out loud

Scenario 2: One of your hobbies is drawing. You created a cool graphic of one of your drawings in your spare time. You asked your boss at Lobotech if he wanted to use it as the company logo. You boss thought it was great and asked you to add the graphic to CMS for the HTML site so it will be available when the webmaster decides to use it.  The graphic is currently on the desktop of your computer under the name logo.gif

-remind user to think out loud

Scenario 3: You are the webmaster for Lobotech. You decided you want to change the color for active links to white (#ffffff) in the default style sheet for the HTML site.  You know the name of the style sheet file is newwap.

-remind user to think out loud

Scenario 4: You are the head of Human Resources for Lobotech. You asked an underling to put a company announcement for the annual picnic on the company web site. You now want to check that addition to the company web site before it goes live.

-remind user to think out

      Appendix C: Consent Form

     Our Masters group in the School of Information Management and Systems at UC Berkeley is conducting a study to assess the viability of our User Interface for a Web content management system.

      If you volunteer to participate in this study, you will be asked to follow through a prototype, give comments about the interface in the context of pre-defined tasks, and answer some questions and make any relevant comments associated with your experience. Your interactions with the computer and comments will be recorded by an observer in the group.

     There are no direct benefits to you for participating other than what may be an educational experience in viewing interesting ways we have conceived to manage Web content. There are also probably no risks to you; you will simply be using a mock computer, following through the interface, and commenting on your experience with it. We will not name you when we discuss our observations of your interaction or your comments.

     If you accept these terms, please write your initials and the date here:             

      Appendix D: Background Questionaire

Please answer the questions below in order to help us understand your background and experience

Name:_______________________

Age:______              Gender:_____

Highest Education Level Achieved:________________

Major you are working on or Current Employment:______________________________

Computer experience:

1. What is the total length of time you have been using computers?                                          

2.  Have you ever worked in web development?                                       Yes      No

3. If yes for #2, How long have you been involved in web development?______                     

4. Below is a list of components that can be used in web development.  Please circle the components that you are familiar with and then indicate how much experience you have had using this component with respect to web development.

            Component                                      Months of Experience

            Style sheets                                                                          

            Graphic files                                                                         

            Putting text on web pages                                                   

      Appendix E: CMS Information and Task Instructions for the Users

For the purpose of this study suppose that you are part of the web development team in an organization called Lobotech.  This system allows you to manage your style sheet files, graphic files, and text files, which we refer to as documents.  You are able to combine these files and then templates are used to create web pages from these files.

In addition, Lobotech has three different sites within the company.  One site is for wireless devices and it is called the WAP site.  The second site is for users who have Flash enabled software and it is called the Flash site.  The third site is the basic HTML site and is called the HTML site.

Finally, you need to know that your username is cmarin and your password is test 44.

Tasks

Scenario 1: You are working for Lobotech and need to add a company announcement for the annual picnic to the company web site. You are going to use the Content Management System (CMS) to complete this task. As part of your job you know that the content for company announcements is stored under the category AboutUS in the CMS.  You also know you want to let the default style sheet and graphic apply for all three sites.

Scenario 2: One of your hobbies is drawing. You created a cool graphic of one of your drawings in your spare time. You asked your boss at Lobotech if he wanted to use it as the company logo. You boss thought it was great and asked you to add the graphic to CMS for the HTML site so it will be available when the webmaster decides to use it.  The graphic is currently on the desktop of your computer under the name logo.gif

Scenario 3: You are the webmaster for Lobotech. You decided you want to change the color for active links to white (#ffffff) in the default style sheet for the HTML site.  You know the name of the style sheet file is newwap.

Scenario 4: You are the head of Human Resources for Lobotech. You asked an underling to put a company announcement for the annual picnic on the company web site. You now want to check that addition to the company web site before it goes live

     Appendix F: Post Test Questionaire

1. Overall, I found the navigation easy to use (circle one).

strongly agree     agree     neutral     disagree     strongly disagree

2. Overall, I found the instructions clear (circle one)

strongly agree     agree     neutral     disagree     strongly disagree

3. Overall, I found the terminology clear (circle one)

strongly agree     agree     neutral     disagree     strongly disagree

4. Overall, I felt the flow through the interface was easy to follow (circle one)

strongly agree     agree     neutral     disagree     strongly disagree

5. Any additonal comments or suggestions with regard to CMS.

      Appendix G: Coding Scheme for Raw Data

Codes for Recording Results of Usability Test

S = successfully completed the task

P = Prompt from the test monitor

X = incorrect action

"" = verbatim comment from the user

R - answer user gives when probed by the test monitor

*The scenarios were referred to by their respective numbers (1-4) in recording the data for each scenario.

      Appendix H: Raw Data for Usability tests

User A

User B

User C

      Appendix I: Results of Post-Test Questionaire and Background Questionare

Post-Test questionaire

Questions

1. Navigation easy to use ?
2. Instructions clear ?
3. Terminology clear ?
4. Interface easy to follow ?