Assignment #8: Pilot Usability Study
April 29, 2004

Introduction
Method
Test Measures
Results
Discussion
Formal Experiment Design
Appendices
Presentation Slides
Work Distribution Table

Introduction

The Healthy Communities Network System is an information network system that tracks community health issues, promotes sharing of community health information and best practices, and acts as a networking tool to help people get involved in making community decisions. This system will serve as a set of templates that can be built on and easily customized by local communities using the administrative page, so that each interested community can have its own version of the system while the underlying architecture remains centralized.

One of the first communities that have expressed an interest is Marin County and therefore we will build the first version of the system with customizations made for Marin County. There are many modules we intend to build for the system in order to reflect all the functionalities listed above. Since it is impossible to test out all the modules, we decided to focus our usability evaluation on the �indicator� module for the purpose of this class.

The purpose of this pilot usability study is to improve the current design of our second interactive prototype. We hope that this structured usability study would allow us to gather more concrete inputs and feedbacks for us to improve the next prototype as we are approaching to the final stage of the system development.

Method

Procedure

We selected the following participants to represent potential users from different age groups, genders, and education/occupation backgrounds. We first presented the evaluators with our system homepage. Then we briefly introduced the system by explaining to them that it was a workable web application, which they could navigate through the system like any other ordinary web systems. After a few minutes of experimenting with the system, evaluators were asked to perform three major tasks that were written on papers. Upon finishing with all the tasks, they were asked to complete a ten-questions questionnaire followed by a short five-questions interview (see appendix for complete sets of questionnaire and interview questions).

Participants

Participant 1
� Public Health professional, specialized in infectious diseases
� Female between 26-35 years old
� Medium level of computer literacy
� Minimum familiarity with the Healthy Community Network system

Participant 2
� Professional working on a public health project, with an undergraduate degree in Cognitive Sciences
� Female between 18-25 years old
� High level of computer literacy
� Some familiarity with the Healthy Community Network system

Participant 3
� Psychology undergraduate student
� Male between 18-25 years old
� Medium level of computer literacy
� No familiarity with the Healthy Community Network system

Apparatus

The usability test was conducted in Room 205 at South Hall. A 12-inch monitor laptop computer was used to perform the test. The laptop computer was connected to the internet through the SIMS ethernet network using a physical cable line.

Tasks

We have modified our task scenarios slightly in response to the feedbacks we received from the previous heuristics evaluation, for which we were able to pinpoint out what specific parts of our system render further evaluation in this usability study. Since the low-fi prototype, we have been making gradual progress to improve our system design. Now as our prototype design is becoming more and more refined, we felt that we would benefit most from narrowing the tasks to the point that would allow us to get specific and concise feedback on specific functionalities come along with the indicator module. The major goals remain unchanged as before, which are �get indicator information�, �share indicator information�, and �search indicator information�.

Goal 1: Get Information on Air Quality
You are a public health official working for Marin County. You current project is to do some investigations on current air quality condition in Marin County. You want to use the Healthy Communities system to find out how Marin County is doing on air quality.
1. What is the rating? Is it good, fair, or poor?
2. Is the issue getting worse in past couple years?
3. Is Marin better or worse than most other communities?

What we looked for?
We tried to test if users understood the symbolic meaning of the color on the dials. We also wanted to test whether they know about the meaning of the �view past indicator status� and �compare status across communities� links, and whether the graphs on those pages made sense to them.

Goal 2: Share Information on Air Quality with others
Now you have found out how Marin is really doing on air quality. You want to discuss about the current situation with other people/professionals who are interested in air quality.
1. You want to email them the indicator information you have just found.
2. You also want to view others� opinions on the issue.
3. After viewing some of the comments, you want to respond to at least one of them with your own comments.

What we looked for?
We wanted to observe if users were aware of the existence of the email and post comment functions, and whether they understood the process of actually carrying out the tasks. This required them to go through the registration and login procedures, and the procedures of sending emails and posting comments.

Goal 3: Find out how Marin is doing on other community health related topics
Your project for air quality has been completed. Now you need ideas on another Marin County public health projects. Some possible areas of interest include infant mortality, childhood obesity, and employment rate. How would you find information on those three topics using the Healthy Community System? Which one(s) have a bad status?

What we looked for?
We wanted to observe if users knew how to search for indicator information by using one of the following methods: the link of �view all indicators� on top of the dashboard, the �community indicators� link on the left navigation bar, or the search box throughout the system.

[Top]

Test Measures

This usability pilot study was designed to test the interaction flow of our major tasks. We are looking for if the second interactive prototype of the system is easy for users to perform the 3 task scenarios. Observation, post-testing questionnaire, and interview questions were used in our test to collect quantitative and qualitative results:

 Quantitative measures from our observation :
   • Time taken to complete a task
   • Number of navigational errors made per task
 Qualitative measures from our post-testing questionnaire and interview questions:
   • Ease of use
   • User preferences
   • Level of satisfaction on a various aspects of the application

[Top]

Results

Users took about from 3 to 5 minutes to finish goal 1 of finding the air quality indicator information. All users made 2-3 mistakes as they were not initially familiar with the system and were clicking on different links before getting to the main indicator page for air quality, but at the end they were all able to finish the goal.

User 1 does not have a strong computer background, and therefore made the most errors and took the longest time to complete the goal. She spent a long time clicking on different left and right navigational links without finding where the functions are located. These functions were located on the bottom part of the main indicator page. They were not easily visible to some of the users because the laptop screen we used was only large enough to show the top portion of the main indicator page, and one would have to scroll down before seeing. Verbal assistance was needed for her to scroll down the main indicator page to locate the �Email to Others� and �Post Comments� functions. She would always use the �back� button to find the appropriate pages that she needed. She also spent a long time to find out how to register and login. She almost spent 10 minutes totally to finish the entire goal and she made 16 wrong attempts. Users 2 and 3 were successful in locating and using the functions relatively faster with fewer errors.

Task 3 – Find Out Other Community Health Related Topics in Marin County
   • Infant mortality
   • Childhood obesity 
   • Employment rate

After looking at air quality indicator information extensively, it became relatively easy for users to find information on other indicator topics. All users were able to compete goal 3 in a reasonably short period of time with minimal number of errors or wrong attempts.

Post-Test Questionnaire Evaluation

Ratings (1-5) are based on the ease of use and satisfaction

All users thought that the system and the information presented would be very important and helpful to improve community health. They thought that it was easy to use and navigate after getting familiar with the basic information layout. They were satisfied with their overall experience and only commented minor cosmetic and data flow changes that would make the system even better.

[Top]

Discussion

Most Important Lessons Learned

1. Diverse Computer Skill Levels

Our targeted users are people interested in public health issues, and these people can be public health professionals or just lay people who are interested in improving community conditions. Even within public health professionals, there are a variety levels of computer skills, some health experts are very computer savvy, while others might only be familiar with basic computer functions, and always need assistance or guidelines to navigate a computer system successfully. It is important to balance the two groups. More explanations or guidelines should often be available to users with limited computer skills when they need the help, but these help features should be optional so that expert users do not have to spend time going through them.

2. User Friendly Interface

3. Prevent Errors

Furthermore, we should also prevent errors at the first place. To login to our system, we require a password that is at least 5 characters long, and we should make this requirement explicit to prevent user errors. Otherwise, user might have to try several attempts before providing an acceptable password. Another example is to bring users back to where they were after logging in to the system.

4. Help Should Be Available

Help should be available at places where users might not get confused and do not know where the functions are and which choices to make. There were times when our participants got stuck and did not know where to go and where to seek help, and we had to give verbal assistance at the end. It would have been a great resource for there if help had been available.

What We Might Change For the Interface

1. Home Page

Users can now easily interpret the red/yellow/green as showing the indicator status poor/fair/good, but the average dials are still misleading. We will probably take out the word (avg) in the middle of the dial, and having new ones with needles pointing to the right side or to the left side as indicating above average or below average, so that the graphic representation is consistent with the other one.

2. Indicator Master List Page

The amount of information on the indicator master list�s page overwhelms some users. Only Marin county level indicators would show up initially on the master list, so that users will feel confused seeing indicators at different rationality levels. There will be additional capabilities to look at indicators at other levels (i.e. a drop down box to select to see only national or city or zip code level indicators.).

Another suggestion is to group all subjective indicators with red/yellow/green dials together under one category, and to group all objective indicators with above average/below average dials together as a different group under the same category.

3. Indicator Main Page

The layout of the main indicator page will be better aligned and explained with sections labeled description, explanation, source, source URL, other notes.

Add smaller links of �Email to Others� and �Post Comments� on the top right corner of the middle well, so that users can see these links right away when first get to the main indicator page, and there is no need to scroll down or looking at different places to find these functions. These links will be anchors to go the bottom part of the main indicator page, where these functions are actually located. The font of these functions will also be made larger to catch attention.

The login function is currently used for registered users to post comments. However, after a users log in or register, they are led to a page where they can configure their own system preferences and enter information for their profiles. Often users do not know how to get back to the main indicator page, so a nice capability will be the ability to go back to the original page after a user logs in or registers, but this feature will require extensive reprogramming and therefore might not be implemented for our third prototype.

In addition, a dial explanation link might be added for users to see a pop up screen with explanation on how to interpret the indicator status using the dials.

4. Other Indicator Related Pages

The font of the indicator name of these pages will be made larger so that users know they are still reading related information on that particular indicator.

Appropriate scales and legends might be added to the graphs for users to better understand them.

[Top]

Formal Experiment Design

Hypothesis
Our hypothesis is that to showing the indicators in the region of Marin County only as a default and giving users the ability to customize the region filter will improve the speed and ease of use of the indicator master list page.

Factors and Levels
Factors are the possible designs to show the indicators on the indicator master list page. The three levels are:
   • O – Keeping original design to show all the indicators retrieved from the database with all levels of regionality, including indicators at national, state, county, city, and zip code levels.
   • C - Include a set of radio buttons to select the region level and set the default checked radio button as Marin County.
   • M - Include a two-level dynamic dropdown menu (region level, and region name respectively) and a refresh button to filter the indicators. Set the default filter as Marin County.

Our response variables would be very similar to those in our pilot usability study:
   • Amount of time needed to complete each task (in minutes)
   • Number of navigation errors made while completing each task

Blocking
The experiment will be based on a "between group" concept since we do not need to take into account the learning effect. The experiment will be conducted on a group of twenty-four people. The users will be split into three groups of eight each randomly. For each block, users will be given the same tasks of looking for information on five indicators in Marin County. Each group will have to complete the tasks on factor level of O, C or M respectively. The relatively large number of participants is helpful to get valid results to compare the three designs.

[Top]

Appendices