A pilot usability study was conducted on the FAST interactive prototype. The FAST prototype allows professors at UC Berkeley to complete their Annual Supplement to the Bio-Bibliography (Bio-Bib)--typically a paper-based form--in a convenient, online format. This system allows professors to enter information about their teaching, research, and professional accomplishments, save and preview their changes, and submit their form online. Although the interface contains headings and place markers for the comprehensive Bio-Bib, the focus of our prototype development and testing is on the published works sub-section.

The main focus of the usability study was to assess whether our design reflected an intuitive decision/process flow for the end user and to detect design flaws. We made a major design change in the overall system infrastructure by changing the tabs at the top of the page to provide system-wide navigation instead of indicating sub-sections of the Bio-Bib. For example, in the latest prototype, we have only three tabs at the top of the page that are quasi-modal: home (a place to access all current Bio-Bib drafts or create a new draft), general instructions (general instructions on how to complete the entire Bio-Bib online), and preview and submit (this is a final step to submitting the form). To complete subsections of the Bio-Bib, users can utilize the left navigation bar, which provides hyperlinks to each subsection (e.g. teaching record, published works, creative works) along with a modification date and time.

Another important goal of the usability study was to assess whether the subsections of the Bio-Bib were easy to find on the navigation bar, easy to use, and reflected the diverse set of publication types that faculty produce. We wanted a more inclusive range of publication types to remove ambiguity for professors who have non-traditional research publications.

• male
• full professor, department chair
• College of Letters & Science
• 50-64 years of age

• male
• assistant professor
• professional school
• 24-35 years of age

• male
• full professor
• college of Letters & Science
• 50-64 years of age

Apparatus

Participant 1

• Macintosh OS
• Mozilla Firefox web browser

Participant 2

• WindowsXP OS
• Internet Explorer web browser

Participant 3

• Macintosh OS
• Safari web browser

Tasks

One major task was removed and one was added. The participants were not requested to edit a publication because the prototype coding was not complete. Future tests will allow for this.

Participants were asked to input their teaching record. Although technically outside the scope of this interface test, it enabled us to determine consistency of the layout between sections, and functionality of the left navigation bar as the main location for editing a Bio-Bib. It also allowed feedback for a parallel test where layout of the teaching record was being assessed.

Task #1: Starting a New Bio-Bib

Imagine that you have just been asked by the chair of your department to complete your Bio-Bib for the 2005-2006 reporting period. You decide to complete the form online this year, using the new online Bio-Bib system you have heard about. You open the email from the department secretary that contains website address to the online system, and click on the link. This is the page that you first see. Please complete the task of starting a new Bio-Bib. What is their reaction when first seeing the online Bio-Bib system?

What we assessed

• Did they make any errors in creating a new Bio-Bib?
• Did they seem confused by the structure of the system?

Task #2: Entering a new publication

Please add one of your own publications into this Bio-Bib, and ensure that it is accurately recorded.

What we assessed

• What do they use to store their bibliographic data (e.g. electronic CV, EndNote, Bib-Tex)?
• Did they cut and paste from their bibliographic data file, or do they re-type the information?
• How long it take to enter the publication into the required format, and save?
• Do they preview/not preview before submitting changes?
• Are any of the entry fields confusing to them?
• Do they make any errors?

Task #3: Entering a 2nd publication

Please add a second publication into this Bio-Bib, and ensure that it is accurately recorded.

What we assessed

• Is the system easy to learn?
• How long did takes to enter second publication (was it faster than the first time)?

Task #4: Deleting a previously saved publication

After entering two publications, you realize that you have made a mistake. You remember that the first publication you entered was already entered in the last Bio-Bib that you submitted. Please delete the first publication that you entered.

What we assessed

• Are they able to see the "delete" link? How long does this take?
• Is the confirmation page intuitive?

Task #5: Saving and editing a draft

Please logout of the system, and log back into the system to continue working on the Bio-Bib for 2005-2006.

What we assessed

• Were they able to find the logout button? How long did it take?
• Did they express concern about saving changes before logging out?
• When they log back into the system, were they able to locate their current draft? How long did it take?

Task #6: Entering teaching record

Please enter your teaching record for 2005-2006.

What we assessed

• Were they easily able to find the "teaching record" link on the left navigation bar.
• Is the word ‘Get’ on the teaching tab intuitive or confusing?
• How long did it take?
• What was their response to seeing their teaching record filled-in by the system?

Procedure

We met each participant in their office. One team member gave a short summary of our project as well as an overview of the testing procedure. We explained that this was one iteration of our prototype, and that we were seeking their honest feedback. We reminded them also that there were no wrong or right answers, and that we would like them to "think aloud" during the testing so that we could better understand their experience with our system. The participant was then given an informed consent form to sign before beginning the test.

One member of our team led the participant through each of the tasks. The other team member took observational notes and timed each of the tasks during the test.

Once the tasks were complete, the participants answered a survey of seven follow-up questions to assess their overall experience with the system (e.g. "Did you find anything particularly frustrating or pleasing about the system?"). They were also asked to indicate their gender and age range as part of this survey.

At the end of the testing session and interview, the participants was asked for any additional feedback or questions, and then thanked for their time.

Screenshots

Test Measures

For each of our tasks, we measured number of errors and time to completion, because these are relatively objective measures of efficiency. The starting point was measured at the time the participant put their hand on the mouse and the end point was measured at the final click to complete the task (e.g. clicking “save” to add a publication). We were especially interested in seeing whether participants became faster and more adept at using the system as they progressed through the tasks (i.e. Did they take less time to enter a second publication?).

We also wanted to see what drew a participant’s locus of attention and where a participant looked/expected to find links/buttons to perform certain actions.

We also wanted a more qualitative judgment of participants' reactions to our system. Did they like the "look and feel" of the system? Did they think that they would use this system in lieu of the paper-based Bio-Bib? Did they seem especially pleased, confused, or frustrated by any aspects of the system? We assessed these types of questions in two ways: by observing their facial expressions and reactions during each task, and also by directly asking them about their experience during an informal interview at the conclusion of the testing session.

Results

All users completed all of the tasks successfully. No errors appeared to be due to design flaws. Slower than expected times to completion on a few of the tasks were the result of either coding bugs from web browser incompatibility and screen size settings, or participant difficulty with non-Bio-Bib related subtasks, such as finding and copying a citation from a CV. Also, two-thirds (2/3) of the participants used the preview button to see their entries before saving them, which confirms for us the utility of this feature.

The one Assistant Professor, who had the longest completion time for the first two tasks had the shortest completion time for nearly all of the subsequent tasks. It appears that the learning curve may be less steep for younger or more computer savvy, professors even though they may be less experienced with filling out the Bio-Bib. In fact all participants learned to navigate the prototype quickly. Task #2 which asked participants to add a publication to their Bio-Bib was completed in an average time of 117 seconds (including time searching for CV). Task #3, which asked participants to repeat the process of adding a publication to the Bio-Bib was completed in an average time of 52 seconds (including time searching for CV).

All participants experienced some confusion with regards to the citation style question, when entering a published work. They either were not sure of their style (APA, MLA etc) or did not see it listed in the drop-down box. All three participants selected “unknown” for citation style. One participant was concerned that by selecting “unknown”, how his entry displayed may change when he previewed and saved it, or if his entry would be ‘tested’ in some way to see if it were ‘correct’. Participants were not sure how important it was to select the correct style, but they did not want to spend extra time researching it.

When asked to rate their satisfaction with the Bio-Bib prototype on a scale of 1 to 5 (with 1= very dissatisfied and 5=very satisfied), participants gave the prototype an average satisfaction score of 4.5. All three participants stated they would prefer to use the online prototype rather than the current paper system. The sections of the Bio-Bib participants were the most satisfied with were those that provided automatically filled data (eg. Teaching record) and cutting and pasting ability (eg. Published works entry).

Formal Experiment Design

Description

If we were to do a formal study for this design, we would compare two variations on our design decision to change the tabbed navigation system within our prototype. As mentioned earlier, our previous prototype had ten tabs at the top of the page with the following labels: home, awards, teaching, advising, research, publications, creative works, UC Committees, Service, and Preview. We felt that this looked cluttered, and it also duplicated information that was presented in the left navigation. More importantly though, the tab categories were not consistent; home and preview are system-level functions, while the tabs in-between are sub-sections of the Bio-Bib.

The latest prototype – the one that was tested in the usability study, presented a wholly redesigned navigation structure. The ten tabs at the top of the page were replaced with three simple system-wide functions that were quasi-modal in nature: home (view current drafts and create new drafts), instructions (general instructions on how to complete the Bio-Bib online), and preview and submit (this is a final step to submitting the form). We use the term modal because the options and functions change contingent on the selected mode, but all modes impact the entirety of the report. For example, the ‘Home’ (edit mode) tab allows functionality to select, insert, or modify a specific Bio-Bib, the ‘General Instructions’ tab (preparation mode) allows explanation and instructions on how to complete a Bio-Bib, and the 'Preview and Submit' tab (review mode) allows functionality to preview and submit a Bio-Bib. To complete sub-sections of the Bio-Bib, users must be in edit mode and can utilize the left navigation bar, which provides links to each sub-section (e.g. teaching record, published works, creative works).

A very interesting test would be to compare the two versions of our prototype to see which navigation structure users prefer. The goal of our test would be to evaluate which navigation was more effective.

Hypotheses

We hypothesize that the modal 3-tab navigation system (large menu) will be easier to learn and more efficient than the 10-tab navigation system (small menu).

We also would be interested in seeing the effect of font size on learnability and efficiency, so we would expect that larger font sizes in the navigation bar would be more learnable and easier to use than a navigation bar with smaller font size.

Factors and Levels

• Navigation interface
• Levels
• 3-tab navigation
• 10-tab navigation
• Font size
• 10-point font in the navigation bar
• 12-point font in the navigation bar
• Task types
• Levels: 5 tasks

• Dependent variables
• Participant satisfaction
• Task completion success
• Task completion time

Rationale

Our dependent variables would allow us to assess whether or not we could accept or reject our hypotheses because we include both quantitative and qualitative measures of learnability and effectiveness.

Blocking and Repetitions

We would use a between-subjects design, varying the menu size and font size. To block out the experiment, we would use our 5 tasks, with 5 participants in each of 4 conditions. Latin square would not work in this situation because of the learnability effects that would occur when having participants in multiple conditions.

Discussion

For a "real" usability test, we would make the following four changes:

1. Videotape the tests. Although planned, due to unforeseen circumstances (illness, scheduling), our team could not videotape the tests and as a result a lot of detail was lost to memory. However for a "real" experiment, videotaping would be necessary for later review of details such as participant focus, actions, and more accurate timing of tasks.
2. Have one person in charge of measuring time. Because of scheduling conflicts, not all three team members could be present for all tests. Having one person in charge of time keeping would free the note-taker to focus on the participant’s actions, emotions and reactions to the prototype.
3. For more accurate times, we would arrange for the test coordinator to become "invisible" during actual task tests. The participant would be instructed to pretend that they are alone with the prototype and that no one is around to help him/her with it. This would better encourage the participant to think aloud but focus on the task and not ask questions of the coordinator. This would help us log more accurate task completion times.
4. We would have a larger number of participants (8-12) from a more diverse set of backgrounds with regards to ladder rank, age, sex (all of our participants happened to be male), discipline and experience with the Bio-Bib.

What you might change in your interface from these results:

After group discussion about redesign possibilities raised by the usability test results, we agreed upon the following substantial design changes for the next iteration. We felt that the changes, discussed below, best addressed the needs of newer and more experienced professors alike.

We will make the tabs at the top correspond to modes: Preparation, Active Bio-Bib, and Submission. This change will also provide a sense of progression and location within the Bio-Bib completion process. The Preparation mode will be represented by a tab called "Instructions". Within this tab, we plan to have a section that provides background information (What is the Bio-Bib?), an explanation of the features and terminology within the Bio-Bib, as well as an overview of instructions/tutorial for its completion. The Active mode will be represented by a tab called “Select year”. This will also be the default first page users will see after they log in. In the selection tab, users will select from one of three choices: 1) create a new Bio-Bib, 2) continue/edit a current (un-submitted) Bio-Bib and 3) view a previously submitted Bio-Bib. Once a user selects a year, a third and fourth tab appear for the first time. (If a third option is selected, only the fourth tab will appear). The third tab shows the active year(s) in which the user is working. In this tab, the left menu of Bio-Bib subsections (teaching, published works, etc) appears for the first time. It is in this mode-tab that users will complete the Bio-Bib. When the Bio-Bib is complete, the user will have a final preview and opportunity to submit the Bio-Bib in the fourth tab.

Also based on feedback from this usability test, we decided to reverse an earlier design decision and reintroduce the "complete" button. This would give the user immediate feedback on the left menu of which subsections of the Bio-Bib had been “marked as completed” by the user. Other design changes we plan to make will be to add icons next to our rollover ToolTips to make them more noticeable, adding descriptions of where all auto filled data comes from and who to contact if errors are detected within it, and adding a place for "fellowships/grants" in the overall structure. In the published works section, we plan to 1)switch the order of the citation text box and a select citation type drop-down box, 2) add "convention papers" and "software" to the publication type list, 3) make entry fields required to reduce errors.

Below is a list of re-design possibilities that resulted from observations of and discussions with each participant.

• Participant #1
1. Putting the citation example link in a bigger font (participant did not notice it until it was pointed out).
2. Look up Oxford and Harvard Press Styles.
3. Eliminating the instructions tab, moving all instructions to right side and making them page specific.
4. Add an icon next to the tool tips to make them more noticeable (Wallace did not notice the tool tips)
• Participant #2
1. Communicate that citation format is not important ("Optional")
2. Eliminating General instructions tab, replacing with page specific link (on right side?)
3. Adding a "What is the Bio-Bib?" link for new and Junior faculty. Discusses purpose of Bio-Bib not just from administration’s perspective of how it aids the University, but also for new faculty "What is the Bio-Bib to me? How does it help me. Where does the information go, who uses it for what".
4. Adding a tutorial or introduction that shows the scope of the Bio-Bib (what type information will be requested, what materials / documents will be needed) and also makes the process more transparent.
5. Making the entry fields (ex. publication type) required to reduce errors.
6. The COMPLETE button. Participant 2 questioned the usefulness of the time stamp without some indication that the section was complete. Suggested and supported a “complete” button and some meter that showed "ex. 33% complete" based on % of "completed sections."
• Participant #3
1. Asking for a month and year for publication in tree to help publications be automatically organized by academic year in future iterations of the prototype.
2. Switching the order of the citation text box and select citation type drop-down box. Participants were drawn to the biggest object on the page (text box) and almost missed the select drop-down box.
3. May need to increase the size of the text of the left navigation links, and the "edit"/"delete" buttons.
4. Adding a place for "Fellowship/Grant s" and overall structure.
5. Adding "convention papers" and "software" to the publication type list.

Work Distribution