Assignment 4Introduction
We conducted our first usability tests on a paper prototype of the online Bio-Bib form. In our current conceptualization, the Bio-Bib is being designed to ultimately facilitate data entry into five major categories:
- teaching
- advising
- publications
- creative works
- professional service/activities
For the remainder of the project we have decided to further limit our scope to the categories of publications and creative works. For this first round of testing, we focused solely on the publication section because of time constraints. We will incorporate the creative works section into the second round of usability tests.
The purpose of the tests was to get assess whether the navigation flow was intuitive and efficient to participants as well as get feedback on the overall design and appearance of the prototype. Our goal for the test was to identify through tester feedback, any aspects of the prototype’s content, design and structure that were missing and/or confusing and thus would need to be redesigned in our next prototype iteration. top of page
Prototype
In developing our first prototype, each team member independently drew (freehand) our own ideas of the design, layout and organization based on our discussions of the interviews, personas, scenarios and sketches from the previous assignment.
In the following meeting, each member described his/her drawings and as a group we decided which elements would be the most useful for our prototype. After compiling the best design features from each draft, each of us assigned ourselves to develop an interaction flow for a different section of the prototype. In the next meeting, we discussed each flow, compiled the best features of each and developed a consistent flow standard.
We then created the test prototype in Illustrator. (To view the prototype in full, click here).
Method
Participants
We used three participants from Boalt Hall School of Law for our low-fi prototype test. One participant was a tenured faculty member who was also serving as the Associate Dean of the law school, and the remaining two participants were designates for professors in the law school. Although we ideally would have liked to get representation from a variety of academic departments, scheduling conflicts limited us to these three participants.
We targeted our first participant for recruitment because we wanted to get the perspective of someone who fills out the Bio-Bib report for himself, and often reviews the Bio-Bib reports for other professors in his department. Participant 1 has been a professor in the law school for over twenty years, and fills out the Bio-Bib report each and every year. He has also been the Associate Dean of Boalt for the last three years.
Participants 2 and 3 work in the Law-Faculty Support Unit at Boalt, and were referred to us by participant 1. Although they themselves do not have to complete the Bio-Bib reports because they are not faculty members, they are often designated with the task of completing the Bio-Bibs for professors that they work with. Because it is not unusual for support staff to complete the Bio-Bib for faculty members, we felt that it was important to gather input from this group of users.
Procedure(including task scenarios and test measures)
All three team members were present at all of the interviews, and we took turns acting as the tester, the “computer,” and the note-taker. We began each interview by introducing ourselves and describing the goals and scope of our project. We identified each of our roles during the test, and explained that we were gathering feedback on the first paper prototype of our proposed system. We emphasized that were seeking their open and honest feedback on our drafts, and that there were no wrong answers. Before we began with the tasks, we asked them to look over the consent form for the project, and sign if they agreed to participate (they all agreed).
The tester explained to participants that during the test, they would be asked to “think aloud” as they completed four tasks. In order to make our participants feel more comfortable with the process, the tester demonstrated an example of the “think aloud” process by completing the task of logging into the computer system with a calnet ID.
Before beginning with task 1, we asked participants three background questions in order to understand their function and familiarity with the Bio-Bib process:
- What is your name and function? Within which department do you work?
- Can you briefly describe the nature of your work?
- Have you ever filled out the Bio-bib form? If so, how many times?
The user was then presented with the four tasks, one at a time. The tasks were all designed around the “publications” portion of the Bio-Bib report, because we wanted to understand how users approached the entry of journal articles, books, and other published materials. We made note of when them seemed confused, when it took longer to complete a task, and when they made errors, but we did not collect more specific quantitative measures such as response time.
When necessary, we reminded them to “think aloud,” and if they seemed stuck, we asked them to explain what was confusing to them. If they indicated that they would perform an action that we had not anticipated, then we presented them with a blank screen that said, “We have not yet completed the design for this portion of our system. What would you expect to see here?”
Tasks
Task 1:
Imagine that you are a Professor of Law at Boalt named Alison Randall. You published 3 articles in the last 3 years, and would like to enter this information into your Bio-Bib form for your upcoming review. Start by using the information below (for your first article) to enter information into the Bio-Bib form. Note that you are the primary author of the article, and your colleague Mark Lund is the secondary author.
Randall, A. & Lund, M. (2005). Trademark Law in the 18th Century, Harvard Law Review, 23(3), 150-164.
- Did they immediately see the ‘Publication type’ drop-down box, and did they know which item to select from this menu?
- Did they correctly enter their name as an author for the publication? Did they seem confused or irritated that they had to enter each author individually?
- Did they understand what the “role” drop-down box meant?
- What did they expect to happen when they clicked on “add to author list?”
- What did they expect to happen when they clicked on “add to publication list?”
- What was their reaction to the summary page after they added the first publication?
- Did they know how to add another journal publication after reaching the summary page?
Task 2:
After you finish entering the information for your Harvard Law Review article, you realize that you accidentally misspelled your own name! Add a letter “l” to your name in order to correct the spelling to “Allison,” and resubmit the article to your Bio-Bib.
- What did they do in order to change the spelling of their name? How long did it take them to find the “edit” button?
- How would you save your changes? Did they see the “save” button, or did they go immediately to “add publication?”
- How would did they submit the entire publication to their Bio-Bib?
Task 3:
Assume that you have already entered your three articles into the Bio-Bib, and you would now like to enter a different type of publication. Last year, you wrote a chapter titled “From Racine to Beaumarchais: Following the Barber,” for a book titled “European Dramaturgy in the Eighteenth Century.” The co-editors of the book are Paul Norman and Janet Jewell. This book has been accepted for publication by the Yale University Press in New Haven, Connecticut, but is currently in press.
Please enter this publication information into the Bio-Bib.
- Did they immediately see the ‘Publication type’ drop-down box, and did they know which item to select from this menu? Did they seem more comfortable with the process than they did in task 1?
- Given that the book has two editors, how did they input this information? Did they seem confused that the author and editor fields looked different?
- What did they do for the series title? Did they seem bothered if they left it blank?
- Did they understand the terminology “published” versus “pre-print?” What year did they select for the pre-print? Why?
Task 4:
Last year, you served as an expert witness for a judicial hearing on employment discrimination. Please enter this information into the Bio-Bib.
- How did they say that they would navigate to the correct place on the screen in order to enter this information?
- Was it clear to them from the title of the tab “Professional Service” that this was the correct location?
- How did they feel about skipping the “Creative Works” tab and going directly to the “Professional Service” tab (i.e. did they think that the tabs had to be completed in the order in which they were presented)?
- Does the overall organization and terminology for tabs make sense? Are the categories clear?
After the participant completed the four tasks, we followed up with some general feedback questions:
- What were your general feelings about the interface? Was there anything that was confusing to you?
- Let’s revisit the journal entry page for a moment. Was it confusing to have the author field first, or would you have preferred it to come later?
- Does the terminology that we used make sense to you (i.e. role, peer refereed, student refereed, archived, published, pre-print)? If the terminology is confusing, what would you suggest instead?
- Do the names of the tabs seem clear to you? If we were to provide tips or examples, where would you want those to be?
We ended our meeting by thanking the participant for volunteering to help us with our study. top of page
RESULTS
Task 1:
In this first task, two of the three participants correctly indicated that they would click on the “publications” drop down box in order to enter a new journal article. One participant said that she would try to enter information into a grey box that read “no publications have been entered,” because she assumed that it was a text box that she could copy and paste a full citation into.
The terminology for some of the fields within the journal article entry box seemed confusing to participants. The difference between co-authorship and non-primary authorship was not clear to all three participants because in the field of law, the lines between these two types of authorship are blurred. They also did not know what “refereed” meant, although participant 1 was able to guess that “student-refereed” meant student-edited.
He explained that most leading journals are not peer reviewed in the field of law, and though some are student edited, it is not always obvious which it is. Also, the referee status does not directly correlate with status of the publication. None of the participants knew what “archival” meant. The format of the citations that were presented on the summary page were not in the form that the law school typically uses, as pointed out by all three participants.
Two of the three participants were insistent that they wanted a “preview” button before submitting changes (they did not anticipate that they would be presented with a preview AFTER they added each publication. In addition, when asked what they would do to add another journal article, two of the three participants took a bit of time and thought to realize that they would have to go up to the “add publications” drop down menu again at the top of the page. They seemed confused by the edit and delete options listed on the summary page.
Task 2:
Although all three participants correctly identified the “edit” button to change the spelling of their name, the “save” button was missed by 2 participants. All three of the participants seemed pleased that the text boxes around the author fields were highlighted in yellow after clicking the “edit” button, so that they knew that they could edit those fields.
Two of the three participants also reiterated that they wanted a “preview” button that led to a preview page before submitting changes. It was not immediately clear to all three participants that they would select “add publication” at the bottom of the page in order to save changes, though they said that they would select this button because it was the only option that was presented. One participant expressed concern that she might duplicate the publication with a slightly different spelling of her name if she clicked “add publication” after changing the spelling of her name.
Task 3:
In this task, all three participants immediately went to the “publications” drop down box in order to add a book chapter to their list of publications. Response time seemed considerable faster than task 1 for clicking on this initial drop down box. Two of the three participants also noted that they wanted the “editor” field to look identical to the “author” field, in which names are inputted individually. The first participant also said that he expected his name to appear automatically in the author field, since he had already entered a previous publication. He wanted his first and last name to be pre-filled, with the role unselected.
The term “pre-print” was confusing for all participants. Participant 1 explained that he is not accustomed to entering publications into the Bio-Bib that have not been officially published. He suggested that if some faculty wanted to enter publications that had not been printed, they must be at the final stages of “in press,” and that this terminology was slightly more clear to him.
The participants seemed pleased that they summary page separated the publications by type: journal articles and book chapters. Participant 2 also mentioned that she would like to see referee and archive status on the summary page, as well as have the functionality to sort entries by date and last name of first author.
One interesting issue that emerged from this task was the distinction between being an editor of a book, and being the primary author of a book. Participant 1 mentioned that he was recently the editor of a book, but because he put so much work into it, he would consider himself the primary author. After some thought, he suggested that we have an “editor” subcategory under “books,” and then list these books separately under “edited books” on the summary page.
Task 4:
All three participants said that they would put expert witness testimony under the “professional service” tab. Participant 1 mentioned that he would also include things like briefs under the category of professional service.
All three participants also said that they would feel comfortable skipping the creative works tab, but that they would probably look under all tabs to see what they contained. Participant one said that in terms of the organization and naming of the tabs, he would group them as teaching, research, and service (broken down into university service and professional service). He would group “creative works” under research because he regards creative works and publications and somewhat equivalent, but he acknowledged that this grouping might confuse other people.
Overall:
The overall reaction to our paper prototype was positive from all three participants. They were not bothered by the fact that they would have to enter information for publications individually, but one participant did mention that it would be easier to be able to cut and paste a full citation from a CV. All participants also indicated that the colors and “look and feel” of the paper prototype was pleasing to them.
Participant 3 suggested that we add a first tab for “personal information,” with the faculty member’s name, level, and other background information. As a designate for faculty members, she felt it was important to have a place to check and edit this information if necessary.
We also showed participants one screen layout for journal entries that was organized slightly differently from what they had been tested with: the author, article, and journal fields were separated by boxes. All participants preferred this second version because it was more “clean.”
Discussion
We learned from our first round of usability tests that our first prototype was, on the whole, generally well designed and understood. All of the participants were ultimately able to complete the tasks given to them.
Four small but immediate changes we plan to make are to:
- remove the shading from around the text "No publications entered" when users first access a tab. It was suggested that the shaded area, which appears in the form of rectangle, may suggest that it is a text box.
- pre-fill the professor's name in the author section of the publications tab. This will save the users a bit of time.
- add an 'Editor' option under Book in the publications tab.
- We plan to move the Advising tab next to the Teaching tab or combine the two, since both are often linked together in professors' minds and keeping the tabs on opposite sides of the page creates unnecessary cognitive load.
Therefore we may reevaluate some aspects of the prototype in our next round of tests by offering comparisons. For example, another possible change may involve the creation of an extra preview button. We may, in our next test, offer the participants an additional scenario with and without the extra preview option to get feedback on its necessity.
Other possible changes include:
- adding a bibliographic formatting tool similar to endnote so that different departments can view their publications entries in formats of preference
- changing the terminology of roles for authorship because the current options may be confusing in some departments. This is a more difficult issue to resolve than
- programming archival and referee information for journals on the backend. Users would select a journal title from a list. Information about the journal's archival and referee status would be automatically encoded. The user would never have to see the questions on the screen. However, this option would involve long term maintenance to keep the journal fields accurately pre-populated with titles.
top of page
Work Distribution
Dondrea |
Eunice |
Rich |
|
| Interaction Flow Processing | 33% |
33% |
33% |
| Interaction Flow Implementation | 0% |
0% |
100% |
| Prototype Script | 0% |
100% |
0% |
| Prototype Testing | 33% |
33% |
33% |
| Introduction, Results, Discussion | 100% |
0% |
0% |
| Prototype, Methods, Test Measures | 0& |
100% |
0% |
| Web Page | 0% |
0% |
100% |