Introduction
The Berkeley Calendar Network tool is a web-based application designed for the U.C. Berkeley community. It allows any campus group or department to add their events to a centralized repository in order to facilitate sharing of event information. Events can be designated as "private" in which case they are only available to the event owner's calendar or "public" in which case they are available to any calendar in the network. Each group can use the tool to customize its own dynamic, web-based calendar showcasing its events. The tool also allows users to pick and choose specific events from other calendars or set up a subscription to automatically receive events according to some criteria (e.g."lectures and seminars sponsored by Haas").
The purpose of our testing was to determine whether the flow of the application was natural, easy to learn, and easy to use, whether the features we provided were valuable to users, and whether we were missing any additional features. In addition we wanted to test the improvements made to functionality, consistency, language, interaction flow, and design since the last interactive prototype.
[Top]Method
Goals
- Ensure that participants can find and use the various functions in the interface
- Determine whether the functionality provided is valuable
- Determine whether the interaction flow is natural and intuitive and matches current user workflow
- Determine whether the application will meet the needs of these calendar administrator
Participants
We had three participants for our study. Two were female and one was male. Two of the participants are currently calendar administrators on campus and the third is a calendar content manager. One of these participants had a calendar which is an aggregator, and two were fairly high-level calendars which primarily posted their own events but sometimes or often had a need to share their events with other departments or organizations on campus. All of the participants are very familiar with web-based applications and other calendaring tools.
Apparatus
Our usability tests were performed in the upstairs computer lab of South Hall. Because our prototype was designed with an intended screen resolution of 1024x768 we used a workstation configured to that resolution. Every subject used the same computer running Microsoft Windows 2000 and used the Internet Explorer 6.0 browser for running through the prototype tasks.
Task Scenarios
Scenarios 1 and 2 are the same scenarios that we used for the earlier testing in Assignment 4. Scenario 3 was revised to include an extra step that would test new functionality that was added since the last prototype.
Scenario #1: Post, Edit, & Delete Events
You are the calendar administrator for the Mechanical Engineering department. You have been using the calendar system for a few months, and subscribe to events from many different calendars. You have chosen to manually approve all events that you subscribe to before posting them to your calendar. Today you are logging in to review new events that came in over the weekend.
- Login with username: me and password: me
- Review your pending events
- Post all EECS events to your calendar (3 events). You are familiar with EECS events and don't feel it necessary to review each one in detail.
- You are not sure whether you want to include the Bio Engineering "Guidant Information Meeting" and so decide to review it more closely.
- You decide that you don't want to post it on your calendar and want to remove it from your pending events list.
- You received an email asking you to change the time for the "ME Grad Visiting Day" on 4/16/04 to 2-4pm. This event has already been posted to your calendar. Find it and make the time change.
- Logout
Scenario #2: Add Event & Administer Calendar
You are the calendar administrator for the Mechanical Engineering department and need to add a new event to your calendar and change the background color of your calendar.
- Login with username: me and password: me
- Add the following event:
Date: 4/25/04
Time: 1-2pm
Title: How Smart Helmets Save Lives
Event Type: Lecture
Location: Etcheverry Hall - Post it to your calendar immediately
- Find and review the event you just posted
- Change the background color of your calendar to blue
- Change the site navigation that is displayed on your calendar to vertical
- Logout
Scenario #3: Find Campus Event, Set Up Subscription & View Departmental Calendar
You are the calendar administrator for the Biology department and have been using the system for a few weeks, but so far only for your department's own events. Today you want to start adding some other campus events to your calendar.
- Login with username: biology and password: biology
- Find the following campus event (it is not currently on your calendar):
Environmental Science, 4/8/04, 4:00pm: "The Conquest of Bread: 150 years of CA Agribusiness" - Post it to your calendar
- Set up a subscription to receive all seminars sponsored by the Civil & Environmental Engineering and Environmental Science departments that contain the word "environment" in the description. You want to manually approve all events before they post. You want to receive events that occur 2 weeks ahead or less. Name this subscription "Environmental Seminars."
- Look at your department calendar
- Logout
Procedure
All four of our team members were in the room for each testing session. One team member conducted the testing session. She introduced the application, explained the purpose and logistics of the study and described each task scenario. She was the only one who spoke during the testing. The other team members observed and took notes during the testing. One timed how long it took for the participants to complete each task. All of the team members were involved in asking questions at the end of each testing session.
Each participant was given the three task scenarios described above. The test conductor handed out three task sheets describing each scenario. The participant was given a chance to read through the tasks and ask any questions before she began. For each task, the participant was encouraged to think out loud and explain her actions as she interacted with the computer. After all three tasks were completed, there was a brief interview in which all four team members participated. The participant was asked to share her overall impressions of the interface, tasks and workflow as well as to answer some specific questions about the system. Each testing session lasted approximately one hour. Participants were then given a brief survey to complete (see appendix).
[Top]Test Measures
For each participant we measured how long it took to complete each task and noted any significant errors made during each task in the User Testing Logs. We also measured each participant's opinion regarding ease of use and overall satisfaction with the application in the survey. Our Results table summarizes these measurable findings.
[Top]Results
Task Time Completion
Participant |
Scenario 1 |
Scenario 2 |
Scenario 3 |
| 1 | 3m48s | 4m16s | 6m05s |
| 2 | 3m17s | 4m28s | 6m09s |
| 3 | 3m55s | 4m45s | 5m14s |
| Average Time/Task | 3m40s | 4m30s | 5m49s |
Survey Answers
For each of these questions, participants were asked to choose an answer from the following scale:
- 1 - Disagree
- 2 - Somewhat Disagree
- 3 - Neutral
- 4 - Somewhat Agree
- 5 - Agree
Question |
Participant 1 |
Participant 2 |
Participant 3 |
| 1. I found the Berkeley Calendar Network (BCN) system intuitive and easy to use. | 5 - Agree | 4 - Somewhat Agree | 4 - Somewhat Agree |
| 2. I think the BCN system would be a valuable tool for many campus departments. | 5 - Agree | 5 - Agree | 5 - Agree |
| 3. I think the BCN system would be a valuable tool for my department. | 5 - Agree | 5 - Agree | 3 - Neutral |
| 4. The BCN system would be an improvement over my department's current event management system. | 5 - Agree | 5 - Agree | 1 - Disagree |
| 5. My department could not use the BCN system because key features that we need are missing. | 1 - Disagree | 1 - Disagree | 1 - Disagree |
| 6. Although I don't think my department could use the BCN system as our primary event management system, I could imagine using it to send events to other calendars on campus since I would only have to enter the events in one place to send them to multiple calendars. | 3 - Neutral | N/A | 5 - Agree |
| 7. The "Format Calendar" sections (including the ability to upload a cascading stylesheet or XSL transform) seemed to cover all the options I would need to customize my calendar's look and feel. | 5 - Agree | 3 - Neutral | 4 - Somewhat Agree |
| 8. The ability to "Recommend" events to other departments is a valuable feature. | 5 - Agree | 5 - Agree | 5 - Agree |
| 9. The ability to "Export" events in an Excel, comma delimited, or standard calendar format is a valuable feature. | 5 - Agree | 5 - Agree | 3 - Neutral |
| 10. The ability to create a "Subscription" is a valuable feature. | 4 - Somewhat Agree | 5 - Agree | 5 - Agree |
| 11. The way the "Event Manager" and its component parts (Pending, Posted, & Archived Events, Create Event, Search Campus Events, Subscribe, and Export) are organized and grouped together is intuitive to me. | 5 - Agree | 4 - Somewhat Agree | 4 - Somewhat Agree |
| 12. I would find it valuable to easily view my live calendar through the BCN tool by clicking on "Calendar" as opposed to simply navigating to my calendar on the web. | 5 - Agree | 5 - Agree | 5 - Agree |
| 13. I like the fact that both the "Calendar" and "Format Calendar" options are exposed to me on the navigation bar. I prefer this to having both of these options under a single "Calendar" option where I had to do an extra click to get to at least one of them. | 4 - Somewhat Agree | 5 - Agree | 3 - Neutral |
| 14. I like the fact that as I make changes in the "Format Calendar" section, I can see a styled version of a generic calendar so I can tell what my changes will look like. | 5 - Agree | 5 - Agree | 5 - Agree |
| 15. I would like to have the ability to do a search not only on the "Posted" tab of the "Event Manager," but also on the "Pending" and "Archive" tabs. | 5 - Agree | 5 - Agree | 5 - Agree |
| 16. I like being able to see a nicely formatted summary of event information by clicking on the event's link, and using an "Edit" button to go to the "Create Event" form when I needed to make changes to an event. I prefer this to having no link from the event's name, and having two buttons, one for "Edit" (which will be visible only for my own events and take me to a populated "Create Event" form) and one for "View" (which would take me to the nicely formatted summary of the event information). | 5 - Agree | 5 - Agree | 4 - Somewhat Agree |
| 17. The event search options in "Search Campus Events" include everything I would need. | 5 - Agree | 5 - Agree | 5 - Agree |
| 18. The "Subscriptions" page and the "Search Campus Events" page show two different ways of presenting the same search options. In "Search Campus Events," the more detailed options are hidden behind an "Advanced Search" link. I prefer this method to the way all the search options are displayed on the "Search Campus Events" page. | 4 - Somewhat Agree | 5 - Agree | 5 - Agree |
| 19. I like the way the "Create Event" form is organized and do not find it too overwhelming. | 5 - Agree | 4 - Somewhat Agree | 4 - Somewhat Agree |
| 20. I would like to be able to customize what fields are hidden and shown by default in the "Create Event" form. | 3 - Neutral | 5 - Agree | 5 - Agree |
| 21. I found the workflow of the system natural, and think it follows my normal workflow when dealing with events. | 5 - Agree | 5 - Agree | 4 - Somewhat Agree |
Problems Experienced by Participants
- Event Manager
- Users had trouble finding some functions in the sub-nav
- Users would like to be able to search all tabs
- Users did not think it was clear that the search boxes in each tab acted on that tab only
- Users did not notice confirmation messages
- Format Calendar
- Users were confused about the difference between “Preview Calendar” in the Format Calendar section and “Calendar” on the nav
- Restore default settings was unclear
- Event Details: The flow from this page doesn’t make sense if the user doesn’t want to take any action
- Subscription: It is confusing to include “Event Date” in the list of choices offered
- Create Event form : The concepts of a “Public/private” event and an “Event Listing Contact” weren’t clear to users
Discussion
Overall, our participants were pleased with our application. They found the system relatively easy to use, found that the interaction flow was natural and intuitive and matched their current workflow, and thought that the functionality provided was valuable. However, some thought functionality that they wouldn't use in their department, such as the Export function, was not as valuable. All participants thought this application would be helpful to many campus departments including their own. We were even surprised to find that one high-level calendar owner who had functionality on her site that we do not plan to reproduce in this release still had a use for our system. Even if she didn't use our application to create her website, she thought using the Create Event form in our application would be a great way to send event information to multiple departments. This is a task she regularly performs now by visiting multiple sites and entering data individually for each calendar.
Our testing also verified that some interface design decisions we had made were helpful to our users. All participants liked being able to view their live calendar through the application, and most were really excited that we had provided a preview of what their formatting changes would look like in the Format Calendar section. We verified that users preferred to be able to click on the event's link to see a nicely formatted summary of event information, and appreciated the shortcut "Edit" button in the Event Manager which allowed them to bypass this screen and go directly to a populated version of the Create Event form. We also verified that the search options we offered in "Search Campus Events" seemed to be the right ones, and that the Create Event form was not too overwhelming, thought it was clear that it could still use some explanatory text for some of the more complicated fields. Two of the three participants thought it would be helpful to be able to be able to customize the way the Create Event form appeared to the user in the application.
Changes We Plan to Make to the Interface
Based on the results of the usability testing, we plan to make the following changes to our interface:
- Event Manager
- Change navigation so Event Manager includes a “Department Name” Calendar Events section which will encompass the Pending/Posted/ Archived tabbed section
- Put search functionality on all three tabs and change label to “Search Posted Events” etc
- Add green background to confirmation messages
- Create Event Form
- Change “Event Status” to “Event Sharing Status”
- Add explanatory text to “Event Listing Contact”
- Navigation
- Change “Calendar” on nav to “View Live Calendar”
- Format Calendar
- Add explanatory text to “Preview Calendar”
- Remove “Restore Default Settings”
- Event Details
- Change “Cancel” button to “Back”
- Subscription
- Remove “Event Date” from the options offered
Changes for the Real Experiment
In a more formal usability study we would first pre-test our task scenarios carefully to ensure that there was no unnecessary confusion due to their wording. This would include making sure that all terms used in the scenarios matched those found in the interface exactly. We would also further segment our task scenarios so that each scenario represented a truly distinct task. This would allow us to time each individual action instead of having times that conceptually encompass multiple tasks. It would of course be helpful to develop more task scenarios which covered a wider range of our application's functionality. We were unable to accomplish this for this usability testing session because we did not have enough time to build out all our planned functionality.
Formal Experiment Design
Hypothesis
Users will more easily be able to locate the link to their live departmental calendar in the newer version of the navigation.
Factors and Levels
The Factor is the version of the navigation. The two levels are:
- O - Old version in which the link to the live departmental calendar is one of the main items in the top level of navigation and is called "Calendar".
- N - New version in which the link is separated from and placed to the side of the main navigation items. The link is called "View Live Calendar".
Blocking and Repetitions
The experiment will be conducted with 16 participants who have similar levels of web application and calendar application experience. The participants will be split into two groups of 8 each. Each group will conduct the same task of locating and viewing their departmental calendar using both versions of the navigation. Group A will see the old version (O) first and Group B will see the new version (N) first. Time to complete the task will be measured for each version of the navigation and the participants will be asked to rate the ease of use for the task upon completion of the task with each version of the interface.