Introduction
The Berkeley Calendar Network tool is a web-based application designed for the U.C. Berkeley community. It allows any campus group or department to add their events to a centralized repository in order to facilitate sharing of event information. Events can be designated as "private" in which case they are only available to the event owner's calendar or "public" in which case they are available to any calendar in the network. Each group can use the tool to customize its own dynamic, web-based calendar showcasing its events. The tool also allows users to pick and choose specific events from other calendars or set up a subscription to automatically receive events according to some criteria (e.g."lectures and seminars sponsored by Haas").
The purpose of our testing was to determine whether the flow of the application was natural, easy to learn, and easy to use, whether the features we provided were valuable to users, and whether we were missing any additional features. [Top]
Prototype
Our lo-fi paper prototype consisted of wireframe screenshots of our task scenarios. The wireframes were intended to show the basic layout and interaction of our designs without additional detailed graphics. The GUI elements (i.e. buttons, text boxes, text fields, etc.) were created from a template and looked similar on all of our screens. Each screenshot included a browser frame to set the context of our web-based application. In addition to the screenshots, we printed stick-ons (i.e. dropdown menus, tables, etc.) to modify parts of the screen as users interacted with the interface. The paper prototype was printed in grayscale on 8.5 x 11 paper. We taped several pieces of paper together when we needed to create a longer, "scrollable" screen. In our design, certain parts of the screen would change color based on user interaction. In those cases, we marked our grayscale prototype with a colored highlighter.
View prototype screenshots based on task scenarios
Interaction Flow Diagram:
View Larger Diagram
[Top]
Method
Participants
Three campus calendar administrators participated in our study. Each of the participants' calendars has different goals and thus dictates different needs for the system. Participant #1 only wants to display events from her own department but wants multiple users within her department to be able to add and edit events on the calendar. Participant #2 wants to display her departments own events but also aggregate events from other departments. Participant #3 does not create any events and instead has a calendar intended to aggregate and display all of the events of interest to the Berkeley campus community.
Procedure
All four of our team members were in the room for each testing session. One team member conducted the testing session. She introduced the application, explained the purpose and logistics of the lo-fi testing and described each task scenario. She was the only one who spoke during the testing. One team member acted as the computer by choosing the correct "screen" to display in response to the participants actions; she did not speak during the testing except to explain the computer's actions. A second team member assisted with the computer by organizing the pieces of the paper prototype for each task but did not interact directly with the participant. She also timed how long it took each participant to do each task. The last team member sat beside the participant as an observer and took notes on a laptop.
Each participant was given three task scenarios. The test conductor handed out three task sheets describing each scenario. The participant was given a chance to read through the tasks and ask any questions before she began. For each task, the participant was encouraged to think out loud and explain her actions as well as to physically interact with the prototype by using her finger as the mouse pointer. After all three tasks were completed, there was a brief interview in which all four team members participated. The participant was asked to share her overall impressions of the interface, tasks and workflow as well as to answer some specific questions about the system. Each testing session lasted approximately one hour.
Task Scenarios
Scenario #1: Post, Edit, & Delete Events
You are the calendar administrator for the Mechanical Engineering department. You have been using the calendar system for a few months, and subscribe to events from many different calendars. You have chosen to manually approve all events that you subscribe to before posting them to your calendar. Today you are logging in to review new events that came in over the weekend.
- Log in
- Review your pending events
- Post all EECS events to your calendar (3 events). You are familiar with EECS events and don't feel it necessary to review each one in detail.
- You are not sure whether you want to include the Bio Engineering "Guidant Information Meeting" and so decide to review it more closely.
- You decide that you don't want to post it on your calendar and want to remove it from your pending events list.
- You received an email asking you to change the time for the "ME Grad Visiting Day" on 4/16/04 to 2-4pm. This event has already been posted to your calendar. Find it and make the time change.
Scenario #2: Add Event & Administer Calendar
You are the calendar administrator for the Mechanical Engineering department and need to add a new event to your calendar and change the background color of your calendar.
- Login
- Add the following event:
Date: 4/25/04
Time: 1-2pm
Title: How Smart Helmets Save Lives
Event Type: Lecture
Location: Etcheverry Hall - Post it to your calendar immediately
- Find and review the event you just posted
- Change the background color of your calendar to blue
Scenario #3: Find Campus Event & Set Up Subscription
You are the calendar administrator for the Biology department and have been using the system for a few weeks, but so far only for your department's own events. Today you want to start adding some other campus events to your calendar.
- Login
- Find the following campus event: Environmental Science, 4/8/04, 4:00pm: "The Conquest of Bread: 150 years of CA Agribusiness"
- Post it to your calendar
- Set up a subscription to receive all seminars sponsored by the Civil & Environmental Engineering and Environmental Science departments that contain the word "environment" in the description. You want to manually approve all events before they post. You want to receive events that occur 2 weeks ahead or less. Name this subscription "Environmental Seminars."
Post-testing Interview Questions
- Would you want to push (suggest) events that you don't own to other calendars (e.g. to gain publicity for the event)?
- What is the most logical way to separate events in the Event Management section (e.g. Live vs. Pending events)?
- What kinds (if any) changes to events do you need to be notified of (e.g. canceled events)?
- Do you understand what the buttons mean (e.g. does the "Post" label make sense)?
- On what types of fields would you need to search your posted events (e.g. Title, Type, Description, Department)?
- Are there any event details that you would need to see about an event that normally would only be seen by the event owner (e.g. Sponsor Contact Info)?
- Would it be more helpful when you click "View" to see a preview of the event, or the event info in an Add form format?
- When entering an event, would you ever enter "related events" so that users who searched and found your original event would be pointed to the related events?
- When editing your events, would it be helpful to first see a preview of the event as it would appear on the calendar (in list format), or would you rather go straight to the Add form?
- Would you more often set up a filter to send events you may be interested in to your work queue, or search for a specific event? Would it be annoying to always do a search first to see what kind of results you would get before adding a filter?
- Within your event subscriptions, would you want to limit how far in the future the events you receive occur?
Test Measures
- Is the flow of our application natural and easy to learn, even for first-time users?
- Do the different sections of the application make sense and map well to tasks currently performed by calendar owners?
- Are the labels, categories, and functions of the application clear?
- Have we provided all the functionality a calendar owner would need? Have we provided too much functionality?
- Which of our two proposed types of search interfaces do our users prefer? Are we providing the right number and type of search fields to the users?
- Does the application contain too many confirmation screens?
Results
| Task | Subject #1 | Subject #2 | Subject #3 | |
|---|---|---|---|---|
| Scenario 1: Post, Delete, & Edit Events | ||||
| Find Pending Events | 1 | 2 (not sure which department the events are for) |
1 | |
| Understand Meaning of Pending Events | 3 (confused about whose calendar she was dealing with) |
2 | 1 | |
| Understand Meaning of Source vs. Sponsor | 1 | 3 | Did not mention | |
| Multiple Select Pending Events | 1 | 1 | 1 | |
| Post Pending Events | 1 | 2 | 3 (though he might want to choose export) |
|
| Understand that newly posted events are put yellow, at top of Posted Events | 4 (didn't think was important in a short list) |
2 | 1 | |
| Understand Select All | Didn't notice | 4 | Didn't notice | |
| Review an Event | 2 | 1 | 1 | |
| Understand Event Details | 2 | 1 | 1 | |
| Find Event in Long List | 2 | 3 | 1 | |
| Use Sort Function | 2 | 1 | 2 (found when prompted) |
|
| Use Filter Function | 3 (too much clicking) |
3 (didn't notice) |
3 (found when prompted) |
|
| Understand Meaning of Delete Event | 1 | 3 (thought she may be deleting it off of event owner's calendar) |
1 | |
| Understand Meaning of Posted Events | 3 (thought it was a shared calendar) |
1 | 1 | |
| Scenario 2: Add Event & Administer Calendar | ||||
| Find Add Event | 1 | 1 | 1 | |
| Find Required Fields on Add Form | 2 (what is short title?) |
2 | 1 | |
| Understand On-off Campus/Campus location? | 2 | 2 | 1 | |
| Understood Using Calendar Icon to Navigate to Date | Didn't mention | 1 | 1 | |
| Understand Some Sections of the Add Form are Hidden | 2 | 4 (suggested changing font of “Hide”) |
1 | |
| Save Event | 1 | 1 | 1 | |
| Find Posted Event | 1 | 1 | 1 | |
| View Posted Event | 1 | Didn't do | Didn't do | |
| Choose Calendar to Find Formatting Options | 1 | 5 (went through all other options except calendar first) |
2 (thought My Profile or Event Manager might be right place) |
|
| Chose Format Calendar | 1 | 1 | 1 | |
| Understand Color Wheel Concept | 1 | 1 | 1 | |
| Scenario 3: Find Campus Event & Set Up Subscription | ||||
| Find Search Campus Events | 5 | 3 | 2 (tried Posted Events first) |
|
| Understood How to Choose a Search Criteria | 1 (but didn't like) |
1 (liked it) |
1 | |
| Find Subscriptions Section | 1 | 2 (tried Pending Events first) |
1 | |
| Find Create New Subscription | 1 | 1 | 1 | |
| Understand Ability to Multiple Select in Drop Down | 1 | 1 | 1 | |
| Find All Search Criteria | 1 | 1 | 1 | |
| Finding Events “2 weeks ahead or less” Make Sense? | 1 | 1 | 1 | |
| Understand concept of a subscription | 3 | 2 | 2 | |
| Understand Concept of Recommend | 1 (thought was an OK idea) |
1 | 1 | |
| Understand Concept of View vs. Edit events | 2 | 1 | 1 | |
| Likes | ||||
| Intended Audiences | Maps | Event Manager | ||
| Export | Being able to update the CSS | Recommend | ||
| Being able to import images | ||||
| Subscription Expiration Date | ||||
| Export | ||||
| List Format | ||||
| Dislikes | ||||
| Event Type (not needed) | Deleting (confusing) | Search too complicated | ||
| Have to click too many times | Additional Info (confusing) | |||
| Event Participants (not needed) | ||||
Key:
Level of Difficulty
- Very Easy
- Easy
- Average
- Hard
- Very Hard
Critical Areas
[Top]
Discussion
Major Findings
Overall, we were pleased to find that our general concept made sense to our test participants, and that they found the flow of the application was fairly logical. However, as we knew we would probably discover, several of our participants had difficulty with some of the terminology we used to describe the structure and functions of the application. We were also happy to hear that we seemed to be providing about the right level of functionality to calendar owners, although our Add Event form and Event Search function need to be simplified.
Other findings included:
- Application Structure
- Users thought the separation of the "Event Manger" section into "Pending" and "Posted" events was logical, but all of them suggested different labels for the "Posted Events" section.
- Most users thought the "Add form" was too long, and would like a shorter version to use for quick entry of simple events.
- Most users preferred "fewer clicks," including making the Event Search simpler, and reducing the number of confirmation screens. No user thought that there were too few search criteria in the Event Search.
- Most users wanted to "Preview" events before posting them to their calendar, despite the fact that they were given instructions that didn't state that they needed to do this.
- All users understood basic concepts such as multiple select, color wheels, and the concept of having events sent to their queue, for example, "2 weeks ahead or less."
- Terminology
- The names we had given to the different sections of our application "Event Manager," "Calendar," "Search Campus Events," "Subscriptions," "My Profile," and "Event Archive" did not give our users enough information about what was contained in them.
- Users were confused about the difference between the concepts of "Event Sponsor," "Event Owner," "Event Listing Contact" and "Public Event Listing Contact."
- Features
- Users did not find or completely understand the "Filter By" function on in the "Event Manager" section.
- Users liked the new features this system would give them, such as Recommending and Subscribing to Events.
- The Export functionality was important to several of our users, as they create publications from their list of Events.
- Most users wanted to be informed of canceled Events, but some preferred receiving an email as an alternative to using our application.
What We Were Not Able To Test
Due to our time limitations, we decided not to use color in this round of testing, opting instead to test the overall logic and flow of the application. When participants were unable to find certain buttons, such as "Filter By," we were not sure if it was because the button was poorly located, or because they had difficulty seeing contrasts on our colorless paper. It often seemed that the top navigation bar was not conspicuous enough, and that this caused our participants more difficulty than expected in navigating through our application.
We also did not have time to create a complete "Add Event" page, a complete "Format Calendar" page, a "My Profile" page, or a sample of what the block and list view calendars generated by our application would look like. We plan to complete this portion of our application before our next round of testing.
Changes We Plan To Make
Based on our Results and Major Findings above, as well as suggestions given by our participants, we plan to make the following changes:
- Restructure the the top navigation
- Give users a more logical grouping of application features. This will include using different fonts and spacing to group things that are semantically related.
- May want to change "Calendar" section back to "View Calendar" and "Administer Calendar."
- May want to make "Event Archive" part of the "Event Manager."
- Rethink our terminology
- "Posted Events" could be changed to: "Approved Events," "Published Events."
- Clarify or re-name "Search Campus Events."
- Clarify the meaning of Deleting an event from the "Pending Events" section.
- Clarify or find new names for "Event Sponsor," "Event Owner," "Event Listing Contact" and "Public Event Listing Contact."
- Clarify the meaning of "Advanced Date/Time Options."
- Simplify the Add Form
- At a minimum, we will add a "Quick Event Add" form.
- Simplify the Event Search
- We will expose all the "Event Search" criteria instead of having the user choose criteria one at at time.
- We may want to feature a keyword search that can search all Event fields instead of limiting it to "Title" or "Description." This will depend on whether the database could quickly return results using this method.
- Reduce number of confirmation pages
- Either remove all confirmations except "Delete" confirmations, or add a setting that allows the user to turn confirmations off.
- Add "Notes" feature
- Will allow Calendar Administrators to send a note to other Calendar Owners to whom they are recommending Events.
- Reposition, highlight, and possibly rename "Filter" button.
- Reposition "Export" button.
- Separate "Date" and "Time" in all displays of Event data.
- Make sure that Calendar Administrators can distinguish between Events that come in from an on-line "Add" form versus "Drafts" created by their staff.
- Investigate adding email notification functionality for added, changed, and deleted Events.
- Investigate ways to give different levels of access to Calendar Administrators.
- Determine whether we will allow departments to have customized Event Types.