Assignment 8: Pilot Usability Study

I213 User Interface Design and Development
Prof. Marti Hearst

Nate Agrin
Jessica Kline
Andrew McDiarmid
Ken-ichi Ueda

Contents

1. Introduction
2. Methods
3. Results
4. Formal Experimental Design
5. Discussion
6. Work Contribution
7. Appendices

Introduction

iNaturalist is a web application that enables participants to log and share information about their observations in nature. Our pilot usability study focused on iNaturalist's "add observation" interface, by which participants can add observations as annotations on a map. For this study, we were generally concerned with ensuring that our interface minimizes participants' errors, conveys instructions simply and clearly, and permits the editing of a previously added observation easily. To this end, we anticipated possible problems with the existing interface, developed two alternatives, and tested all three: (1) a left-click adding interface, (2) a right-click adding interface, and (3) a modal tool selection adding interface. We developed three task scenarios, the first two of which were repeated on each interface in a Latin square design, and measured the time to complete each task as well as the number of errors preformed during each experiment. From this study, we hoped to gain further insight into the types of questions and data we might gather in a formal usability study, for which we propose two additional experiments. This informal study will also help us judge the overall design of our interface, and provide valuable user feedback on the usage of the system.

Methods

Participants

Participants were selected from team members' contacts, and in general reflected our primary personas (Jenny Takahashi, Adam Thompson): a casual naturalist uploading few observations at a time.

Participant	Gender	Age	Profession	Computer Experience	Nature interests
1	Male	mid-20s	Caregiver for foster children	Intermediate/Advanced	Hiking, taking photos, ocean exploring
2	Female	25	Primary school teaching assistant	Intermediate	Backpacking, hiking, traveling/general exploring
3	Female	mid-20s	Neuroscience lab technician	Intermediate*	Flowers, bugs, reptiles

* Participant three was unfamiliar with some Google Maps functionality, e.g. click-and-drag panning and double-click to recenter. This affects the usefulness of her results for experiment two, which was designed in part to assess susceptibility to error in the left-click interface, where attempting to drag the map might accidentally create observations.

Apparatus

In all the experiments a white Macbook was used along with a typical Apple brand external keyboard and generic external mouse (two button style with a centered scroll wheel). On screen the Firefox web browser was used to display the interface and was maximized to utilize the entire screen. Any changes in interface were performed by the team members and were not part of the tasks performed by the test participants. During the experiment screen capture software was used to record the user's actions and any vocalizations they made about their actions and the interface. The screen capture software was not visible to the user, and an ambient microphone was used, eliminating the need for the user to wear a microphone during the tests.

Experiments

The first and second experiments were designed to test the three separate interfaces. Although it is generally not a good idea to let participants test different interfaces with the same tasks, we gave each participant the same tasks to complete for each interface. We did this because our sample size was so small that there was not much room for grouping or blocking. We did, however vary the interface order in a Latin square design to expose any bias in the ordering and/or learning effects. The third experiment was designed to test the observation edit functionality; this experiment used one interface to test this functionality. The tasks for each experiment read as follows:

Experiment 1: Discoverability

We currently have 3 interactions which each allow for observations to be added to a map. However, we are unsure which method is the most natural to use. The specific issues here are how well our interface communicates the method for adding observations to the map, as well as the amount of errors, frustration, or surprise people experience when using potentially overlapping control paradigms for navigating the map and adding observation (e.g. does double-clicking zoom or will it add an observation?).

Hypothesis: Users will discover how to add observations the fastest using the modal interaction, followed by the left click interaction, and then the right click paradigm. The modal interaction will produce the fewest errors when adding an initial observation, followed by the right click interaction, while the most errors are predicted to be a result of the left click interaction.

Variables:

Independent

• Interaction Type (Modal, Left Click, Right Click)

Dependent

• Time to add a single observation
• Number of errors until addition of a single observation

Task: You recently went to Tilden Park and spotted a Monarch Butterfly. Please use iNaturalist to add the species name to any location in Tilden Park.

Blocking:

Participant	Interaction Type
1	Left Click	Right Click	Modal
2	Right Click	Modal	Left Click
3	Modal	Left Click	Right Click

Experiment 2: Efficiency

The efficiency task tests the speed and number of errors for each of the three interactions described in Experiment 1 in actual use. The expectation is that controls which conflict with both moving the map and adding observations, specifically the left-click interaction, will cause more errors than either the right or modal interactions.

Hypothesis: Single click interactions (left and right clicks) will allow users to enter observations more quickly than a modal interaction.

Variables:

Independent

• Interaction Type (Modal, Left Click, Right Click)

Dependent

• Time to add 6 observations while having to place them in disparate locations
• Number of errors encountered during the addition process

Task: You recently made trips to Point Reyes and Half Moon Bay. Please record your trip observations onto iNaturalist:

1. You went to Point Reyes the morning of Sunday April 22, 2007, and observed a Chantarelle mushroom and a Turkey Vulture. Please add the species name, date/time, and photo to any location in Point Reyes. Also please include the following description to the Chantarelle mushroom observation: "mushrooms are yummy."
2. You went to Half Moon Bay the afternoon of Saturday April 28, 2007. And you observed an Olives Aeolid (a type of sea slug) and an Anna's Hummingbird. Please add the species name, date/time, and photo to any location in Half Moon Bay. Also please include the following description to the Olives Aeolid observation: "sea slugs are very pretty."

Blocking:

Participant	Interaction Type
1	Left Click	Right Click	Modal
2	Right Click	Modal	Left Click
3	Modal	Left Click	Right Click

^{Figure 1: Left-click (original) interface. The directions in the lower left read "Click on the map to add an observation. Drag the marker to change its location."}

^{Figure 2: Right-click interface. The directions in the lower left read "Right-click on the map to add an observation. Drag the marker to change its location."}

^{Figure 3: Modal interface. The directions in the lower left read "Click on the 'Add a Marker' button, then click on the map to add an observation. Drag the marker to change its location." Note the 'Add a Marker' button in the upper right.}

Experiment 3:

This was designed as a qualitative test of using the interface with a relatively large number of observations, and how intuitive it is to both edit an existing observation and move an existing observation. Participants only tested the Left Click interface because editing and moving works the same for all three interaction types.

Hypothesis: Users may experience some difficulty sifting through a large number of observations, but will have no difficulty making edits or moving observations once found.

Task: You recently added a group of 10 observations and have realized that you need to make some edits. Please edit the following two observations:

1. You originally reported finding a Pocket Gopher in Cesar Chavez State Park. However, you realized that you spotted it at the Albany Bulb. Please edit the observation's location.
2. You originally reported a mushroom in Tilden park and named it "Mario Brothers Mushroom". However, after talking to a mushroom-expert friend you've learned that this mushroom is called an "Amanita muscaria". Please edit the species name.

^{Figure 4: Editing an existing list of observations in Experiment 3}

Procedure

We began the pilot usability study by making additional changes to the second interactive prototype. These changes included fixing the calendar interaction and creating two additional interface interactions: adding observation by right clicking on the map and adding an observation through a modal interaction.

Once we finalized these prototype changes we began brainstorming a formal study. We then discussed and chose the most important experiment components to test in our pilot study. (Several ideas not pilot-tested appear in our formal design below.) We also developed the task scenarios, practiced interacting with the interfaces in accordance with the task scenarios, and prepared for other aspects of the evaluation by printing consent forms and task descriptions and setting up the equipment used in our experiment.

Once a participant arrived, the greeter/facilitator, Andrew, introduced him or her to the other group members and explained their roles during the evaluation: Nate as technical liaison and Ken-ichi and Jess as a note takers. The facilitator then explained the purpose of our interface and the usability evaluation. The facilitator also noted that questions during the evaluation should be directed to the facilitator, requested that the participant verbalize any thoughts or actions as he or she interacted with the computer, asked permission to use screen capture and voice recording software, and informed the participant that they were free to leave the study at any time. Lastly, and most importantly, the greeter emphasized that the purpose of the evaluation was not to test the participant, but rather to evaluate the interface and discover both its positive and negative aspects. After the participant signed a consent form and asked additional questions, the facilitator described and displayed the tasks for the participant to complete.

After we completed the experiment we asked the participant the following questions regarding their experience using iNaturalist and its tested interfaces:

• What did you like about the interface?
• What did you not like about the interface?
• How did you feel about changing the location?
• How did you feel about editing an observation?

• Is there any important functionality missing from the interface?
• Did you prefer the left click, right click, or modal interface? Why?
• Do you think you would actually use a system like iNaturalist? Why or why not?
• Do you have any questions?

We concluded by thanking the participant for both their evaluation of the prototype as well as their time. After the final experiment, Ken-ichi and Nate ate at T-Rex Barbecue (http://www.t-rex-bbq.com/) with our third participant. Highlights included the ribs, macaroni and cheese, and brisket. (Andrew and Jess are both jealous.)

Results

Experiment 1

Left-click interface

	Participant 1	Participant 2	Participant 3
Time	1:30	0:32	3:22
Total errors	0	1	0

Participant 1 spent time looking for the instructions, and also spent time exploring the interface. Participant 2 knew where the instructions were from exposure to a previous interface, but still spent time reading them.

Right-click interface

	Participant 1	Participant 2	Participant 3
Time	0:30	1:05	0:27
Total errors	0	6	0

Participant 2's errors were all left clicks in attempts to add an observation.

Modal interface

	Participant 1	Participant 2	Participant 3
Time	0:32	0:50	0:55
Total errors	2	1	1

All participants made at least one error left-clicking to add observations.

General prototype

All participants made use of the browsers built-in form field auto-completion, and while none of them seemed to find the instructions for adding an observation immediately, they all found them eventually.

Experiment 2

The following includes the quantitative and qualitative results of this experiment.

Left-click interface

	Participant 1	Participant 2	Participant 3
Time	4:18	3:12	3:22
Total errors	0	1	2

The errors specific to this interface include accidentally adding an observation when panning the interface.

Right-click interface

	Participant 1	Participant 2	Participant 3
Time	3:04	5:54	4:29
Total errors	1	2	2

The errors specific to this interface include initially left-clicking to add an observation.

Modal interface

	Participant 1	Participant 2	Participant 3
Time	2:54	4:23	5:46
Total errors	0	1	2

The errors specific to this interface include automatically left-clicking to add an observation.

General prototype

The errors specific to the general prototype include pressing the upload button before a photo is chosen. In this experiment we also observed our participants' limited use of the calendar widget, regular use of the auto-complete functionality, and preference for navigating the map through the displayed arrows (rather than the click-and-drag functionality). We also noticed that our participants often looked for feedback after initially clicking on a text field and regularly clicked on the date field in order to change its color from grey to black.

Experiment 3

Participants generally found searching through several observations and making edits fairly simple. Participants 2 and 3 both used the sidebar only to find the identifying letter for the observation they needed to edit, and then hunted for the corresponding marker, while participant one immediately clicked on the sidebar to open the callout. All participants clicked on the sidebar to open the callout for the second edit. All participants dragged the marker to move the observation without hesitation, and all described the experience as simple and obvious.

Follow-up Interviews

What did you like about the interface? Participants described the overall interface as "simple to use" and with a "really simple, clean setup". One participant specified that she liked the location of the observation side bar and the ability to open an observation callout in both this sidebar and through the map marker. This participant also noted that she liked the drag and drop functionality, which allowed her to be more precise about specifying a location.

What did you not like about the interface? Participants did not like the location of the instructions because they were not prominently displayed. One participant complained that the interface provided too slow of an interaction.

How did you feel about changing the location? Participants found it easy to change the location of an observation, one participant appreciated that the instructions included how to change a location.

How did you feel about editing an observation? Participants found it easy to edit an observation (as well as delete observations).

Is there any important functionality missing from the interface? Participants noted that they would like to see the observations of others as well as search for named-places. They also suggested moving the interface instructions to a location above the map. One participant suggested including additional information such as weather conditions and another participant suggested providing a way to look up both scientific and common names.

Did you prefer the left click, right click, or modal interface? Why? Two participants preferred the left-click interface (they are both Mac users that don't regularly right-click) and the other participant preferred the right click interface. No participant especially liked the modal interface because it required additional steps and the mode setting was located in an awkward location.

How would you describe your experience using iNaturalist? One participant noted that she found experience easier than expected. She explained that she is often intimidated by new web applications and really enjoyed her experience.

Do you think you would actually use a system like iNaturalist? Why or why not? One participant noted that he would definitely use iNaturalist and predicted that it would be appealing to anyone with a passing interest in nature. The two other participants explained that they especially enjoyed the photo component of the interface, however both were hesitant to speculate becoming actual users of the system. (However, one hesitant participant noted that she would probably use iNaturalist as a teaching tool when she has children and also suggested the site as an alternative to traditional travel guidebooks).

Do you have any questions? (and other notes) Participants generally preferred completing the callout before clicking "Done". And one participant explained that she was at first hesitant to add an observation, but once she completed one observation it an easy and intuitive experience.

Formal Experimental Design

Our informal usability tests allow us to consider the design of formal tests to be carried out with more participants. Each experiment was amended slightly and two other potential experiments were derived from our analysis and initial brainstorming. For each experiment we would attempt to solicit the participation of approximately 30 individuals in order to capture a more significant amount of experimental data.

Experiment 1

Three groups would be created, and each would test only one of the interaction paradigms (left-click, right-click, modal) in order to determine the speed at which an interaction might be learned. Based on our informal analysis we would predict that the left-click interaction would be the easiest to learn, as it appeared to be the participants' default guess. Right-click and modal interfaces would probably require about the same amount of time to learn because they both required that participants find and read the provided on-screen instructions. The results of this experiment might then inform our decision to test the location of the instructions in order to determine if a particular placement of the instructions was more obvious.

Hypothesis: The left-click interaction will be the fastest to learn and generate the fewest errors, while the right-click and modal interfaces will require about the same amount of time to learn and generate equivalent numbers of errors.

Variables:

Independent

• Interaction Type (Modal, Left Click, Right Click)

Dependent

• Time to add a single observation
• Number of errors until addition of a single observation

Experiment 2

Three groups would be created and each test one interaction paradigm (either left-click, right-click, or modal) while completing a similar task to that in the informal Experiment 2. However, unlike the informal Experiment 2, we would train the participants briefly on using a typical Google Maps interface, specifically introducing participants to Google's click and drag panning interaction used to move the map, eliminate the image upload portion of the test, and slightly increase the number of observations to add to six, where the participant would add two observations in three locations. The purpose of this test would be to determine if one of the three interactions (left-click, right-click, modal) was easier to use and produced fewer errors when adding observations in disparate locations on the map. Our predictions based on the informal test suggest that the left-click interaction would be error-prone, as the left-click also provides significant movement interactions in a typical Google map.

Hypothesis: The left-click interaction will produce the fewest errors, but the right-click interaction will be the fastest to use.

Variables:

Independent

• Interaction Type (Modal, Left Click, Right Click)

Dependent

• Time to add all observations
• Number of errors until completion of the last observation

Experiment 3

All participants would be tested on the same tasks as in the informal Experiment 3 with the following changes: more observations would appear to have been added, requiring the participant to hunt or search more thoroughly through their list, and each user would preform five edit operations instead of two. Based on our informal experiments, we predict that most participants will find editing easy, and that more errors will be recorded simply because more edit operations will be preformed during the experiment.

Experiment 4

We are curious if participants might change their behavior with the map after becoming familiar with the interface. We predict that when adding only a few observations, participants would probably complete each observation one at a time. When adding more observations, however, we predict that participants would likely opt to add many observations on the map by clicking multiple times, and then going back and filling in the form data for each observation. Here again we will divide the participants into three groups. Each group will be asked to add either five, ten or twenty observations after having plenty of time to explore the interface and become familiar with its interaction. In this experiment we will not vary the click interaction, and will constrain the interface to the 'left-click' interaction for all groups. During the experiment we will record the behavior of each participant and the total errors and time taken to preform each task.

Hypothesis: As the number of observations to add increases, users will fill out fewer form fields per new observation.

Variables:

Independent

• Number of observations to add (5, 10, 20)

Dependent

• Form fields completed before clicking "Done" for a new observation (entering dummy values as placeholders should count as 0 fields completed)

Experiment 5

Experiment 2 of our informal analysis required a significant amount of time to complete and suggests that the map-centric interface may prove to be overbearing when adding many observations in one sitting. In our final formal experiment we would test the number of observations participants add before they feel that the interface is obtrusive and better adding interactions might exist. While highly subjective, this test would aim to elicit qualitative responses from the participants about what is slowing them down or how the interface might encourage them to add many observations (such as displaying their friends' nearby observations, etc.). We would predict that participants would be able to add approximately ten observations in one sitting before feeling that a more streamlined interface would be useful.

Hypothesis: Most participants will complete at least 10 observations before expressing fatigue.

Variables:

Independent

• Interaction Type (Modal, Left Click, Right Click)

Dependent

• Number of observations added before vocalizing fatigue

Discussion

Lessons Learned

Although the quantitative data we collected are not useful due to issues with our experimental design and the low number of replicates (see Formal Design below), the qualitative results were informative. In general, all participants described the system as simple and easy to use, with caveats including speed of response and lack of pre-existing data. All participants could easily imagine situations in which they might use iNaturalist, but all of them described their use strictly in terms data consumption, not contribution.

We derived four main changes we need to make to our interface based on this study:

1. Move the Instructions - All participants spent time searching for instructions when the expected left click behavior did not work, and most specifically suggested that they be moved to a more prominent location. Our initial designs actually had the instructions at the top of the map, but we moved them because we thought they would clutter the interface with information that would only be read once, if ever. We have decided to move the instructions back to the top, but to place them in a bar that can be closed (or minimized) to reduce clutter.

2. Primary Button Emphasis - Users tended to hover over Cancel/Done button pairs, both in the observation callout and in the photo upload lightbox. We believe this is due to lack of distinction between the two, and intend to resolve this by coloring OK (or Upload) differently, or by changing the Cancel button to a link, as suggested in our last review. Additionally, the photo upload lightbox has a gap below the buttons, and both Cancel and Upload are too close to the Browse button, which we believed caused hesitation. We also intend to fix these problems.

3. Default Date Color - We designed the date selection box to inherit the date of the previous observation, but the inherited date was the same grey color as the tip text, causing users to click the date box so they could see it turn black and know the date was set. We will change this so the inherited date is always black.

4. Feedback for Selecting Form Fields - Some participants clicked within text fields in the observation form and then hesitated, as if waiting for something to happen. We believe this indicates a need for more feedback from text fields in the form, and intend to amend the problem by altering the color of text fields when they are clicked on.

One thing we were not able to learn was whether one of the three observation adding interactions was superior. This was due to the limitations of the pilot study, and we believe the more rigorous formal experiment described above might settle the issue.

Formal Design Rationale

Our informal experiments suggest that we changed several aspects of our experiments for our formal experiment design. Experiment one was designed to determine the speed at which participants would be able to learn how to interact with the map. We noticed that the left-click interaction was the quickest learned, but that once participants discovered the instructions in the bottom left of the interface they were able to determine the other click interactions quickly. Our formal test would take into account this learning behavior by only exposing each participant to the interface once during the experiment.

Our second experiment is the most changed of the original three. The modifications we have proposed, including teaching participants about the map, demonstrating the click and drag panning of the map, removing the image upload task, and adding more observations in different locations is designed to test each of the three click interactions (left-click, right-click, modal) for errors, time to completion of tasks and qualitative analysis specifically when the click interactions might interfere with the participant's expectations on how to navigate the map. We decided that teaching participants about the map would eliminate an issue we experienced where some of our participants were not previously familiar with all of the features of Google's maps. The image upload task was deemed to be irrelevant to the task of testing adding observations while interacting with the map, and extended the time it took to complete the task unnecessarily. Finally, adding more observations in more locations would provide more experimental data and potential for errors to study each click interaction while navigating the map.

Editing the observations on the map appeared to be a simple task for each user in our informal study, however we recognized that on our informal edit page, we only presented the participant with eight observations to choose from. In order to test the interface in a real world setting, we would add many more observations, possibly up to twenty or thirty, from which the participant would have to find and correct the right observation. This would provide us with insight into how the interface could be improved to facilitate editing or if the interface was sufficient for editting in its current configuration.

Experiments 4 and 5 were not originally tested, however they were developed based on the observations we made in our informal test. Experiment 4 was developed because we noticed that as users learned how to interact with the interface, they often added incomplete observations and noted they could return to edit them later. This did not immediately appear to distress them and suggested that participants, when adding more observations, might find faster methods of interacting with the map than filling out each observation one at a time. Experiment 5 was also developed after the informal testing of Experiment 2, where we noticed that adding four observations required a significant amount of time. We are looking for a simple method of testing the limits of the interface we've designed for the casual user. The results from Experiment 5 might inform the design of a power user's data entry form that is not necessarily map oriented.

Work Contribution

Task	Andrew	Jess	Ken-ichi	Nate
Recruitment/scheduling	5%	30%	60%	5%
Experiment design	25%	25%	25%	25%
Pilot study execution	25%	25%	25%	25%
--Roles	Facilitator	Observer	Observer	Technical facilitator
Discussion/Write-up	25%	25%	25%	25%

Appendices

Appendix 1: Experiment 1 Data

iNaturalist.org Pilot Usability Study
Experiment 1: Discoverability	Participant 1	Left Click			Right Click			Modal
		Start Time	2:40:45		Start Time	2:41:35		Start Time	2:42:22
		End Time	2:41:15		End Time	2:42:05		End Time	2:42:54
		Total Time	0:01:30		Total Time	0:00:30		Total Time	0:00:32
		Errors	None		Errors	None		Errors	Clicked tool twice before noticing it was active
		Notes	Spent time looking for instructions		Notes	Checkout out calendar but didn't use it		Notes	Didn't use calendar
			Also spent time exploring the system			Curious about calendar (later commented that rarely uses the calander feature but likes it there)			"Said this interface was ""unecessarily complicated"""
									Had trouble finding the tool
									Carefully read the instructions
	Participant 2	Right Click			Modal			Left Click
		Start Time	3:54:17		Start Time	3:55:30		Start Time	3:56:38
		End Time	3:55:22		End Time	3:56:20		End Time	3:57:10
		Total Time	0:01:05		Total Time	0:00:50		Total Time	0:00:32
		Errors	L-clicked to add 5 times		Errors	Clicked w/o setting the mode once		Errors	Accidentally added an observation once while dragging to pan
			Clicked title of sidebar when trying to add
		Notes			Notes	Zoomed in before adding an observation		Notes	Read the instructions
						Did not use autocomplete			Used pan buttons even though she was instructed that she could drag to pan
						Quickly found instructions (learned activity from previous interface)			Quickly found instructions (learned activity from previous interface)
	Participant 3e	Modal			Left Click			Right Click
		Start Time	5:32:01		Start Time	5:33:05		Start Time	5:33:32
		End Time	5:32:56		End Time	5:33:25		End Time	5:33:59
		Total Time	0:00:55		Total Time	0:00:20		Total Time	0:00:27
		Errors			Errors			Errors
			Clicked on map before selecting map marker mode						Autocomplete did not work
		Notes			Notes			Notes
			"Saw ""Add Map Marker"" button before instructions"
			Used autocomplete

Appendix 2: Experiment 2 Data

iNaturalist.org Pilot Usability Study
Experiment 2: Efficiency	Participant 1	Left Click		Right Click		Modal
		Start Time	2:46:02	Start Time	2:50:34	Start Time	2:53:51
		End Time	2:50:20	End Time	2:53:40	End Time	2:56:45
		Total Time	0:04:18	Total Time	0:03:04	Total Time	0:02:54
		Errors	Zoomed out too far	Errors	Left clicked on to add an observation	Errors	None
		Notes	Used pan buttons	Notes	Did not like reopening a callout	Notes
			Didn't use calendar		Used abbreviated names
			Moved an observations w/o instruction		"Seemed to hesitate after clicking in txt fieldsm possibly due to lack of feedback"
			Navigated with zoom and pan		"Started abbreviating species names: sign of tedium?"
			Bug occured when using the photo upload
			Used autocomplete
	Participant 2	Right Click		Modal		Left Click
		Start Time	4:00:49	Start Time	4:06:57	Start Time	4:11:38
		End Time	4:06:45	End Time	4:11:20	End Time	4:14:50
		Total Time	0:05:54	Total Time	0:04:23	Total Time	0:03:12
		Errors	Left clicked first, but only that one time"	Errors	Hit upload button before selecting a photo	Errors	Missed hitting the Upload button
			Didn't select a photo before adding an observation
		Notes	Confused about whether each observation gets a seperate callout	Notes	Waited for feedback when entering info in a text box	Notes	Used calendar
			Confused about species name label (should it be a scientific name)		Discovered and liked autocomplete		Used GMap pan buttons
			Seemed to want the greyed date to be active		Used the calendar		"Clicked greyed pre-set date to turn it black, thinking this set it"
			Didn't use the calendar		Zoomed and panned to navigate
			After forgetting to add a comment for the sea slug, immediately clicked the obs in the sidebar to edit	Muttered, "Oh, upload," before hitting the Upload button
	Participant 3	Modal		Left Click		Right Click
		Start Time	5:35:49	Start Time	5:41:50	Start Time	5:45:25
		End Time	5:41:35	End Time	5:45:12	End Time	5:49:59
		Total Time	0:05:46	Total Time	0:03:22	Total Time	0:04:29
		Errors		Errors		Errors
			Clicked on map before selecting map marker mode		Clicked upload before photo chosen		Left-clicked before right-clicked (X2)
			Double-clicked to add an obs		Accidentally added an obs while panning
		Notes		Notes	Clicked greyed date to set	Notes	Used pan buttons and dragging to navigate
			Had no trouble editing an uncomplete observation		Used pan buttons		Spent time exploring
			Does not use calendar		Didn't use calendar		"Became concerned about accuracy, began zooming in closer to add obs"
			At first expected actual time field (instead of time generalization)				Likes the sidebar
			Some difficulty using zoom and pan map buttons

Appendix 3: Experiment 3 Data

Participant 1

• did not hesitate to click on the sidebar and expected it would open the callout
• found dragging markers to move observations intuitive
  • recognizing "X" as the actual location of the marker not immediately obvious, but learned quickly
• closed the callout before the dragging the marker
• recommended showing usernames in the sidebar

Participant 2

• zoomed in before making any edits
• hovered over the sidebar when searching for the observation to edit but did not click on it when found.
  • Said, "OK, it's 'H'"
  • Proceded to search the map for the corresponding marker
• dragged marker to move it immediately
• clicked the sidebar for the second edit
• said it was easy to edit observations and drag them to move them around the map
  • thought the task would be more complex but found it very easy

Participant 3

• first found the observation on the sidebar, noted it's identifying letter
  • wondered alowed if she could click on it, then did so
• clicked on the sidebar immediately for the second edit

Appendix 4: Follow-up Interview Data

Participant 1

What did you like about the interface?

• Described the interface as a "really simple, clean setup".

What did you not like about the interface?

• Complained the interface was somewhat slow.
• Generally doesn't use calendar widgets for date selection on any web site other than flight booking sites.

How did you feel about changing the location?

• Found it easy to change the location of an observation.

How did you feel about editing an observation?

• Found it easy to edit an observation.

Is there any important functionality missing from the interface?

• Suggested moving the instructions above the map. Thought the bottom was a place for copyright or other ancillary information. Found instructions quickly regardless.

Did you prefer the left click, right click, or modal interface? Why?

• Liked the Left Click interface best.
• Thought the Modal interface was the worst, because added an unecessary extra step to the process of addding an observation.

How would you describe your experience using iNaturalist?

• Liked the delete link.

Do you think you would actually use a system like iNaturalist? Why or why not?

• Would actually use the site.
• Would certainly use it to look at the observations of others. Would like a discussion forum. Thought it would be appealing to anyone with a passing interest in nature.

Do you have any questions? (and other notes)

• Grew up using a Macintosh.
• Often moused over the Done button after having completed an observation, then moused out for a few seconds before returning and pressing Done. Said he had no qualms about pressing done, just checking completeness.
• We performed a very brief, ad-hoc test in which we asked him to add an observation, and then pan the map by dragging, as opposed to using the pan buttons as he had done during the test. He immediately added an observation accidentally when trying to pan, then proceded to drag cautiously.

Participant 2

What did you like about the interface?

• describe the interface as "simple to use"

What did you not like about the interface?

• instructions for adding an observations were hard to notice at the bottom of the map, once found, adding was easy

How did you feel about changing the location?

• Saw this functionality in the instructions

Is there any important functionality missing from the interface?

• would like some way to make identifications and look up species names
  • both common and scientific names

Did you prefer the left click, right click, or modal interface? Why?

• Preferred the right click interface and did not like the modal interface (required extra steps)

How would you describe your experience using iNaturalist?

• Found experience easier than expected (often is intimidated by new web applications)

Do you think you would actually use a system like iNaturalist? Why or why not?

• no
• but might use iNaturalist if she had children
• Would be helpful as a teaching tool (for kids, students, etc). Also would be a more interesting/fun alternative to traditional guidebooks when traveling (more pictures)

Do you have any questions? (and other notes)

• The first time she added an observation she was a bit apprehensive, but once she did it once, it was very easy
• Wanted to add all information to a callout before clicking "Done"

Participant 3

What did you like about the interface?

• using the sidebar to open upo callouts
• draging and dropping markers to move them (allowed participant to be more precise about location)

What did you not like about the interface?

• instructions not noticable at first

Is there any important functionality missing from the interface?

• wants to see the observations of others
• wants to be able to search for locations
• would like to see information such as weather conditions

Did you prefer the left click, right click, or modal interface? Why?

• Left Click
  • click to add is intuitive
  • as a Mac user, right-clicking is not obvious
  • Modal tool is noticeable, but awkward

Do you think you would actually use a system like iNaturalist? Why or why not?

• would not us iNaturalist without photos