Participants
We selected three participants, two of whom were totally unfamiliar with our application. The third was one of our low-fi prototype test participants. We did this in order to get completely fresh reactions to our application, and yet also benefit from the feedback of someone who had been part of the original concept development process. All of the participants are males between the ages of 26 and 34; two are master’s students at the School of Information Management and Systems at UC Berkeley who have an interest in business, while the third (the low-fi participant) is a graduating MBA student at UC Berkeley with particular interest and expertise in the wireless telecommunications industry in China.

Apparatus
All tests occurred at the School of Information Management and Systems. The first test was conducted in a quiet classroom after school hours, while the others took place mid-afternoon in the conference nook of a computer lab used by many SIMS students. We used a lap-top computer with wireless connectivity to run the tests because our relationship visualization uses Flash 7, which is not loaded on the school’s computers. We started with a Mac, but ran into problems with links, for some reason; we then switched to a PC and had no further problems.

Tasks
Because the scope of our project had changed significantly since the low-fi prototype was tested, we created new task scenarios for our pilot usability study. These scenarios, and what we were looking for during their testing, are as follows:

-- Task Scenario 1
You are interested in job or internship opportunities in the Chinese wireless industry but aren’t sure if you want to work in China or just work for a company that has close ties with Chinese companies. Your task is find out where some of the major telecom firms are located in China (China Mobile, China Unicom, China Netcom, and/or China Telecom) and which non-Chinese firms have connections with them.
Functionality Tested. For this task, we first looked to see if users could navigate easily to the company overview pages to find company locations. We also wanted to get user feedback on the general content of these pages (overview, statistical information). We then wanted to introduce users to the relationships visualization and see how quickly they could understand what the tool could do and how well they could manipulate the various controls.

-- Task Scenario 2
You have just scheduled an interview with China Telecom. As part of your interview preparation, you want to make sure you know as much as possible about the company’s products and competitors. How would you go about finding this information on the MappingChina site?
Functionality Tested. This task tested how easily a user could produce a specific type of relationship visualization (competitors) and, in particular, product/service information using the visualization tool. Because the resource we used to collect our data lists a company’s specifically-named products and the competitors for those products, rather than the general types of products the two companies offer and compete with, we linked products to the competitive relationship. However, we were concerned that this might be too indirect a way to show product information.

-- Task Scenario 3
You are writing a paper on the wireless industry in China and want to find out what the various sectors are and how they compare to each other in terms of market cap and number of companies. You also want to see who the top players are in each sector and how they relate to companies in other sectors in terms of contracts and alliances.
Functionality Tested. This tested whether users could navigate to the sector pages to find the market breakdowns and top players. It further probed the users’ understanding of how to use the relationship visualization, and whether they would use the sector pages or the visualization to find market caps.

Relationship Visualization Tasks
In addition to the three task scenarios, we had participants work through 13 short test questions that explored the range of options offered by the visualization. For this, we wanted to see how quickly and easily users could find certain information, and whether signposts to that information were adequately clear.

Procedure
Three Mapping China team members were present at two of the tests, and two team members were present at the third test. Team members functioned as facilitator, observer/note-taker, and observer/time-keeper. (For the third test, the note-taker was also the time-keeper.) We wrote a script that the facilitator read to each participant. After reading the introduction and before proceeding to the usability test, the facilitator had participants read and sign a consent form. We did not audio- or video-record the tests. The facilitator then stepped the participants through each scenario and visualization task, and the observer recorded participant actions, navigational “errors,” and other responses. The time-keeper kept track of how long it took participants to work through each task. After completion of all tasks, we had participants verbally answer a brief questionnaire about the features of the site, its navigation, and what they found most problematic and what they most liked.

The top three issues uncovered by our pilot usability test dealt with home page content, overall site navigation, semantics, and usage issues with the relationships visualization.

Our home page still needs a lot of work. We have a drop-down box listing all industries, but the “Featured Companies” list shows only telecommunications firms. One participant was confused by the image of the relationships map. Two others clicked on the circular, map-diagram icons next to the MappingChina banner thinking they were navigational buttons. One user clicked on the red bulleted text thinking it was also a navigational link.

We are still not achieving optimal navigational flow. Users can’t get back to a company page or the home page from the relationship visualization page. There a multiple ways to navigate the site, but not all directions produce the same results. The “View Companies By Industry” link takes the user to a list of companies arranged by industry and sector; however, the left sidebar industry drop-down takes the user to an industry overview page with sectors listed along the left sidebar. We need to think more whether this could be problematic. Other participants did not understand what the left sidebar “sectors” box did, thinking it might be a site-wide navigational menu. Users could not really see how to access sector information.

Indeed, none of them really understood what “sectors” even meant. They all interpreted it as being synonymous with “industries,” rather than being subcategories of particular industries.

Our visualization could use a lot of tweaking. Users didn’t understand which views they had available or how they got those views. There were many minor issues as well: sometimes the user had to click multiple times on a company circle to select it; the visualization in general was considered “busy”; the actions that the various buttons achieved were not immediately apparent; and users didn’t always realize that the visualization centered on one company and that they could switch the focus to another company.

We learned a number of things from the pilot usability study:

1. There are differences among users and their computer abilities and aptitudes. More advanced users tended to figure things out on their own; lower-level users became frustrated more quickly with some of the tasks.

2. We need to make the site more clear, both in terms of navigational flow and content description. The visualization might be simplified, with different views clearly described and features parceled out among the different types of views, rather than having a lot of data shown at one time.

3. We discovered that timing the task scenarios didn’t give us any usable metrics because the users all progressed through the site at their own pace. Timing reflected the user’s personality more than the functionality of our site. The visualization test timing was a little more successful, as users had already worked through the task scenarios and so were more familiar with the site’s functionality; but task times still reflected the willingness of the user to complete the tasks. One user abandoned tasks more quickly than the others.

4. All of our users felt that the site’s content, other than the visualization, was too generic to be truly useful. They wanted more in-depth analysis. But all could see the value of the relationship visualization feature.

5. Many errors were caused by bugs in the coding. For future testing, the prototype should be as highly functional as possible.

Problem
Based on the experiences of the participants of our pilot usability test, we would like to experiment further with different design presentations of the relationship visualization. Currently, the diagram shows, on one screen, the relationship graphics (circles representing companies and industries, with lines between them indicating the relationships, and text labels showing the names of companies and sectors), selector boxes for viewing different types of relationships and relationship paradigms (all companies/sectors or only companies with relationships), a symbol legend, and text boxes listing relationship details such as product names and descriptions, dates and nature of strategic alliances, etc.

In short, the visualization presents a good deal of information in one screen area. This posed somewhat of a problem for our pilot usability test participants. All commented on the substantial amount of data being presented and expressed difficulty figuring out how to fully manipulate and understand it. While this could be because all of the users were new to the visualization who might become more comfortable with it over time, we still pondered how we could minimally change the design to make it less overwhelming, for both new as well as seasoned users.

Solution
We came up with a possible solution that would separate the graphical data from the text data, which we would move to a table at the bottom of the screen. This would have several benefits:

• It would allow users to focus on the company interactions;
• It would provide a separate area for the text data, which our users didn’t fully notice because manipulating the company circles took all their attention; and
• It would allow more room for the visualization, which currently crowds its allotted space.

However, it would also have some drawbacks:

• Users would have to scroll down to see text detail, whereas currently this information is right next to the visualization;
• If a company has many relationships, the table could become very long; as currently designed, details for only one relationship show up at one time; and
• The purity of the visualization would be compromised.

Hypothesis
We believe that separating the graphical and the text data will simplify the visualization and help users accomplish tasks more quickly and efficiently.

Experiment Design
Our experiment would test whether the overall benefits of separating the text from the graphical data outweigh the drawbacks. We would select a user group, have them perform a specified set of tasks on both of the UI designs, measure their completion times and numbers of errors, and then compare the two performance sets. Based on this, we would assess our hypothesis and make any required design changes.

User Participants
A test pool of 12 users would be adequate for our purposes. Because we noticed some variability among our pilot usability study participants according to computer proficiency level, we would group users according to two skill levels – one more basic, one more advanced. Although our experiment falls naturally into left brain/right brain testing conditions, we would not control for this at this time.

Tasks
We would have users perform the same visualization tasks as our pilot usability study participants. (See our Script and Questionnaire document, in the Appendix.) These are 13 tasks that ask users to find a certain piece of data.

Task Order
We would group the users by skill level. 3 users in each group would start with the table design and 3 with the current (sidebar) design. We would then switch the designs and have users perform the tasks again.

Metrics
We would track task completion times and error rates for each task, then calculate averages for each design.

Evaluation of Results
If the completion times and error rates for the table design outperform the sidebar design, we would have proven our hypothesis. However, factors that would contribute to the uncertainty of our conclusion would be the small number of users in our experiment, the varying levels of skills of those users, and any differences due to left brain/right brain strengths. This last factor could be a considerable factor: the table design might work better for left-brain users, while the sidebar design might be preferable for right-brain users.