|
|
|
Assignment
4: Low-fi Prototying & Usability
Testing
- Introduction
- The System
- Purpose and Rationale
of Experiment
- Prototypes
- Description of Prototypes
- Method
- Test
Measures
- What we measured or
looked for and why.
- Results
- Discussion
- What we learned from
the low-fi evaluation
- what we'll change
in our interface from these results
- what the evaluation
could not tell us.
- Appendices
- Materials (script,
task instructions, forms)
- Raw data (critical
incident logs)
- Work
Distribution Table
Introduction
|
| The
System |
|
|
MappingChina is a
web-based tool that we are developing to help business-savvy
students and professionals conduct high-level market
research about China. One feature of our tool is
helping people do a quick competitive or "five
forces" analysis of an industry in China. While
our prototype focuses on a single industry -- that
of wireless telecommunications in China -- our intention
is to create a scaleable infrastructure design that
can feasibly expand to other industries and geographies,
if our team were given more time and resources.
|
| Purpose
and Rationale of Experiment |
|
|
The purpose and rationale
of our lo-fi prototyping and usability testing was
to present our initial design concepts and task
flow ideas to participants who are part of our target
user group and gain insight into the types of competitive
data required by them, the optimal page location
for such information, and the types of components
that would best support the competitive overview.
Feedback from the testing process will determine
the major design direction for our project, in terms
of general layout and navigation. We intentionally
selected a new set of three participants for this
phase in order to get new perspectives on the tool.
|
|
Description of Prototypes
We
refined the scope of our project following instructor
feedback and similarly rewrote our user
scenarios. Picking up on a common goal of
our users, we chose to focus on providing a quick competitive
overview of the wireless telecommunications industry
in China. Such analysis could not only supply summary
information about an industry or sector but also identify
supplier, customer, partner, and competitive relationships.
There currently appears to be no simple, online way
to accomplish this. Current alternatives include loosely
structured Internet searches (typically using Google)
and phoning industry analyst friends (as described by
Stella, one of our sample users). We also presented
mapping features, news feeds, and general city information.
(See our System
Flow Diagram for a basic visualization of
our site's components.)
We
devised two major design concepts, one resembling
a portal (Prototype
A) and one centered on a search function
(Prototype
B).
Prototype
A
The
portal prototype depicts five components: industry
competitive analysis using a five forces diagram;
industry concentration map; general city information;
top news story links; and search function.
Industry
Competitive Analysis. The
user selects an industry to analyze by clicking a radio
button. This brings up an Industry page that gives a
brief overview of the industry and presents a clickable
value chain for that industry. The user selects the
value chain segment he or she wishes to analyze, which
brings up a sector overview and five forces diagram.
This feature is also clickable, allowing users to get
specific information on industry competitors, potential
entrants, buyers, suppliers, and substitutes. By clicking
on a particular five forces box, users will get a Five
Forces Analysis page with specific information related
to that force. The sidebar for both Industry and Five
Forces Analysis pages contains the search function and
respective news story links.
Industry
Concentration Map. This
feature shows the geographic concentrations of the various
industries in China. There is a slider bar that allows
users to view changes over time.
General
City Information. Our
intention was to present general statistics on major
cities, such as Shanghai, Beijing, and Guangzhou. By
clicking on a city, users will access a page that gives
a profile of the selected city; its area, population,
and GDP; major industries and companies; and city map.
Top
News Stories. This
sidebar feature shows users the top headlines for the
high-tech industry overall.
Search
Function. This
allows users to query the site using company name, industry,
city, or news keywords.
Prototype
B
Prototype
B presents the same information as Prototype A using
a more advanced search function as the entry point.
From the home page, users can enter search parameters,
such as part of all of the company name, or keywords.
They can select the News checkbox to find only news
articles. The industry radio button on our prototype
was set to the wireless telecommunications industry
since this is the focus of our project. After entering
search parameters, users click the Search button.
If
the News checkbox was not selected and the user
had entered data in the Company Name field, the
resulting window would show the list of companies
closest matching the input supplied by the user.
The matching records also provided a one-line summary
of the company, such as name, city, and industry
sector. Clicking a company name on the list would
open the company summary page.
If
the News checkbox was selected, the resulting window
would show the list of news stories, preceded by
a short list of frequently-occurring terms ("hot
topics") in the news stories, and top headlines.
If
only the Industry was selected, the resulting window
would give an overview of the industry, including
value chain, top players, five forces analysis,
history of the industry in China, and recent industry
news headlines linking to full stories. Clicking
on top player names would lead to that company's
overview page.
Industry
Sector pages are similar to Industry pages, with
their own value chain, but including a reference
to the industry value chain to show what position
the sector has in the overarching business. Industry
Sector pages would also include a list of top players,
five forces analysis, history of the industry sector
in China, and recent news headlines.
Company
overview pages include business summaries and contact
information similar to those of company profiles
in Yahoo!Finance.
|
|
Method
|
| Participants
|
|
Participant
1 is a male second-year MBA student at
the Haas School of Business at UC Berkeley who has
an undergraduate degree in electrical engineering
and computer science. He was born in China and raised
in Hong Kong. He has several years of experience
in enterprise software and Internet startups in
Silicon Valley and extensive experience in managing
business analytic systems. At Haas, his focus is
on global technology management. He looks forward
to pursuing a career in the software or telecommunications
industries tying together China and Silicon Valley.
(He speaks Cantonese, Mandarin, and English.)
Participant 2 is a male Ph.D. student
in the School of Information Management and Systems
at UC Berkeley. He has extensive training in information
systems design as well as business. He is not actively
looking to find information about China's high-tech
industry but is interested in learning more about
that market for general knowledge purposes. His
interface design expertise gave us much-needed guidance
in selecting from various design options.
Participant 3 is an international
male graduate student looking to work for a multinational's
China office this summer. He currently is a visiting
scholar at the School of Information Management
and Systems at UC Berkeley and his focus is on the
economics of information. He hopes to explore opportunities
in China's high-tech market by researching how multinationals
are performing in China. He laments the amount of
work it takes to gather news and research and is
looking forward to using a tool like MappingChina
to find industry overviews and company profiles.
|
| Task
Scenarios |
|
| For
each prototype, we asked our participants to go
through the tasks required by four different scenarios.
Prototype
A
Scenario 1. Find out the
companies and product announcements that made the
headlines relevant to China at the 3GSM conference
last week in Cannes.
This
scenario looked to see how people would use the
Search and News features of our site.
Scenario
2. ChinaSoft
is a company coming to hire at Haas in a few weeks.
Find out the kind of information you would normally
seek out in deciding whether to apply for the job.
Please walk us through the procedure you would expect
with this interface, and tell us what information
you are specifically looking for.
Here,
we were looking for two things: how people would
find our site's company information, and what information
people look for when searching for a job.
Scenario
3. You've heard about the WAPI
standard in China. What is it and what companies
are actively working on WAPI solutions in China?
This
scenario tested how a user would find both company
and industry information.
Scenario
4. Thinking about alternative
business opportunities in China, you are considering
taking the entrepreneurial route, taking WiFi to
the Chinese people. Does such a market already exist?
You want to make a list of some companies you might
consider as partners or suppliers in China.
Again,
this scenario tested how users would find company
and industry information but looked further into
the types of information entrepreneurs would seek.
Prototype
B
Scenario 1. [Same as Prototype
A.]
Scenario
2. You are writing
a business case study on Bejing GongSi Corp. How
do you size up the company and its competitive enrionment?
What statistics and data do you look for/at?
This
scenario sought information about how business students
research case studies.
Scenario
3. What companies are leading
the productization of TS-CDMA-based products in
China?
As
for Prototype A, this scenario tested how a user
would find both company and industry information.
Scenario
4. [Same as Prototype A.]
|
| Procedure
|
|
| Three
MappingChina team members were present at each test,
functioning either as the facilitator, computer,
or observer. We wrote a script
that the facilitator read to each participant. After
reading the introduction and before proceeding to
the usability test, the facilitator had participants
read and sign consent
and records
release forms (although we ended up not
taking any pictures during the tests in order to
not impede on the participants' concentration).
The facilitator and computer then stepped the participants
through each scenario. We provided a copy of the
script to two partipants when it became apparent
that they would be more comfortable having a copy
of the scenario questions to refer to. The observer
recorded participant responses and questions. After
completion of the scenarios for both prototypes,
we debriefed the participants, asking them their
impressions of the prototypes, what they would have
liked to see on each of the pages, and what problems
they had with either the content, format, or navigation.
Although the facilitator was still the primary "speaker"
of the group during this process, the computer and
observer also asked questions.
|
|
Test Measures
|
|
|
| Because
we are still refining the scope of our project, our
primary goal for the usability testing was to put our
ideas into an interface context -- make them "real"
-- and then get feedback on content and placement. While
we didn't apply any formal metrics to the test and responses,
we looked to users to shed light on a number of issues:
- what
they thought of the "five forces" analysis,
both in concept and implementation;
- how
users wanted to enter the site, either through a portal
or a search interface;
- how
intrigued people were by the mapping feature;
- what
things users would like to see mapped;
- how
helpful is the news feature and how is it best categorized
and displayed;
- whether
city information is useful; and
- how
users would use a search function.
|
Results
|
|
|
| The
results
of our usability tests helped us reach consensus
on the issues discussed above in "Test Measures"
and alerted us to the presence of new issues we
had not noticed before. For example, the first
tester pointed out that the five forces term (developed
by Michael Porter) and diagram could be intimidating
to someone who had never seen or heard of it before.
A more accessible and generic term would be "competitive
analysis."
A major point of confusion found for both prototypes
was the search function. Testers were not sure
how to limit or specify the domain of their search,
and they didn't know what to expect in the results
page. There was also confusion over exactly what
constitutes an industry, as some technology terms
span multiple industry categories. Users liked
Prototype B's search interface, as it seemed to
allow them to focus keyword searches in specific
and multiple categories, but they expressed reservations
about the need to "look for the deep question
right away," rather than browse for information.
All
testers expressed doubts about the maps. While
intrigued by a visual indicator of where companies
and industries are located in China, users didn't
really know what they would do with that information.
They did offer a number of ideas for presenting
company and industry data on a map, which we will
be evaluating.
The
testing process revealed that our tool is not
the best for conducting a job search. One participant
told us bluntly that he probably wouldn't use
MappingChina for a job search, seeing it instead
as a tool for conducting competitive analyses
of an industry. We also realized that general
city information might also be outside the best
strategic scope for our project.
Our
tests also provided us with crucial basic information,
such as using the same web page format for industry,
sector, and company overview pages, joining relevant
information currently in a sidebar with the main
page, and providing a more hierarchical and apparent
way to "drill down" for information.
|
|
Discussion
| The
most important result of our usability testing
was seeing which features users viewed as most
central to our application. We offered mapping,
job information, news, and general city information
as well as industry analysis, and the consensus
seemed to be that if "five forces" or
competitive analysis was the key component of
our tool, we should dispense with some or all
of the other features. Our users' reactions
and feedback shifted our perspective back to focusing
on this core element.
Based
on this and other observations, we were able to
rank the various features in terms of importance:
1.
competitive industry analysis / efficient, flexible
search function
2. news
3. maps
This
ranking will drive the refinement of our design.
We will probably merge the home pages of the two
prototypes and more prominently feature the industry
analysis element, better-classify industry categories,
and hone the searching function to allow users
to search within selected keyword categories.
We will also develop a more interesting and sophisticated
way to visualize and implement the competitive
analysis. We'll eliminate the "City"
section of our site and reevaluate how we'll present
news. We'll think about better, more apparent
ways to hierarchically structure industry and
company information and refine the list of data
we'll include for each. At the most basic level,
we'll standardize the design of the pages and
consider how to present them, whether as pop-ups
or other panes or by using frames.
What
the evaluation didn't tell us was exactly which
industry and company data we'll need to include.
That will involve more research and discussions
with business students and professionals. |
Appendices
|
|
| | |
