IS 213 - Communications Spectrum

The USA Radio Spectrum Information & Awareness site is an information service targeted at policy analysts and journalists who are interested in understanding current spectrum allocation policy and usage. The site provides a tutorial that covers general information about current spectrum usage and allocation policy. It also offers regional and industry-specific views of Radio Spectrum usage data.

Since the last stage of the project we trimmed the scope further, and now intend to offer data views only by region and by industry. The site now offers three primary paths: the tutorial, browse the data by region, and browse the data by industry. To access the tutorial and all of the data views, users are required to learn only two different interactional schemes.

See the revised interaction flow for a current overview of the page flow of the site.

2. Prototype

Our primary goal at this stage of the design process is to test the basic organization of information on the site, as well as the labeling and navigation. Accordingly, our low-fi pages and test scenarios emphasize navigational elements.

We chose a standard graphical metaphor for the electromagnetic spectrum that is maintained consistently throughout the site. A horizontal bar representing a range of the spectrum is the header graphic on each page. The section of the spectrum that the page's content refers to is pulled out and magnified.

See photographs of the low-fi prototype.

3. Method

We have, up to this point in our design process, discussed project ideas with potential users over the phone. Due to our inability to get face-to-face access to these individuals we found testers in Berkeley who match our personas as closely as possible.

3a. Participants

Participant 1

Participant 1 is a second year masters student in the Goldman School of Public Policy at UC Berkeley. Participant 1 is a female in her early 30s. Her focus is International Economic Policy. She had little previous knowledge of the communications spectrum, but she does have some knowledge of the ongoing controversies surrounding its use.

Participant 2

Participant 2 is a middle aged, female librarian for the Institute of Governmental Studies at UC Berkeley. She is very knowledgable about political issues and an expert researcher.

Participant 3

Participant 3 is a second year SIMS student who has a Bachelor of Arts degree in Communications. Participant 3 is a female in her mid-twenties and has a lot of computer experience. She has little previous knowledge of the communications spectrum and the issues surrounding it.

3b. Task Scenarios

Test Scenario 1: You know very little about the communications spectrum. The site provides a tutorial. In order to learn more about the spectrum before continuing using the site, use the tutorial (in any manner that suits you.)

We were looking for users to begin this scenario by clicking on one of the two links to the tutorial from the main page. Then users could either navigate through the tutorial sequentially or browse portions of the tutorial that interested them. The low-fi prototype provided sufficient means for testers to use either of these methods.

Test Scenario 2: You would like to investigate how Broadcast TV spectrum usage in Helena, Montana compares to Broadcast TV spectrum usage in New York City. Use the system to find pages containing information that could help you do this.

For this scenario, we expected users to choose the 'Browse by Industry' option from the front page of the site. This brings users to a list of all possible industries from which they can choose. For this scenario, users were expected to select Broadcast TV. Then, upon arriving at the city selection page, users were to select Helena and New York City to arrive at the data pages.

Test Scenario 3: You live in Helena, Montana. You are interested in determining the differences between the Broadcast TV spectrum usage and the AM/FM radio spectrum usage within Helena. Use the system to help you determine this.

For this scenario, we expected users to choose the 'Browse by Region' option from the front page of the site. This brings users to a list of all possible cities from which they can choose. For this scenario, users were expected to select Helena, MT. Then, upon arriving at the industry selection page, users were to select the Broadcast TV and the AM/FM Radio industries to arrive at the proper data pages.

3c. Procedure

The low-fi prototype is comprised of thirteen sheets of paper, each representing a page of the web site. The home page contains a unique header and core content area, as well as a left side navigation bar that is constant throughout the site. As testers navigate to different areas of the site, we place the pages they choose over the center of the home page, effectively changing the header and body copy of the lower level pages, but maintaining the left hand, site-wide navigation.

After discussing the purpose of our project, describing the testing process, and having our testers sign consent forms, we sat down with each tester at an empty table and presented her with the home page. One person played computer and simply swapped in the pages that the testers requested. The other person described each of the tasks to the tester and took notes.

4. Test Measures

Scenario 1 was designed to:

Determine whether the tutorial is intuitive. We expected users to have seen web-based tutorials before and we wanted to determine if the navigation scheme of our tutorial would appear familiar to testers.
Determine if the spectrum industries were organized into appropriate categories that users could easily identify and understand for future system use.

Scenario 2 and 3 were designed to:

Determine if users found the two available data mining processes easy and intuitive.
Determine if users easily understood the difference between the two data mining processes and their results.
Determine if the two data comparisons return information that is valuable to users.

All three scenarios were designed to:

Test the navigability of the site.
Determine if users find the site terminology comprehensible.

5. Results

We gave users post test surveys to give us an indication of how well the low-fi prototype measured in the areas we were testing. We designed this survey based on a combination of Jacob Nielsen's heuristic test factors and of the test measures which we developed above. Several of the question results are shown below (in all cases, 1 indicated the extreme negative answer while 5 indicated the extreme positive answer.) Please see Appendix B for the full survey.

Rate how well you felt you were able to maintain awareness of your location in the system at any given time.

1

2

3

4

5

User 1

X

User 2

X

User 3

X

Rate how well you were able to understand the terminology used on the site.

	1	2	3	4	5
User 1			X
User 2				X
User 3				X

Rate the consistency of the interface, specifically regarding the terminology and navigation techniques used throughout the site.

	1	2	3	4	5
User 1				X
User 2				X
User 3				X

Rate how easy you felt it was to navigate through the tutorial.

	1	2	3	4	5
User 1				X
User 2			X
User 3			X

Rate how easy you felt it was to navigate through the rest of the site (i.e. outside of the tutorial).

	1	2	3	4	5
User 1				X
User 2				X
User 3				X

Rate how easily were you able to differentiate between the information that was returned to you in the second scenario to that of the third scenario.

	1	2	3	4	5
User 1					X
User 2			X
User 3			X

Scenario 1 Results

Users were able to complete scenario 1 without too much difficulty. As we expected, testers chose both navigation methods in order to use the tutorial. Two users completed the tutorial sequentially while one user chose to view topics which were of interest to her by clicking on the links at the top of the tutorial. After she browsed 3 of 4 categories, she chose to move on.

Users had the most trouble understanding how the 'frequency and wavelength', 'users', and 'equipment' information boxes on the left hand side of the tutorial pages related to the industry information on the right hand side of the tutorial pages. One user noted that she thought the top box related to the top industry, the middle box related to the second industry, and the third box related to the third industry, while in fact the three boxes are supposed to contain information about the industries as a whole.

The tutorial feature which received the most positive feedback was the spectrum graphic at the top. It took users two tutorial pages to understand how the spectrum graphic related to the industries listed below it, but upon grasping the concept, users indicated that it helped them tremendously in understanding the basic concept of how the spectrum is allocated.

Users didn't seem to experience any difficulty navigating through the tutorial. On our post test questionnaire, all three users indicated that they had previous experience with web-based tutorials, and that the navigation techniques used in our tutorial seemed familiar to them.

Scenario 2 Results

Scenario 2 proved to be by far the most problematic of the three scenarios. All three users attempted to find data for two cities by selecting to 'Browse Data by Region' rather than 'Browse Data by Industry.' Upon realizing that choosing 'Browse Data by Region' would only allow them to view information for one city at a time, two users indicated that they would simply open another browser window while the third user stopped performing the task. After taking users back to the main page, they had no trouble completing this task.

After the usability test, the three users stated that the site provided no up front information about what the two links would allow them to do and further, that the labels of the two links were counter-intuitive to what they actually did.

Scenario 3 Results

Scenario 3 was designed to make users compare two industries within one region, i.e. to complete the counterpart act of scenario 2. Having gone down this path while performing scenario 2 incorrectly, users automatically completed this task the correct way. However, all three users indicated that had they performed scenario 3 before scenario 2, they would have run into similar problems as they did with scenario 2.

6. Discussion

Based on the results of our low-fi usability tests, we are planning to change the following aspects of our interface:

Home Page

Include instructions in the main content section to indicate what the 'Browse by Region' and 'Browse by Industry' options allow users to accomplish.
Change the titles of 'Browse by Region' and 'Browse by Industry' to be something that is more intuitive to users.
Make the use of the logo as a link to the main page more apparent.
Include content indicating what kind of information users can find by clicking on links in the 'Related Links' page.

Tutorial

Include a terminology dictionary to help users understand the terminology used throughout the site.
Include a clear division between the technologies and the left hand information boxes.
Make a stronger connection between how the spectrum graphic relates to the industries, by lighting up particular spectrum regions when the user scrolls the mouse over the industry listing in the main content area of each page.

Data Browsing Pages

Possibly include a front page to instruct users about the selections that they can make throughout the data browsing process as well as what data will appear at the end of the selection process.

What the Evaluation Did Not Tell Us

How well users will be able to piece together the information from the tutorial and the data browsing features in order to understand a whole concept within the Communications Spectrum.
If this content is at a level which beginner users can understand.

7. Appendices

Appendix A: Usability Test Script

Appendix B: Post Test Questionnaire

Appendix C: Usability Test Logs

Appendix D: Full Photo Set of Prototype

8. Work Distribution

SIMS 213 Spring 2003

SIMS 213
Spring 2003