Most Valuable Evaluation Technique
We felt that the pilot usability test was the most beneficial technique for advancing the development of this system. In a domain with such specialized knowledge as genomics, there is no substitute for input from people working in the field. As we progressed through the stages of development in this project, perhaps our greatest hindrance was the fact that nobody on the design team had the level of knowledge about the field that our users have. Questions would often arise as we worked, which we noted for future asking of test participants, but the opportunities to ask were so removed in time from the design work that we still had to do a lot of guessing. We felt in the end that a project like this could benefit from participatory design, involving a scientist from the user community much more directly in the design process.
Working with a functional prototype rather than a paper one gave users a much better opportunity to react to the system in a realistic way. Even we, as testers, felt more confident of user perceptions simply because we knew that what they were seeing was much closer to what they would see in the realized system. For example, it was difficult to assess the effectiveness of the popup window in the paper prototype, because it was not clear whether the participants realized that leaving one paper window on the desk below another meant that both windows would be open on the desktop.
Besides our obserations of real scientists using the system, the comments from our participants were extremely helpful. Though we are wary of direct design suggestions from scientists, recognizing that what users do and what they say they do are two different things and that users are not designers, we found the comments made during usability testing to be very helpful and appropriate, probably because they arose in the context of actual use.