Test Measures

Appendices

Observation Log Data: Tester 1 | Tester 2 | Tester 3

..................................................................................................................................

Usability measurement is always a challenging task due to its subjectivity, competing theories and human factors involved. We make planning in our test preparation kit to take measurements of user achievements and satisfaction as long as it is possible and relevant to the interaction. In detail scale we are aiming at measuring the efficacy of our design choices on:

• Navigation: the use of ease leading user to specific pages;
• Instruction and feedback: the natural information overflow the users;
• Terminology recognition. This has always been a problem and we have to invest a considerate propotion of efforts on that with uncertain The pilot test offers a chance to examine our assumptions..

With only 3 participants to carry out the real test, our measurements may not be statistics sound enough in evaluating the overall usability, yet it nonetheless provides feedback to and could serve as a pilot usability test on a broader audience in a formal experiment.

Our measurements embraced subjective and objective dimensions, in quantitative and qualitative forms.

Subjective vs. Objective: Subjective nature of the user experience is important part of our measurements. We plan in various part of our test kit to measure subjective aspect of user satisfaction. Most of them are in qualitative form, such as survey questions, observations, or more open ended general questions. We are more focused on subjective and qualitative data in this regard.

However, quantitative methods are also employed. Likert scale, for example, is used to gather user satisfaction data and would find application for more representative sample. Those data, in a larger pool of participants, could yield more meaningful results to shed light on design choices and improvement.

Objective data includes the quantitative measurement we take during the test. They are independent from user perception or their claim. It is very interesting to notice discrepancy between the subjective and objective results. Specifically, contrary to our assumption, some complication of the interaction and longer process time does not always mean negative user experience. More study will be intriguing in formal experiment.

Quantitative vs. Qualitative: Both of them can be used to acquire subjective and objective data. Most of the user feedback we recorded are in qualitative manner. So are our observation logs.

Quantitative data are gathered in:

• Time taken to perform a certain task;
• Number of wrong clicks user committed before fulfilling the mission;
• Likert scale on user satisfaction;
• Average rating of the above Likert scale data.

In our test preparation kit, we had planned to measure how long it may take a user to complete the Task 2, related to Registration Process. In general few seconds were taken in this task.

Under normal circumstances the first few testers would have been pilot testers who help us work out bugs in the testing procedures and we would follow a protocol more closely with a larger sample pool of users. For this set of tests, we observed and noted difficulties that users would have and the wrong places where they had "clicked" to complete a task. Often, testers spoke out loud about the problems they were having. Notes from those speeches, the task surveys, and discussions with the user helped us understand how a user felt about a particular task, the steps it took, and the amount and quality of information involved.

The user evaluation is presented at Results, while the user navigation process and comments can be found at the Observation Log Data: Tester 1 | Tester 2 | Tester 3.

 

 

^