Discussion

After an Heuristic Evaluation was done on our interface, we made quite a few changes to our tool in order to resolve most of the issues of concern. As a result, our pilot study provided us with positive signs that our changes have made our interface much improved. We also found that most of our users want dynamic updating of active metadata once the metadata is checked from the checklist, rather than having to click Update. This is a feature that we have been planning to implement all along, but just have not developed it yet. Overall, our users found that our interface seems to be rather easy to use and much better than the current metadata management method available for UCWISE.

What we might change in our interface from these results

1. Dynamic Update of Active Metadata. We agree with our users that a dynamic update of active metadata upon selection is a key feature to making our interface simplified and more usable. We do not want our users to have to make too many unnecessary clicks to accomplish their tasks and so we have hopes of implementing that feature into our interface.
2. Altering the Look and Feel of Our Navigation Bar. We have been getting mixed reviews about our navigation bar. Since there is not any real process involved with using our tool, the use of a progress train did not make sense to some of our test users. We are not sure if we are going to keep it or implement a different method of navigation.
3. Making Choose Activity Type Visible. Some of our users missed the Choose Activity Type area in the active metadata section of Bookkeeping. We are most likely just going to highlight that text so it is visible and the first thing the user notices when entering that section.
4. Getting Rid of Go Button. This falls in line with other forms of dynamic implementation. We noticed that some of our test users completely forgot to click on Go after selecting an activity type in Bookkeeping. We might just have the form come up dynamically once the activity type is clicked.
5. Making Reset Text Button Less Prominent and farther from the Save Button. The Reset Text button for the active metadata notes in our interface is too big and too close to the Save button, making it a very easy target for clicking. We want the button to be noticeable but not more overpowering than Save.

What we might change for the real experiment

We would have all the participants go through all 3 scenarios. This is because the fatigue factor would not really be a concern in our experiment given each scenario does not take a long time to complete. Another reason we would like to have all of our participants go through all 3 scenarios is we want to determine the overall satisfaction of our potential users with the metadata tagging tool. Do the three metadata tagging scenarios actually capture the typical scenarios that our potential users would encounter?

We would divide the participants into six groups, and present a different ordering of the three scenarios to each group (e.g. perform scenario 1, 2, 3 for group 1, perform scenario 1, 3, 2 for group 2. etc) to go through. This is to test whether the ordering on the progress train affects the efficiency of the users in selecting and finding the different metadata tagging section. For example, does the fact that the instructor notes metadata section being the fourth item in the progress train causes the user to click on the cognitive learning metadata and the topics metadata before landing on the instructor notes metadata? Or does the user just click on the instructor notes metadata section based on the instructor notes metadata label? We could not measure this if we only have all the users go through only one type of ordering for the scenarios. This is also to test whether the labeling for the progress train is clear or whether we need to add a pop-up help menu to assist the user in the metadata tagging navigation.

Since the real experiment would involve more participants, we would also take a survey to determine from this larger pool of users which types of cognitive metadata tagging tasks they do most frequently, so we can make the more frequently conducted tagging task interface more accessible (such as placing them earlier in the progress train) to the majority of the users.