Heuristic Evaluation for LightsOn
1. [H2 Match between system and the real world] (Severity 1) found by 1
The labeling of the "view lighting energy graph" button on the home page does not convey the content to which it takes users, since a lighting energy graph is already visible.
2. [H2 Match between system and the real world] (Severity 3) found by 3
Date labels on graphs list dates with decimals, like 2.5 and 7.5. Dates should be shown as integers.
3. [H2 Match between system and the real world] (Severity 2) found by 3
In the list of lighting problems, "previous" and "next" link labels lack context. Do they indicate the next day, next month, next year, depending on settings? The link text should indicate what this is the next of.
4. [H7 Flexibility and efficiency of use] (Severity 0) found by 1
On the problems page, it's hard to match up flags with their problems if no room number is associated. Scanning the list for room numbers is inefficient.
A tooltip with the identification info might help.
5. [H6 Recognition rather than recall] (Severity 3) found by 3
The meaning of icons relies on user memory. This could be especially problematic if the flags are used by more than one person.
6. [H4 Consistency and Standards] (Severity 2) found by 1
Flags can be toggled in the long problem list but not in the floorplan on the problems page or on the home page. The flags should either be static or they should be dynamic everywhere, and they should look more like buttons if they're dynamic.
7. [H5 Error prevention] (Severity 3) found by 2
Some items on the list have a green check to indicate that they are fixed, but they are on a list of "lighting problems". This makes it unclear whether an item is an actual problem or not. Users could erroneously attempt to fix a problem that has already been fixed. Fixed items belong in a separate list under lighting maintenance.
8. [H7 Flexibility and efficiency of use] (Severity 2) found by 1
In the problem list, there is no indication of where the user is in the set of next/previous "pages". Either indicate a page number or put them all on one page.
9. [H7 Flexibility and efficiency of use] (Severity 0) found by 1
In the problem list, it is difficult to determine which days are the weekend days, since there is no indication of days of the week. We included this because the task says to find the lights that broke "over the weekend", but we're not sure the significance of weekends to users.
10. [H7 Flexibility and efficiency of use] (Severity 2) found by 2
There is no way to set a duration for which a certain schedule runs. For changes not meant to be permanent, the user will have to come back and undo the changes at the end of the time period.
11. [H4 Consistency and standards] (Severity 2) found by 1
After editing a schedule, the save button is hard to find. It's typical to find the save button just below the settings it saves. It is odd to change a button label from edit to save in the same location. Users may not notice the button has changed purpose and not realize they need to save, especially since their changes do not need to be saved in other windows.
12. [H6 Recognition rather than recall] (Severity 2) found by 3
The requirement to select a room by clicking on it is not obvious. Users may not realize that the rooms are clickable or that they need to begin with making a selection.
13. [H3 User control and freedom] (Severity 3)
You need to give a way to cancel out of an edit for scheduling. The back button doesn't work here and shouldn't be relied upon in any case.
14. [H7 Flexibility and efficiency of use] (Severity 3) found by 2
Turning off all lights requires viewing one screen for each floor and clicking every room where a light is on. This is tedious. Also, it would be easy to turn off lights only on the first floor and forget the other floors. At a minimum, the number of lights turned on on each floor should be shown.
15. [H2 Match between system and the real world] (Severity 2) found by 1
Fonts are rather small, especially for a 51-year-old with failing eyesight. It is very aesthetic but difficult to read--especially the room numbers, which are not resizeable.
16. [H5 Error prevention] (Severity 3) found by 1
When turning lights on and off, the on/off buttons are visually separated from the selection area, so they are easy to miss. Users may not realize they are needed, especially when starting with all lights off, as the highlighting of rooms could be perceived as indication that they are toggled on.
17. [H2 Match between system and the real world] (Severity 2) found by 2
Placement of the current usage information on the "what lights are on" page is unexpected. Users may look for this information under "lighting energy usage".
18. [H2 Match between system and the real world] (Severity 3) found by 2
Graphs need a key to indicate which trace corresponds to which month, date, etc.
19. [H6 Recognition rather than recall] (Severity 2) found by 3
Selection vs. a light being on is a little confusing. They look pretty similar. The user has to remember which was which.
20. [H3 User control and freedom] (Severity 2) found by 1
In the schedule lights page, a user can hide the building plan and then click edit for a given schedule. However, when he clicks edit, the building plan is still hidden from view and it is not obvious how to add or remove rooms from the schedule.
21. [H7 Flexibility and efficiency of use] (Severity 2) found by 1
When I click "add new schedule", we expect the new, empty schedule to be immediately editable. Forcing the user click on "edit" is unnecessary and time-consuming.
22. [H7 Flexibility and efficiency of use] (Severity 2) found by 1
Users should be able to keep an inactive schedule in the system and make it active at will.
23. [H2 Match between system and the real world] (Severity 2) found by 1
The energy graphs should be printable, and the lines should be distinguishable in black and white. Dotted lines vs. solid lines would be good.
24. [H2 Match between system and the real world] (Severity 1) found by 1
The Fl label on lighting fixtures is ambiguous. One of us thought it was "fluorescent".
25.[H2 Match between system and the real world] (Severity 0) found by 1
The system should tell what kind of light each one is (fluorescent vs. incandescent, wattage) so that the person fixing it is prepared.
26. [H2 Match between system and the real world] (Severity 1) found by 1
Floorplan under lighting problems is labeled "unresolved problems". This heading doesn't apply to the floorplan specifically, yet it's placed directly above it.
Numbers of errors for each heuristic
| Heuristic | Number of errors |
|---|---|
| H1 | 0 |
| H2 | 10 |
| H3 | 2 |
| H4 | 2 |
| H5 | 2 |
| H6 | 3 |
| H7 | 7 |
| H8 | 0 |
| H9 | 0 |
| H10 | 0 |
Numbers of errors at each severity level
| Severity level | Number of errors |
|---|---|
| 0 | 3 |
| 1 | 3 |
| 2 | 13 |
| 3 | 7 |
| 4 | 0 |
General comments
It concerns us that it was so easy to turn off the lights in the third-floor men's bathroom. The fact that none of the other bathrooms had their lights on implies that someone might have been using it! My god, what if someone is locked in the bathroom in the dark at night! Our concerns would be put at ease if there were some sort of indicator that a light was on because it was scheduled to be on, or because someone had turned it on. If someone turned it on manually, the user had probably better leave it on!
We question the need to have so many floorplans. For example, it seems to us that the "turn lights on page" and the "what lights are on" page don't need to be separate.
There needs to be a way to print out a report. Besides getting a list of fixtures needing attention currently, users may want to get reports that tell when each problem was fixed.
It might be nice to show how many lights on each floor are broken. We suggest a table showing the light, the floor, and when it was broken.
Overall Recommendations
Overall, the LightsOn team has done an excellent job in providing a smooth interaction flow throughout the website and clear escapes from almost every state. The aesthetic design is excellent, and we really liked the summary dashboard information. Some very web 2.0 interactions; hopefully these will become more intuitive with time.
The main areas for improvement include conveying to users what parts of the website are interactive, and providing more information and communication as needed. It would also be instructive to flesh out more real-life use cases, in particular with regard to the maintenance staff and to extremely large or small buildings. There are some real problems with consistency in words and icons.