Challenges to Assessing Usability in the Wild: A Case Study

Abstract

This article describes one part of a human factors study conducted over 3 months in a petro-chemical manufacturing plant in Australia. The project had two purposes, namely, to identify issues to be included in a training course for plant operators and to identify low-level usability-related software issues that might be rectifiable prior to system implementation. After interviewing 28 operators and eight managers, the operators were observed on the job while interacting with the old system. Finally, the 3-part usability assessment comprising 2 expert inspections and a user-based quasi-walkthrough was conducted. As the study took place shortly before a new, off-the-shelf automated manufacturing system was implemented, it was not possible to test an interactive version, relying instead exclusively on static screens. This made it impossible to provide user performance data, which could have helped to convince management of the seriousness of certain problems. One of these proved so severe that an engineer had to be present 24/7 in the control room for 6 months following system cutover because the operators were unable to achieve the required product quality. Based on the data, suggestions are made for expanding the usability construct to include assessment of perceived technology usefulness and to refine the concept of attitude in mandatory settings.