A Test-Battery Approach to Cognitive Engineering: To Meta-Measure or not to Meta-Measure, that is the Question!

Abstract

This paper describes a task battery approach to system evaluation and cognitive engineering. The limitations of current techniques are discussed. An alternative paradigm using both task-specific measures and design driver measures is suggested. Such high level design driver measures have been referred to as meta-measures (Hardiman et al, 1995). In an experiment conducted to evaluate the utility of using meta-measures within the context of cockpit system assessment, an explanatory display for air combat decision support was tested within a realistic flight simulation. Multiple measures of performance were taken, together with the Situational Awareness Rating Technique (SART) as a meta-measure of Situational Awareness. Results showed that although task-specific measures clearly indicated one candidate design as producing better performance, they allowed little explanation of the underlying mechanism. Concurrent analysis of the meta-measure, however, enabled clarification of the cause of the performance differences, in this case through improved understanding, more information being available, and the information being more task relevant. The potential advantages of this approach for cognitive engineering are discussed.