Comparing control room operators' and experts' assessment of team performance using structured task-specific observation protocols and scenario replay

Per Øivind Braarud

doi:10.1016/j.apergo.2021.103500

Abstract

Operators' self-assessment has received limited interest within process control or human-system evaluation. Research on self-assessment has been criticised for poor assessment methodology, and consequently, its status is unclear. This study hypothesised that, given adequate assessment methods (such as task-specific assessment items and scenario replay), we could observe relatively accurate self-assessment results. Eighteen licensed operators and two experts assessed team performance in six nuclear control room scenarios. The results reveal an overall agreement between operators and experts, measured by the intraclass correlation coefficient, ranging from 0.60 to 0.70, which lies close to the intraclass correlation coefficient of 0.75 for the experts. This demonstrates potential for achievement of relatively accurate operator self-assessment for complex work. The agreement varied in a similar manner for both expert agreement and operator-expert agreement across eight performance dimensions. In addition, the operators’ self-assessment provided additional information beyond observer assessment in identifying non-acceptable performance items.

Highlights

Operators’ performance and work conditions receive considerable interest due to their importance in ensuring safe and efficient operation
Regarding the self-assessment of performance, the operators quickly became familiar with navigating the replay tool and the performance assessment software
This study provided participating operators and experts with the same scenario replay tool, the same event specific assessment protocol, and similar assessment procedures to investigate self-assessment as compared to expert assessment

Summary

Introduction

Operators’ performance and work conditions receive considerable interest due to their importance in ensuring safe and efficient operation. Team performance assessment in complex human-machine settings are challenging to undertake (Vreuls and Obermayer, 1985; Rosen et al, 2008) and involve substantial degrees of judgement regarding the safety implications of observed behaviours (Hall and Brannick, 2009). Subject matter experts observe team performance and use structured observation protocols developed via task analysis in their assessment of performance (Landy and Farr, 1983; Hall and Brannick, 2009; Wildman et al, 2013). Self-assessment is conducted through comparing self-observed performance against a certain standard (Colthart et al, 2008; Van Loon, 2018). One must assume the validity of self-assessment when applying its use (Boud, 1995; Van Loon, 2018)

Results

Discussion

Conclusion