Metrics for Supervisory Control System Evaluation

Abstract

Little formal guidance exists on how to select a set of meaningful metrics for a human supervisory control system, which is inherently complex with significant embedded autonomy. With the increasing reliance on automation in these complex settings, it is critical that key performance metrics be identified to indicate not only operator and automation performance, but integrated human-system performance as well. To this end, this chapter will describe a supervisory control metric taxonomy that classifies different metric classes across supervisory control systems, and provide example metrics, how they relate, and how this taxonomy can be used to identify a robust set of metrics. In addition, we discuss selection of a parsimonious set of metrics based on a cost-benefit analysis approach, which ultimately depends on the overarching objectives of the researcher or practitioner.