Optimal Human–Machine Teaming for a Sequential Inspection Operation

Abstract

A novel mixed initiative optimal control system for intelligence, surveillance and reconnaissance (ISR) operations which entails human–machine teaming has been developed. The scenario entails a camera-equipped unmanned air vehicle sequentially overflying geolocated objects of interest, which need to be classified as either a true or false target by a human operator. The vehicle is allowed a prespecified number of revisits, such that an object can be looked at, a second time, under better viewing conditions. The overarching goal is to correctly classify the objects and minimize the false alarm (FA) and missed detection (MD) rates. We design a stochastic controller that computes if and when a revisit is necessary and also the optimal revisit state, i.e., viewing altitude and aspect angle. The concept of operation is such that the critical task of detection/pattern recognition is relegated to the human operator, whereas optimal decision making is entrusted to the machine. The stochastic dynamic programming-based decision algorithm is, however, informed about the performance of the human operator via an empirical human perception model. The model is experimentally obtained in the form of state-dependent confusion matrices. The optimal closed-loop ISR system is shown to experimentally achieve a FA rate of 5% and MD rate of 12%, which are significantly lower than the open-loop operator-only performance metrics. The performance improvements that were observed are relevant to a particular operator, and thus, the study suggests that the same improvements could conceivably be achieved with other test subjects.