Opportunities for Human Factors Measures in Military Operational Test and Evaluation

Abstract

Military Operational Test and Evaluation (OT&E) is conducted under operationally realistic conditions across a full spectrum of military operations to determine the readiness of a system to be fielded. OT&E seeks to assess the operational effectiveness and suitability of the system under test. Operational effectiveness measures the overall ability of a system to enable mission accomplishment when used by warfighters in a representative operational environment. Operational effectiveness seeks to evaluate organization, doctrine, and tactics. Operational suitability measures the overall ability of a system to be placed in field use. Operational suitability assesses reliability, availability, and human factors. To date, little effort has been placed on evaluating the warfighters' performance as part of the overall ability of the system to enable mission accomplishment. In fact, any warfighter performance issues are normally reported by exception, which means that it is not evaluated unless a problem is identified during testing (i.e., there is no formal plan to evaluate the warfighter performance component of overall system performance). Most analysis focused on the human factor has been done through the use of opinion surveys. Although, opinion surveys can provide insight into how well a warfighter likes a new system, they do not provide information on the new system's impacts on the warfighters' cognitive capabilities and ultimate performance, nor on the impact on the performance of a team of warfighters. Over many years, the human factors domain has developed measures of cognition (mental workload, situation awareness, decision making, etc.) and teamwork (collaboration, coordination, shared SA, social network analysis, etc.) that should be applied during military OT&E. These types of human factor measures can be used to evaluate both operational effectiveness and suitability, providing a method of analyzing the human component in mission effectiveness and the impact of the system under test on the human component. Without objectively evaluating the human component of the overall system, it is impossible to truly evaluate operational effectiveness and suitability. This panel will present the need for more objective, comprehensive, and automatically-collected human performance measurements in military OT&E, and earlier in the development and testing, especially in relation to command and control (C2) systems, as well as touch upon the potentially useful human factor measures currently available and how to select the correct ones to use.