Responsible machine learning for United States Air Force pilot candidate selection

Abstract

The United States Air Force (USAF) continues to be plagued by a chronic pilot shortage, one that could be exacerbated by an accompanying shortfall in the commercial airlines. As a result, efforts have increased to alleviate this shortage by finding methods to reduce pilot training attrition. We contribute to these efforts by setting forth a decision support system (DSS) for pilot candidate selection using modern machine learning techniques. In view of the recent Responsible Artificial Intelligence Strategy published by the United States Department of Defense, this research leverages interpretable and explainable machine learning methods to create traceable and equitable models that may be responsibly and reliably governed. These models are used to regress candidates’ average merit assignment selection system scores based on information available for selection and prior to training. More specifically, using data provided by the USAF from 2010 to 2018, this paper develops and analyzes multiple interpretable models based on Gaussian Bayesian networks, as well as multiple black-box models rendered explainable by SHAP values and conformal prediction. A preferred pair of interpretable and explainable models is selected and embedded within a DSS for USAF pilot candidate selection boards: the Air Force Pilot Applicant Selection System. The utilization of this DSS is explored, the analyses it enables are discussed, and relevant USAF policymaking issues are examined.