Cognitive health screening for aviators would assist in managing a shortage of experienced pilots. Extending pilot careers by optimizing their cognitive health would address both the number and quality of pilots available for airline and general aviation operations. The present work tested the validity of an online screening tool for pilots that measures aviation domain-relevant cognition. Sixty-five licensed pilots (18–80 years, M = 48.8, SD = 16.3) with varying levels of experience completed a 30-min online cognitive health screening tool for pilots. Risk status was determined via a novel metric using self-reported incidents. Machine learning algorithms identified the cognitive factors most useful in identifying pilots with increased risk for accidents and serious incidents. Support vector machines and boosted decision tree algorithms provided the most reliable and strongest classifications models of pilot risk. Findings support the use of this short online screening tool for highlighting performance issues with domain-relevant cognitive abilities based on the Dynamic Mental Model for pilots, such as situation awareness and prospective memory. Understanding personal cognitive challenges is the basis for customized skill maintenance designed to augment cognition for those interested in safely extending their piloting careers.