Reweighting AT-SAT to Mitigate Group Score Differences

Abstract

The Air Traffic Selection and Training (AT-SAT) test battery is the selection tool for applicants for Air Traffic Control Specialist (ATCS) positions within the Federal Aviation Administration (FAA) who have not previously been employed as an air traffic controller. AT-SAT is an aptitude test developed to predict the likelihood of successfully learning ATCS skills. Before operational use, however, concerns were raised about the low passing rate of incumbent (who are fully trained and certified) ATCS personnel (who participated in the initial research) and score differences between groups, which could result in adverse impact (possible unfair discrimination). To address these concerns, the subscores of AT-SAT were reweighted, and the additive constant was changed to yield a new total score. The present study compares the original and new scoring methods using data from 724 developmental ATCSs who volunteered to take AT-SAT. An average increase of 4.86 points was found with the new scoring method; the notional passing rate (achieving a score ≥ 70) changed from 58.8% to 80%. American Indian/Alaskan Native, Hispanic, and black participants showed the greatest average increase in overall scores, 6.97, 6.98, and 7.02, respectively. The increase in scores of Hispanic and black participants was significantly higher than the increase in scores for white participants [F(4, 689) = 6.186, p < .001]. However, a chi square analysis showed no differences between groups for the number of participants whose failing score with the original scoring method changed to a passing score with the new scoring method. Additionally, a Spearman rank correlation coefficient of .85 was found between the two scoring methods, indicating that the ranking of individual participants did not change significantly. Moreover, no differences were found between groups in rank ordering of the two scoring methods. No significant gender differences were found between the scoring methods, with the scores for males increasing an average of 4.58 points and scores for females increasing an average of 5.67 points under the new weighting method. This study found that the new weighting formula has benefited all groups and is likely to reduce the potential of adverse impact.