Guidelines for bootstrapping validity coefficients in ATCS selection research.

Abstract

Abstract : This technical report 1) reviews the literature on bootstrapping estimation procedures and potential applications to the selection of air traffic control specialists (ATCSs), 2) describes an empirical demonstration of procedures for estimating the sample size required to demonstrate criterion-related validity in ATCS selection, and 3) provides summary guidelines and recommendations for estimating sample size requirements in ATCS selection test validation using bootstrapping procedures under conditions of direct and indirect range restriction. Bootstrapping estimates the sampling distribution of a statistic by iteratively resampling cases from a set of observed data. Confidence intervals are constructed for the statistic, providing an empirical basis for inferential statements about the likely magnitude of the statistic. Correlations between scores on the written ATCS aptitude test battery and subsequent performance in initial qualification training for a large sample of 10,869 controllers hired between 1986 and 1992 were bootstrapped in an empirical demonstration of the methodology. Finally, a three-step sequence of procedures is described for use in future bootstrap estimates of confidence intervals. Recommendations for sample size requirements in future ATC criterion validity studies include: 1. Results suggest samples of at least N 175 to ensure the 90% confidence interval for does not contain 0. 2. Assumptions of bivariate normality in traditional parametric estimation procedures are not justified in the current data. Note that this observation may result in confidence intervals that are wider or narrower for any given sample size than intervals obtained from traditional parametric estimation. 3. Corrections for direct range restriction did not substantively influence whether the bootstrapped 90% confidence interval contained 0. Future applications should assess whether this holds true.