Validity Of Cognitive Ability Tests Research Articles

A critical review of the use of cognitive ability testing for selection into graduate and higher professional occupations

AbstractThis article presents a critical review of the use of cognitive ability testing for access to graduate and higher professional occupations to promote further debate and reflection in both the academic and practitioner community. The main contentions are that the practice of applying cognitive ability testing in these contexts has strong potential to both maintain and exacerbate social inequality in access to higher occupations and professions, and that validity evidence does not justify this to the extent that has previously been presumed. Five critical observations are examined, namely (1) evidence of adverse impact in test outcomes; (2) the tendency to position cognitive ability testing early in selection processes in high‐volume recruitment; (3) recent evidence challenging the meta‐analytic validity of cognitive ability tests; (4) weaknesses in historical primary validity studies; (5) conceptually flawed examination of differential validity evidence in the literature. Implications for practice are discussed, contrasting strategies that involve modifying selection systems that include cognitive testing, versus removing and replacing cognitive tests.

Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

AbstractPrevious studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta‐analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta‐analysis of the Hispanic–White mean difference (d‐value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d‐values for indirect range restriction and combine them with other meta‐analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion‐related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.

Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research.

Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent-based approach vis-à-vis the applicant-based approach. To address this issue, we conducted several Monte Carlo simulations, as well as an analysis of college admissions data. Our first simulation study showed that incumbent-based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant-based corrections were quite accurate. Our second set of simulations showed that incumbent-based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant-based corrections tended to accurately estimate population parameters and showed little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent-based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent-based range restriction corrections in lieu of applicant-based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance. (PsycINFO Database Record

Behavior profiling: implications for recruitment and team building

Purpose – This paper aims to address the role of behavior profiling in personnel recruitment and selection and how managers would benefit by using behavior assessment. Psychometric tests that measure cognitive ability and behavioral traits are regularly used to support management decisions in recruitment, team building and leadership development. But are they effective as a determinant of hiring and can their use be justified? Design/methodology/approach – The relationship between personality and job performance has been studied frequently in business psychology, particularly so in the past 20 years. This paper sets out by discussing the advent of personality profiling and exploring the advances made to where we are today with behavior assessment tests. It addresses the views of academics and the use of behavior assessment in the corporate setting. Findings – There are many studies that document the validity of cognitive ability tests and personality/behavior profiling as predictors of work performance. Of the numerous procedures used for recruitment and selection, two combinations – general mental ability (cognitive ability or GMA) plus an integrity test (personality profiling) and GMA plus a structured interview – have the highest multivariate validity and utility for job performance. Behavior profiling is an effective determinant of hiring. Practical implications – Adopting personnel selection methods with high predictive validity ensures that managers can make decisions with a stronger degree of certainty and confidence. Using behavior profiling in conjunction with cognitive ability measures and sound interview techniques maximizes the predictive power of the selection process. Social implications – Behavior assessment tests should also benefit those taking the test by providing a facility that conveys important issues about their personality and behavior. This will allow them to discover areas that need to be addressed to make positive changes within themselves to achieve their ultimate goals in life, both personal and career. Originality/value – This paper addresses, among other things, the research of several people into the use of personality measures in personnel selection. The paper will benefit all those involved with recruitment, team building and leadership development by improving their understanding of the use of behavior assessment tests.

Differential validity for cognitive ability tests in employment and educational settings: Not much more than range restriction?

The concept of differential validity suggests that cognitive ability tests are associated with varying levels of validity across ethnic groups, such that validity is lower in certain ethnic subgroups than in others. A recent meta-analysis has revived the viability of this concept. Unfortunately, data were not available in this meta-analysis to correct for range restriction within ethnic groups. We reviewed the differential validity literature and conducted 4 studies. In Study 1, we empirically demonstrated that using a cognitive ability test with a common cutoff decreases variance in test scores of Black subgroup samples more than in White samples. In Study 2, we developed a simulation that examined the effects of range restriction on estimates of differential validity. Results demonstrated that different levels of range restriction for subgroups can explain the apparent observed differential validity results in employment and educational settings (but not military settings) when no differential validity exists in the population. In Study 3, we conducted a simulation in which we examined how one corrects for range restriction affects the accuracy of these corrections. Results suggest that the correction approach using a common range restriction ratio for various subgroups may create or perpetuate the illusion of differential validity and that corrections are most accurate when done within each subgroup. Finally, in Study 4, we conducted a simulation in which we assumed differential validity in the population. We found that range restriction artificially increased the size of observed differential validity estimates when the validity of cognitive ability tests was assumed to be higher among Whites. Overall, we suggest that the concept of differential validity may be largely artifactual and current data are not definitive enough to suggest such effects exist.

Racial/ethnic differences in the criterion-related validity of cognitive ability tests: A qualitative and quantitative review.

The correlation between cognitive ability test scores and performance was separately meta-analyzed for Asian, Black, Hispanic, and White racial/ethnic subgroups. Compared to the average White observed correlation ( = .33, N = 903,779), average correlations were lower for Black samples ( = .24, N = 112,194) and Hispanic samples ( = .30, N = 51,205) and approximately equal for Asian samples ( = .33, N = 80,705). Despite some moderating effects (e.g., type of performance criterion, decade of data collection, job complexity), validity favored White over Black and Hispanic test takers in almost all conditions that included a sizable number of studies. Black-White validity comparisons were possible both across and within the 3 broad domains that use cognitive ability tests for high-stakes selection and placement: civilian employment, educational admissions, and the military. The trend of lower Black validity was repeated in each domain; however, average Black-White validity differences were largest in military studies and smallest in educational and employment studies. Further investigation of the reasons for these validity differences is warranted.

Test anxiety and the validity of cognitive tests: A confirmatory factor analysis perspective and some empirical findings

The validity of cognitive ability tests is often interpreted solely as a function of the cognitive abilities that these tests are supposed to measure, but other factors may be at play. The effects of test anxiety on the criterion related validity (CRV) of tests was the topic of a recent study by Reeve, Heggestad, and Lievens (2009) (Reeve, C. L., Heggestad, E. D., & Lievens, F. (2009). Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence, 37, 34−41.). They proposed a model on the basis of classical test theory, and concluded on the basis of data simulations that test anxiety typically decreases the CRV. In this paper, we view the effects of test anxiety on cognitive ability test scores and its implications for validity coefficients from the perspective of confirmatory factor analysis. We argue that CRV will be increased above the effect of targeted constructs if test anxiety affects both predictor and criterion performance. This prediction is tested empirically by considering convergent validity of subtests in five experimental studies of the effect of stereotype threat on test performance. Results show that the effects of test anxiety on cognitive test performance may actually enhance the validity of tests.

Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests

The assessment of cognitive abilities, whether it is for purposes of basic research or applied decision making, is potentially susceptible to both facilitating and debilitating influences. However, relatively little research has examined the degree to which these factors might moderate the criterion-related validity of cognitive ability tests. To address this gap, we use Classical Test Theory formulas to articulate how test anxiety and test familiarity can influence observed scores, observed score variance, and most importantly, the criterion-related validity of observed scores. The resulting equations reveal that understanding the influence of test anxiety and test familiarity on criterion-related validity coefficients requires the consideration of a number of additional parameters. To elucidate the implications of the model, we present a Monte Carlo simulation. Results show that anxiety and familiarity can have a significant negative effect on the observed criterion-related validity, but also show that this effect is highly variable. In particular, the effect depends heavily upon the relation between these factors and the criterion variable. Additionally, we note that the equations we develop highlight important gaps in the literature; there are few clear empirical estimates of several of the parameters in our formulas. We call for future research to better examine these additional relations.

The predictive validity of cognitive ability tests: A UK meta‐analysis

A meta‐analysis on the validity of tests of general mental ability (GMA) and specific cognitive abilities for predicting job performance and training success in the UK was conducted. An extensive literature search resulted in a database of 283 independent samples with job performance as the criterion (N=13,262), and 223 with training success as the criterion (N=75,311). Primary studies were also coded by occupational group, resulting in seven main groups (clerical, engineer, professional, driver, operator, manager, and sales), and by type of specific ability test (verbal, numerical, perceptual, and spatial). Results indicate that GMA and specific ability tests are valid predictors of both job performance and training success, with operational validities in the magnitude of .5–.6. Minor differences between these UK findings and previous US meta‐analyses are reported. As expected, operational validities were moderated by occupational group, with occupational families possessing greater job complexity demonstrating higher operational validities between cognitive tests and job performance and training success. Implications for the practical use of tests of GMA and specific cognitive abilities in the context of UK selection practices are discussed in conclusion.

EXPLORING OPTIONS FOR SUPPORTING TEST USE IN SITUATIONS PRECLUDING LOCAL VALIDATION

This paper examines some of the options available to practitioners interested in supporting the use of selection measures in an organization, including test transportability, validity generalization (VG), and synthetic validation, reviewing some of the advantages, disadvantages and requirements of each approach. Results of four unpublished, proprietary validation studies are reported which compare validity estimates provided by the job component validation (JCV; a type of synthetic validation) routine inherent in the Position Analysis Questionnaire (PAQ) with observed validity coefficients for a variety of criterion measures. We then examine the accuracy of the JCV procedure in predicting validity coefficients for 51 clerical classifications extracted from an existing PAQ job evaluation database of a large utility company. Predicted JCVs are compared to mean observed validity coefficients for five DOT clerical categories provided by Pearlman, Schmidt, and Hunter (1980). The VG and JCV methods provided highly similar and converging estimates of the validity of cognitive ability tests for predicting performance in clerical occupations. Implications for practice are discussed, particularly the need to use multiple, converging lines of evidence to support test use.

FURTHER EVALUATION OF VERBAL ABILITY SELECTION TEST AND WORK PERFORMANCE VALIDITY WITH PSYCHIATRIC AIDES

Verbal selection test and work performance validity based on two samples of psychiatric aides were both .25, with corrected validities of .60 (n = 40) and .46 (n = 60). When combined with three previous samples (N = 273), meta-analysis yielded a mean validity of .25 (cumulative p = .00003) which provided further evidence of the validity of cognitive ability tests in selection of psychiatric aides.

Quantifying the effects of psychological interventions on employee job performance and work-force productivity.

Quantification of the economic impact of psychological programs in organizations requires determination of (a) the size and variability of the resulting increase in job performance, and (b) the economic value of the increase in job performance. The new methods ofmeta-analysis allow attainment of the first of these, and in relation to the second, utility analysis methods enable us to translate job performance increases into estimates of the economic value of the program. In the area of personnel selection, many meta-analytic studies have already been conducted, resulting in precise and genemlizable estimates of the validity of cognitive ability tests and other selection procedures. Utility analyses show that the job performance increases resulting from use of valid selection methods have substantial economic value. Valid selection produces major increases in work-force productivity. State-of-the-art meta-analyses have not yet been carried out for nonselection interventions such as training or motivational programs. Utility analysis of the results of existing reviews suggest, however, that the economic value of many such programs will prove to be large. The combined effects of selection and nonselection interventions can be expected to produce increases in workforce productivity that are large indeed. Applied psychologists have conducted research on variety of organizational interventions aimed at increasing employee job performance and productivity (Katzell & Guzzo, 1983). The usefulness of this research for business and government has often been bounded by two constraints: (a) the extent to which findings can be made definitive, and (b) the extent to which the impact of findings can be stated in administratively and economically meaningful terms. To render findings definitive, one must reconcile the apparently conflicting results of different studies. To assess the practical impact of findings, one must translate such arcane psychological jargon as p < .01 into economically meaningful statements such as a 10% increase in output or a reduction of $100 million in labor costs. Recent advances have been made in both areas under the rubrics meta-analysis and utility analysis. This article summarizes, in broad outline, the application of these techniques to the areas of personnel selection and organizational interventions. Meta-analysis is collection of techniques for quantitatively cumulating results across studies (Glass, McGaw, & Smith, 1981; Hunter, Schmidt, & Jackson, 1982). Meta-analysis has shown that in many areas, there is no real conflict between the results of different studies; the apparent differences are due to sampling error and other artifacts. A stateof-the-art meta-analysis allows the reviewer to correct for the effects of several artifacts that distort findings in individual studies: sampling error, error of measurement, and restriction in range. A review of meta-analysis methods for correlation coefficients and for effect sizes can be found in Hunter et al. (1982). Utility analysis is the assessment of the economic or social impact of organizational programs (Katzell and Guzzo, 1983). A key problem in psychological research is that impact is usually measured on psychological rather than economic scales. For example, job performance is usually measured by supervisor ratings. Thus special analysis is needed to translate findings into economically meaningful terms, such as dollars of labor savings. In personnel selection, formulas for assessing utility have been available for over 30 years (Brogden, 1949; Cronbach & Gleser, 1965) but have not generally been applied because one parameter in these formulas (the standard deviation of job performance in dollar terms) has been difficult to estimate, Base' line formulas for estimating this parameter now exist (Hunter & Schmidt, 1982; Schmidt, Hunter, McKenzie, & Muldrow, 1979; Schmidt & Hunter, Note 1). Furthermore, the methods of utility analysis have now been extended to the assessment of nonselection organizational interventions such as trainApril 1983 • American Psychologist Copyright 1983 by the American Psychological Association, Inc. 473 ing or performance incentive programs (Schmidt, Hunter, & Pearlman, 1982).