Correction For Range Restriction Research Articles

A longitudinal meta-analysis of range restriction estimates and general mental ability validity coefficients: Better addressing overcorrection amid decline effects.

Psychometric corrections can be crucial for obtaining valid operational results, but concerns are rising about potential overcorrections for general mental ability (GMA) validity coefficients. Our two-part study identifies a source of overprediction: using national norms rather than recent local applicant pool variance for range restriction corrections. Study 1 demonstrates increasing homogeneity in Wonderlic occupational applicant pool variance across four data time waves, suggesting they are no longer interchangeable with the general working population, a divergence attributable to a rise in education. Study 2 employs the Morris meta-analytic approach to gauge the impact of using national norms over occupational ones in range restriction. An analysis of 649 GMA validity coefficients from four time waves of General Aptitude Test Battery and Wonderlic data shows a radical drop in corrected and uncorrected correlations, indicating that historical corrected GMA validity coefficients differ from contemporary ones by up to 16-fold (i.e., an R² of 42.3% vs. 2.6%), and range restriction corrections are now minimal in about 75% of cases. This drop in correlations appears due to the filtering effects of increased education, both due to the demands of the knowledge economy and credentialism, where organizations are using college or university degrees as a proxy for GMA. Credentialism is an incredibly inefficient form of GMA assessment, suggesting an urgent societal need to incorporate selection fundamentals more broadly. Altogether, these results indicate that labor market dynamics have a deeper impact on personnel selection than typically appreciated, meaning that many of our estimates have and will eventually age out. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The need for “Considered Estimation” versus “Conservative Estimation” when ranking or comparing predictors of job performance

AbstractA recent attempt to generate an updated ranking for the operational validity of 25 selection procedures, using a process labeled “conservative estimation” (Sackett et al., 2022), is flawed and misleading. When conservative estimation's treatment of range restriction (RR) is used, it is unclear if reported validity differences among predictors reflect (i) true differences, (ii) differential degrees of RR (different u values), (iii) differential correction for RR (no RR correction vs. RR correction), or (iv) some combination of these factors. We demonstrate that this creates bias and introduces confounds when ranking (or comparing) selection procedures. Second, the list of selection procedures being directly compared includes both predictor methods and predictor constructs, in spite of the substantial effect construct saturation has on validity estimates (e.g., Arthur & Villado, 2008). This causes additional confounds that cloud comparative interpretations. Based on these, and other, concerns we outline an alternative, “considered estimation” strategy when comparing predictors of job performance. Basic tenets include using RR corrections in the same manner for all predictors, parsing validities of selection methods by constructs, applying the logic beyond validities (e.g., ds), thoughtful reconsideration of prior meta‐analyses, considering sensitivity analyses, and accounting for nonindependence across studies.

Insights from an updated personnel selection meta-analytic matrix: Revisiting general mental ability tests' role in the validity-diversity trade-off.

General mental ability (GMA) tests have long been at the heart of the validity-diversity trade-off, with conventional wisdom being that reducing their weight in personnel selection can improve adverse impact, but that this results in steep costs to criterion-related validity. However, Sackett et al. (2022) revealed that the criterion-related validity of GMA tests has been considerably overestimated due to inappropriate range restriction corrections. Thus, we revisit the role of GMA tests in the validity-diversity trade-off using an updated meta-analytic correlation matrix of the relationships six selection methods (biodata, GMA tests, conscientiousness tests, structured interviews, integrity tests, and situational judgment tests) have with job performance, along with their Black-White mean differences. Our results lead to the conclusion that excluding GMA tests generally has little to no effect on validity, but substantially decreases adverse impact. Contrary to popular belief, GMA tests are not a driving factor in the validity-diversity trade-off. This does not fully resolve the validity-diversity trade-off, though: Our results show there is still some validity reduction required to get to an adverse impact ratio of .80, although the validity reduction is less than previously thought. Instead, it shows that the validity-diversity trade-off conversation should shift from the role of GMA tests to that of other selection methods. The present study also addresses which selection methods now emerge as most valid and whether composites of selection methods can result in validities similar to those expected prior to Sackett et al. (2022). (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023).

Oh et al. (2023) question a number of choices made in our article (Sackett et al., 2022); here we respond. They interpret our article as recommending against correcting for range restriction in general in concurrent validation studies; yet, we emphasize that we endorse correction when one has access to the information needed to do so. Our focus was on making range restriction corrections when conducting meta-analyses, where it is common for primary studies to be silent as to the prior basis for selection of the employees later participating in the concurrent validation study. As such, the applicant pool information needed for correction is typically not available. Sackett et al. (2022) highlighted that in many situations, range restriction will be small; so, the inability to correct for it results in only a modest underestimate of validity. Oh et al. mention settings that would result in substantial range restriction; here, we present our rationale as to why we view such settings as uncommon rather than as making up the bulk of the studies contributing to meta-analyses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors

AbstractSackett et al. (2022) identified previously unnoticed flaws in the way range restriction corrections have been applied in prior meta-analyses of personnel selection tools. They offered revised estimates of operational validity, which are often quite different from the prior estimates. The present paper attempts to draw out the applied implications of that work. We aim to a) present a conceptual overview of the critique of prior approaches to correction, b) outline the implications of this new perspective for the relative validity of different predictors and for the tradeoff between validity and diversity in selection system design, c) highlight the need to attend to variability in meta-analytic validity estimates, rather than just the mean, d) summarize reactions encountered to date to Sackett et al., and e) offer a series of recommendations regarding how to go about correcting validity estimates for unreliability in the criterion and for range restriction in applied work.

Challenging conclusions about predictive bias against Hispanic test takers in personnel selection.

Berry et al. (2020) noted that predictive bias is a function of three factors: subgroup mean difference on the predictor (dx), subgroup mean difference on the criterion (dy), and test validity (rxy). They used meta-analytic estimates of each of these three to examine predictive bias against Hispanic test takers when cognitive tests are used in personnel selection. They found that tests underpredict Hispanic job performance by an average of .21 SDs, which would call into question the fairness of cognitive test use in personnel selection. We located 119 studies in which all three parameters-dy, dx, and rxy-could be obtained, thus holding sample, setting, and operationalization constant in estimating the three parameters within each study. This produced a substantially different conclusion: We find that tests overpredict Hispanic performance by .04-.20 SDs, depending on assumptions made about artifact corrections. Factors contributing to differences between the two studies include differences in range restriction corrections, sample incomparability, and Berry et al.'s use of rxy estimated from the total sample rather than within the majority subgroup. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range.

This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor-criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic conclusions produces revised validity estimates. Key findings are that most of the same selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10-.20 points. Structured interviews emerged as the top-ranked selection procedure. We also pair validity estimates with information about mean Black-White subgroup differences per selection procedure, providing information about validity-diversity tradeoffs. We conclude that our selection procedures remain useful, but selection predictor-criterion relationships are considerably lower than previously thought. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Bias in Observed Validity Estimates When Using Multiple Valid Predictors

ABSTRACT Simulated data, validity reports and a firefighter predictive validation study are used to examine validity bias created by three common selection problems-range restriction, applicant and incumbent attrition, and nonlinearity created by compression of high selection test scores. Top 20% selection samples drawn from an applicant pool with known validity coefficients demonstrate that the sample validity estimates of the three predictors are differentially biased in both magnitude and direction, depending on the selection strategy used. Concurrent validity designs generally favor novel predictors. Corrections for direct range restriction across situations were mostly ineffectual. With proper scaling, corrections for indirect range restriction are accurate, but cross-variable biasing effects can occur when score distributions of the individual predictors differ. Many of the biases found in the simulation results are demonstrated in a firefighter predictive validation study where variations of Pearson-Thorndike range corrected validities and a full information maximum likelihood (FIML), approaches are all compared as validity assessments. With normalized predictors, both Pearson and FIML methods show that a test of general mental ability and physically demanding job tasks predicted firefighter performance throughout the 30-year study, with no evidence of interactions or a leveling of performance at high test scores.

Accuracy of Range Restriction Correction with Multiple Imputation in Small and Moderate Samples: A Simulation Study.

Accuracy of Range Restriction Correction with Multiple Imputation in Small and Moderate Samples: A Simulation Study.

Not Restricted to Selection Research: Accounting for Indirect Range Restriction in Organizational Research

Range restriction is a common problem in organizational research and is an important statistical artifact to correct for in meta-analysis. Historically, researchers have had to rely on range-restriction corrections that only make use of range-restriction information for one variable, but it is not uncommon for researchers to have such information for both variables in a correlation (e.g., when studying the correlation between two predictor variables). Existing meta-analytic methods incorporating bivariate range-restriction corrections overlook their unique implications for estimating the sampling variance of corrected correlations and for accurately assigning weights to studies in individual-correction meta-analyses. We introduce new methods for computing individual-correction and artifact-distribution meta-analyses using the bivariate indirect range restriction (BVIRR; “Case V”) correction and describe improved methods for applying BVIRR corrections that substantially reduce bias in parameter estimation. We illustrate the effectiveness of these methods in a large-scale simulation and in meta-analyses of expatriate data. We provide R code to implement the methods described in this article; more comprehensive and robust functions for applying these methods are available in the psychmeta package for R.

ON SOME METHODOLOGICAL ASPECTS OF THE STUDY OF HUMAN INDIVIDUALITY

Human individuality, presented on different levels (from biological to social ones), is of a high interest in Russian psychology, and the method of correlation design is widely used among researches, because it allows revealing relationships between multi-level properties of individuality. The present article examines several methodical aspects of the correlation analysis implementation, discussing problems and possible solutions. In particular, it considers the issue of nonlinear dependencies (parabolic, hyperbolic etc.), which are impossible to reveal by common correlation methods, but which can be uncovered by using nonlinear correlations, such as correlation index, correlation ratio, maximal information coefficient, distance correlation, maximal correlation, “partial moments” method. Furthermore, it considers the necessity of visualizing variables correlation (scatterplots) that enables to reveal hidden data structures, for example, subgroups. Special attention is paid to correlations corrections for restriction of range and related difficulties that are well-known, but scarcely researched in Russian psychology. In process of investigating plentiful pairwise correlations between individuality properties on different levels it is important to consider anissue of multiple comparisons, which, however, is rarely taken into the account by researches, leading to false results in many occasions. Moreover, the article examines robust statistical methods, particularly permutation tests and bootstrap. These methods combine robustness and high power. Finally, the study observes such issues as the completeness of results presentation and current debates about significance level, effect size and confidence intervals, reproducibility of psychological researches, and meta-analysis approach. Significance level has often been criticized; interval estimates and effect size were supposed to replace it. However, the problem of Null Hypothesis Significance Testing (NHST) has not been completely solved yet. A possible solution is presentation of complete data on research results including precise significance level, confidence intervals, effect size and etc. These estimations can be then applied in meta-analysis, which allows moving on to a new level of scientific generalizations.

Effects of range restriction and criterion contamination on differential validity of the SAT by race/ethnicity and sex.

We illustrate the effects of range restriction and a form of criterion contamination (individual differences in course-taking patterns) on the validity of SAT scores for predicting college academic performance. College data facilitate exploration of differential validity's determinants because they (a) permit the use multivariate range-restriction corrections to more accurately account for differential range restriction across subgroups and (b) allow for separate examinations of composite performance and specific performance episodes, the latter of which controls for ecological contamination of composite performance due to individuals' choices of performance opportunities. Using data from 363,004 students at 107 U.S. institutions, we found that controlling for course-taking patterns resulted in validity coefficients that were appreciably larger than predictors' correlations with obtained grade point averages (GPAs). The validities of SAT scores for predicting the first-year college performance of Black and Hispanic students were not significantly different from the validity for White students after correcting for both course-taking patterns and differential range restriction, but significant Black-White differences were detected for predicting 4-year cumulative performance. Validity estimates for predicting both first-year and 4-year cumulative performance were significantly smaller among Asian students than White students after making these corrections. The SAT's observed validity for predicting college GPAs was substantially lower for males than females and, unexpectedly, controlling for course-taking patterns increased male-female validity differences. Implications for personnel selection research are discussed. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Selection and academic success of medical students in Hamburg, Germany

BackgroundStudent selection at Hamburg medical school is based on the combination of a natural science knowledge test (HAM-Nat) and pre-university educational attainment.MethodOf the 1565 medical students enrolled in Hamburg from 2012 to 2015 about half were admitted by an entrance test, half by quotas. First, we analysed sociodemographic determinants of entrance test performance. Then, we used regression analysis to describe the interplay of variables in the prediction of study outcome, the role of sociodemographic factors, and differences in the calibration of educational attainment specific to German federal states.ResultsBetter performance in the entrance test was associated with age over 21, male gender, German nationality, first language German and both parents holding an academic degree – effect sizes were small. No differences were found for the birthplaces of parents (a proxy for migration background). Study outcome differed considerably among admission paths: Students admitted by entrance test or the quota for excellent pre-university educational attainment performed markedly better during the first 3 terms than students admitted by the waiting list quota and the quota for foreign students. Gender differences in study outcome were slight with better performance by males. The relation of pre-university educational attainment to study outcome was moderated by the federal state in which secondary schooling took place. Methods for the equating of state-specific grades are explored. The predictive validity of the HAM-Nat after correction for range restriction was r = .31. The relatively low value of this coefficient may be attributed to 3 factors: 1. self-selection of applicants which leads to a validity-enhancing effect that is not revealed by the predictor-outcome correlation, 2. reduction of variance due to a high selection ratio, and 3. high test difficulty, exceeding the demands of the medical curriculum.ConclusionThe HAM-Nat achieves a small amount of incremental validity over pre-university educational attainment. This effect, obtained from correlational analysis, underestimates the validity of the test, because it does not reflect the role of self-selection and other validity enhancing features of the selection process.

Predicting Postprobationary Job Performance of Police Officers Using CPI and MMPI–2–RF Test Data Obtained During Preemployment Psychological Screening

ABSTRACTWe examined associations between prehire California Psychological Inventory (CPI) and prorated Minnesota Multiphasic Personality Inventory–2 Restructured Form (MMPI–2–RF) scores (calculated from MMPI profiles) and supervisor ratings for a sample of 143 male police officers. Substantive scale scores in this sample were meaningfully lower than those obtained by the tests’ normative samples in the case of the MMPI–2–RF and meaningfully higher in the case of the CPI (indicating less psychological dysfunction). Test scores from both instruments showed substantial range restriction, consistent with those produced by members of the police candidate comparison groups (Corey & Ben-Porath, 2014; Roberts & Johnson, 2001). After applying a statistical correction for range restriction, we found a number of meaningful associations between both CPI and MMPI–2–RF substantive scale scores and supervisor ratings. For the MMPI–2–RF, findings for scales from the emotional dysfunction and interpersonal functioning domains of the test were particularly strong. For the CPI, findings for scales indicating conformity with social norms, integrity, and tolerance were strong, as were the findings for an index indicating risk of termination. Hierarchical regression analyses showed that MMPI–2–RF and CPI scores complement each other, accounting for incremental variance in the prediction of job-related variables over and above each other. Implications of these findings for assessment science and practice are discussed.

Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity.

Separate meta-analyses of the cognitive ability and assessment center (AC) literatures report higher criterion-related validity for cognitive ability tests in predicting job performance. We instead focus on 17 samples in which both AC and ability scores are obtained for the same examinees and used to predict the same criterion. Thus, we control for differences in job type and in criteria that may have affected prior conclusions. In contrast to Schmidt and Hunter's (1998) meta-analysis, reporting mean validity of .51 for ability and .37 for ACs, we found using random-effects models mean validity of .22 for ability and .44 for ACs using comparable corrections for range restriction and measurement error in the criterion. We posit that 2 factors contribute to the differences in findings: (a) ACs being used on populations already restricted on cognitive ability and (b) the use of less cognitively loaded criteria in AC validation research. (PsycINFO Database Record

Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research.

Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent-based approach vis-à-vis the applicant-based approach. To address this issue, we conducted several Monte Carlo simulations, as well as an analysis of college admissions data. Our first simulation study showed that incumbent-based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant-based corrections were quite accurate. Our second set of simulations showed that incumbent-based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant-based corrections tended to accurately estimate population parameters and showed little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent-based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent-based range restriction corrections in lieu of applicant-based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance. (PsycINFO Database Record

Realizing the full potential of psychometric meta-analysis for a cumulative science and practice of human resource management

This might be the most opportune time for Human Resource Management (HRM) to benefit from psychometric meta-analysis. Explosion of empirical research, often with conflicting results, hide important takeaways that can guide evidence-based applications of HRM. The science of HRM can turn to meta-analyses and meta-analytic thinking as the antidote to the so-called replication crisis afflicting social sciences in general. In this paper, we focus on issues and potential problems that may threaten the veracity and usefulness of contemporary meta-analyses in HRM. We contend that these problems must be correctly tackled for meta-analyses to realize their full potential in advancing HRM science and practice. We address the problems of identification and inclusion of all relevant effect sizes, as well as appropriate corrections for unreliability and range restriction. We offer concrete proposals to enable inclusion of unpublished, practitioner research and data in HRM meta-analyses.

The Predictive Validity of a Two-Step Selection Process to Medical Schools

Background. A two-step selection process, consisting of cognitive and noncognitive measures, is common in medical school admissions. Objective. To estimate the validity of this process in predicting academic performance, taking into account the complex and pervasive effect of range restriction in this context. Methods. The estimation of the validity of the two-step process included a sequential correction for range restriction and an estimation of the predictive validity of the process in its entirety. Data were collected from 1,002 undergraduate students from four cohorts (2006/07–2009/10) at three medical schools in Israel. Results. The predictive validity of the composite of the cognitive measures with respect to Year 1 performance was high, resulting entirely from the predictive validity of the admission test (a standard measure of ability). The predictive validity of the noncognitive measure was moderate. The predictive validity of the process in its entirety was high, its value dependent on the weights given to the cognitive and noncognitive measures. Conclusion. A cognitive admission test has a high predictive validity with respect to Year 1 performance. The addition of a noncognitive measure in the second step does not markedly diminish the predictive validity of the selection process with respect to academic achievement.

Correction for Range Restriction in Meta‐Analysis Revisited: Improvements and Implications for Organizational Research

In this study, we present a more accurate method for correcting for range restriction (Case V) that expands upon Bryant and Gokhale's (1972) method. We further present detailed steps to incorporate the Case V method into Schmidt and Hunter's (2015) psychometric meta‐analysis methods (both individual correction and artifact distribution approaches). We then evaluate the accuracy of the Case V method vis‐à‐vis existing methods. Monte‐Carlo simulation results indicate that the Case V method provides very accurate estimates for the mean true score correlation and reasonably accurate estimates for the true standard deviation. More important, Case V almost always provides more accurate results than alternative methods (particularly, Case IV). To illustrate how the Case V method works with real data, we conduct a reanalysis of Judge, Heller, and Mount's (2002) meta‐analysis examining the relationships between the Big 5 personality traits and job satisfaction. Results indicate that the true score correlations between the Big 5 traits and job satisfaction have been underestimated, whereas their true standard deviations have been overestimated. Implications for range restriction corrections in organizational research are discussed.

Imperfect Corrections or Correct Imperfections? Psychometric Corrections in Meta-Analysis

There is understandable concern by LeBreton, Scherer, and James (2014) that psychometric corrections in organizational research are nothing more than a form of statistical hydraulics. Statistical corrections for measurement error variance and range restriction might inappropriately ratchet observed effects upward into regions of practical significance and publication glory—at the expense of highly questionable results.