Criterion-related Validity Study Research Articles

Incremental Validity of Locus of Control After Controlling for Cognitive Ability and Conscientiousness

This research examined the criterion-related validity of work-specific locus of control in predicting job performance, including incremental validity after controlling for cognitive ability and conscientiousness. Data from a student sample and from a large employee sample were used to evaluate the scale properties of measures of locus of control, conscientiousness, and cognitive ability. Two concurrent criterion-related validation studies were then conducted to evaluate the incremental validity of locus of control. In both validation studies, locus of control demonstrated overall and incremental relationships with performance after controlling for ability and conscientiousness, such that employees with higher internal locus of control performed more effectively than externals.

Criterion-related validity, self-other agreement, and longitudinal analyses for the Iowa personality questionnaire: A short alternative to the MPQ

The 42-item Iowa Personality Questionnaire (IPQ) was designed to briefly assess the same dimensions of personality measured by the Multidimensional Personality Questionnaire (MPQ; Tellegen, 1982). Results from three studies indicate that the IPQ can be used as a reasonable alternative to the MPQ when time constraints place limits on the length of questionnaires. For example, correlations between self-reports on the IPQ and the MPQ were .67 for Constraint, .75 for Negative Emotionality, and .77 for Positive Emotionality and the IPQ and MPQ demonstrate similar criterion-related validity (Study 1). Importantly, Study 3 illustrates how the IPQ can be used to address the substantive issue of the rank-order consistency of personality traits across time. Specifically, longitudinal results indicated that scores on the MPQ assessed in 1994 forecast scores on the IPQ in 2001. These findings provide further evidence for the continuity of individual differences across time.

The PASS theory of intelligence and the acquisition of a complex skill: a criterion-related validation study of Cognitive Assessment System scores

This study extends the empirical research on the planning, attention, simultaneous, and successive (PASS) theory of intelligence and its assessment via the Cognitive Assessment System (CAS) by examining the criterion-related validity of CAS scores with respect to the acquisition of a complex skill. Accordingly, 92 young adult males completed 7 h of training on a computer-based task that simulated the demands of a dynamic aviation environment. Cognitive and skill-based learning were assessed. Although the CAS Full score was correlated with learning, there were differences in the predictive value of the CAS scales. Successive processing was the only CAS scale to explain a unique proportion of variance in all the learning measures. Scores on simultaneous processing made important contributions to cognitive learning, and attention scores were an important contributor to skill-based learning. Planning scores were not significantly related to any of the learning criteria.

Correlation of clinical examination characteristics with three sources of chronic low back pain

Background context Research has demonstrated some progress in using a clinical examination to predict discogenic or sacroiliac (SI) joint sources of pain. No clear predictors of symptomatic lumbar zygapophysial joints have yet been demonstrated. Purpose To identify significant components of a clinical examination that are associated with symptomatic lumbar discs, zygapophysial joints and SI joints. Study design A prospective, criterion-related concurrent validity study performed at a private radiology practice specializing in spinal diagnostics. Patient sample The sample consisted of 81 patients with chronic lumbopelvic pain referred for diagnostic injections. Outcome measures Contingency tables were constructed for nine features of the clinical evaluation compared with the results of diagnostic injections. Statistical analysis included chi-squared test for independence, phi and odds ratios with confidence intervals. Method Patients received blinded clinical examinations by physical therapists, and diagnostic injections were used as the criterion standard. Results Significant relationships were found between discogenic pain and centralization of pain during repeated movement testing, and pain when rising from sitting. Lumbar zygapophysial joint pain was associated with absence of pain when rising from sitting. Sacroiliac joint pain was related to three or more positive pain provocation tests, pain when rising from sitting, unilateral pain and absence of lumbar pain. Conclusions Significant correlations exist between clinical examination findings and symptomatic lumbar discs, zygapophysial and SI joints. The strongest relationships were seen between SI joint pain and three or more positive pain provocation tests, centralization of pain for symptomatic discs and absence of pain when rising from sitting for symptomatic lumbar zygapophysial joints.

Measuring Inter-rater Reliability of the Sequenced Performance Inventory and Reflective Assessment of Learning (SPIRAL)

To measure the inter-rater reliability of the Sequenced Performance Inventory and Reflective Assessment of Learning (SPIRAL), a twenty-three item scoring rubric designed to assess first and second-year students' competencies such as "acquisition of knowledge," "peer teaching and communication skills," and "professional behavior" in a problem-based learning (PBL) curriculum at the University of North Dakota School of Medicine and Health Sciences. In 2001, the authors constructed a 69-item multiple-choice questionnaire consisting of descriptions of "prototypical" (representing real students) PBL student performances. For each of the 23 SPIRAL items, the authors chose competency descriptions of an "emerging," a "developing," and an "advanced" student (69 prototypical students). These descriptions were obtained from narrative comments by PBL facilitators of real students in their classes in 2000-2001. Seventeen experienced facilitators (PBL tutors) were subsequently asked to rank the 69 prototypical students based on SPIRAL anchor points in the three competency levels. Kendall's coefficient of concordance was used to test inter-rater reliability. Modes were also determined to illustrate the extent to which the ratings of the 17 facilitators aligned with the ratings anticipated by the authors. Overall, inter-rater scoring for all items combined was considered to be reliable (Kendall's coefficient of concordance W =.75). Inter-rater scoring was also determined for each of the 23 SPIRAL items, with Ws ranging from.97 to.53. Facilitators rated students according to 55 of the 69 SPIRAL descriptors as the researchers anticipated. The scoring rubric is reliable. Further criterion-related validation study is warranted.

Comparison of the Dickson glazer sexual functioning inventory (DGSFI) to a standardised clinical interview in assessing sexual functioning in atypical-treated patients: A criterion-related validity study

Comparison of the Dickson glazer sexual functioning inventory (DGSFI) to a standardised clinical interview in assessing sexual functioning in atypical-treated patients: A criterion-related validity study

Predicting Adaptive Performance: Further Tests of a Model of Adaptability

This research further investigates an 8-dimension taxonomy of adaptive performance developed by Pulakos, Arad, Donovan, and Plamondon (2000) and the usefulness of past experience, interest, and self-efficacy predictors developed to measure these same 8 dimensions for predicting adaptive job performance. Participants in the concurrent, criterion-related validation study included 739 military personnel. They completed the 3 adaptability measures as well as more traditional cognitive and noncognitive predictors. Supervisors of the study participants rated their adaptive job performance. Results showed support for the 8-dimension model of adaptability. In addition, cognitive ability, personality, and the new adaptability predictors were shown to predict adaptive performance, with some past experience items adding incremental validity beyond the more traditional cognitive ability and personality measures. Results are discussed.

An Empirically Based Selection and Evaluation System for Collegiate Football

Using a content validity approach, including job analysis, a model was developed for the selection and appraisal of college football players. Evaluation instruments were developed for offensive lineman, quarterback, running back, receiver, defensive lineman, linebacker, and defensive secondary. Criterion related validity studies were then conducted by sending the evaluation instruments to a nation-wide sample of college football coaches who evaluated players at each position. Criterion data (e.g., passing and rushing performance statistics) were collected on the evaluated players. Multiple linear regression analyses conducted by position revealed numerous significant relationships between the predictor variables and the criterion measures. The usefulness of the evaluation instruments was enhanced by developing a simple and cost effective model that generated performance predictions for each position. Strengths and weaknesses of the present system were discussed as well as the role that sport management may play in such an endeavor.

A criterion-related validity study of the nursing-care dependency (NCD) scale

The purpose of this study was to examine some aspects of the criterion-related validity of the Nursing-Care Dependency (NCD) scale. This 15-item counting scale has recently been developed for assessing the care dependency of demented or mentally handicapped in-patients. Its criterion-related validity was investigated by studying the relationship between the Nursing-Care Dependency scale, the Rating Scale for Elderly Patients (RSEP), the Behavior Observation Scale for Intramural Psychogeriatrics (BOSIP) and the Scale for Social Functioning (SSF). Data were collected from 322 demented and 105 mentally handicapped patients using the mentioned instruments. High correlations were found between NCD and RSEP, and NCD and SSF. There was a low relationship between the NCD sumscore and BOSIP subscales-scores. The NCD was able to purposefully distinguish diagnostic groups of demented patients when an external criterion was used.

The “shelflife” of a test validation study: A survey of expert opinion

Anonymous mailed surveys containing a description of a competent and successful content validation or criterion-related validation study were returned by 106 experts in employment test validation: Fellows of SIOP, ABPP Diplomates in I/O Psychology, and experienced practitioners nominated by the first two groups. More than 70% replied that they would seek additional supportive evidence within 5 years or less following the validation. The most frequently suggested action was a review of the target job requirements. Opinions did not differ as a function of demographic and experiential variables—except between those who have been involved in Title VII employment discrimination litigation and those who have not, and between those who have primarily represented employers vs those who have represented plaintiffs more frequently in such litigation. Whereas decrements to the original supporting evidence would lead respondents to shorten their estimates of longevity, a surplus of supportive evidence would not lead to extended estimates. Generalizability and limitations of the survey are discussed as well as implications for Title VII litigation.

The Short Geriatric Depression Scale

This study was undertaken to determine the test-retest reliability and the classification congruence between two popular depression screening instruments, the Short Geriatric Depression Scale (S-GDS) and the Geriatric Depression Scale (GDS). The GDS was administered to 73 independent, community dwelling older adults (mean ± SD : 62 ± 5, range 55-75 years) ; the S-GDS was extracted frorn the GDS for comparison. A subset of 49 subjects was reassessed 8 weeks after the initial assessment. There was systematic classification disagreement between the two instruments. Thus, of 15 cases categorized as depressed by the GDS, 9 (60%) were categorized as not depressed by the S-GDS. Of 7 cases categorized as depressed by the S-GDS, 1 (14%) was categorized as not depressed by the GDS. Test-retest reliability was high for the GDS (r =.85), but only moderate for the S-GDS (r =.67). The current findings do not support the use of the S-GDS as a surrogate for the GDS. In light of the relatively extensive validation for the GDS and the lack of same for the S-GDS, as well as the demonstrated superior reliability of the former, prudence requires that the S-GDS not be used as a depression screening instrument pending the outcome of criterion-related validation studies.

Seven Criterion-Related Validity Studies Conducted with the National Police Officer Selection Test

This is a summary of seven criterion-related validation studies conducted in the United States with The National Police Officer Selection Test (POST), a content-valid, entry-level, basic skills examination used by law enforcement agencies for preemployment screening. Correlations between POST scores and various training and work performance criteria are presented. The results provide additional evidence for the criterion-related validity of the POST as a preemployment selection instrument for law enforcement candidates.

Why Meta-Analysis Doesn’t Tell Us What the Data Really Mean: Distinguishing between Moderator Effects and Moderator Processes

Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (a) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities ofpeer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.

Potential sources of criterion bias in supervisor ratings used for test validation

Four possible sources ofcriterion contamination were investigated in the supervisory performance ratings used for a predictive criterion-related validation study. Supervisors' liking for subordinates had a very large association with their performance ratings independent of the effects of employee ability. Also as hypothesized, expectations of employee qualifications were correlated significantly with initial (1-mo.) performance ratings but not with ratings made after 5 mos. Ethnicity was not associated with 1-mo. performance ratings, but after five months supervisors gave significantly higher ratings to subordinates of the same ethnic group as themselves. No evidence was found of sex bias in the ratings. Estimates of test validity were reduced substantially when the potential sources of criterion bias were controlled statistically. The data are interpreted in the contexts of construct relevance for ratings criteria, possible spurious inflation of employment test validities, and the developmental processes by which supervisor-subordinate relationships are established in the first few months of employment.

Why meta-analysis doesn't tell us what the data really mean: distinguishing between moderator effects and moderator processes

Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (r i) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and n i. We investigate this possibility on studies examining criterion-related validities of peer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between r i and n i precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.

Preemployment Physical Evaluation

There is a growing trend toward using preemployment tests to select employees for physically demanding jobs. Women are, in increasing numbers, entering physically demanding occupations that were traditionally dominated by men. Under current Federal employment law, it is illegal to disqualify an employee for a job because of race, color, religion, sex, national origin, and with the recent passage of the American Disabilities Act (ADA), handicap. Because of gender differences in strength, body composition, and VO2max, preemployment tests for physically demanding jobs tend to screen out more females than males. Employers are using preemployment tests not only to enhance worker productivity, but also to minimize the threat of litigation for discriminatory hiring practices and to reduce the risk of musculoskeletal injuries. The primary ergonomic methods used in industry to reduce the risk of back injuries are preemployment testing and job redesign. When a test results in adverse impact, the validity of the test must be established. Validity in this context means that the test represents or predicts the applicant's capacity to perform the job. Criterion-related, content, and construct validation studies are the means used to establish validity. The validity of preemployment hiring practices for physically demanding jobs has been decided in the courts. The most common reason for ruling an employment practice invalid is the failure to show that the test measured important job behaviors. Much of this litigation has involved height and weight requirements for public safety jobs. The courts have generally ruled that using height and weight standards as a criteria for employment is illegal because they were not job related. If fitness tests comprise part or all of the preemployment test, it is essential to demonstrate that the fitness component is related to job performance. Although there are many factors to consider when establishing a cut score, there is a growing trend toward establishing the cut score on the basis of the job's physical demands, defined by VO2max and strength. This literature is limited because most validation studies are not published. They more typically take the form of a technical report to the governmental agency or company that funded the project. There are published preemployment validation studies for outdoor telephone craft jobs involving pole-climbing tasks; firefighters; highway patrol officers; steel workers; underground coal miners; chemical plant workers; electrical transmission lineworkers; and various military jobs.

The validity of employment interviews: A comprehensive review and meta-analysis.

This meta-analytic review presents the findings of a project investigating the validity of the employment interview. Analyses are based on 245 coefficients derived from 86,311 individuals. Results show that interview validity depends on the content of the interview (situational, job related, or psychological), how the interview is conducted (structured vs. unstructured; board vs. individual), and the nature of the criterion (job performance, training performance, and tenure; research or administrative ratings). Situational interviews had higher validity than did job-related interviews, which, in turn, had higher validity than did psychologically based interviews. Structured interviews were found to have higher validity than unstructured interviews. Interviews showed similar validity for job performance and training performance criteria, but validity for the tenure criteria was lower. The interview is a selection procedure designed to predict future job performance on the basis of applicants' oral responses to oral inquiries. Interviews are one of the most frequently used selection procedures, perhaps because of their intuitive appeal for hiring authorities. Ulrich and Trumbo (1965) surveyed 852 organizations and found that 99% of them used interviews as a selection tool. Managers and personnel officials, especially those who are interviewers, tend to believe that the interview is valid for predicting future job performance. In this article, we quantitatively cumulate and summarize research on the criterion-related validity of the employment interview. Our purpose in this article is threefold. First, we summarize past narrative and quantitative reviews of criterion-related validity studies of the employment interview. Second, we report research that extends knowledge of the criterion-related validity of interviews through meta-analyses conducted on a more comprehensive database than has been available to earlier investigators. Third, we examine the criterion-related validity of different categories of interviews that vary in type and in structure.

Validation of the Federal Aviation Administration Air Traffic Control Specialist Pre-Training Screen

Two formal validation studies of the Air Traffic Control Specialist Pre-Training Screen (ATCS/PTS), a 5-day computer-administered test battery, are described. The ATCS/PTS was designed to replace the nine-week U.S. Federal Aviation Administration (FAA) Academy ATCS Nonradar Screen program that served as the second major test in the ATCS selection system. Review of ATCS job analyses suggested that predictor tests should assess cognitive constructs such as spatial reasoning and short-term memory, and require dynamic, concurrent performance. A proposed test battery was developed, consisting of two computer-administered information processing tests and a simplified radar-based air traffic control work sample. In study 1, predictive, criterion-related validation (N = 438) found that the proposed test battery explained additional variability in scores earned in the 9-week FAA Academy program, after taking into account student aptitude. In study 2, criterion-related validation (N = 297) demonstrated that the proposed test battery was as valid as the 9-week FAA Academy ATCS Nonradar Screen for predicting progress in field training. Preliminary data from a third study conducted after validation of the ATCS/PTS seem to suggest that the abilities assessed by the new computerized tests reflect the abilities required on the job. However, implementation of the ATCS/PTS for actual employment decisionmaking in June 1992 was based on results obtained in the second concurrent, criterion-related validation study. The U.S. controller selection system since June 1992 has consisted of the four-hour written ATCS aptitude test battery followed by, for those applicants earning a qualifying score and dependent upon agency manpower requirements, second-level screening on the ATCS/PTS. Additional research requirements as part of an aviation human factors research program are described. © 1993 John Wiley & Sons, Inc.**This article is a US Government work and, as such, is in the public domain in the United States of America.

Predicting negotiation skills

The purpose of this study was to validate a selection process that was designed to predict negotiation skills for a telephone collections job. First, a job analysis was performed (n=343) to identify the major functions and behavioral requirements of the job. Next, the test, a situational judgment inventory, was developed, and a concurrent criterion related validation study was completed (n=249). The criterion measures were supervisory ratings of performance. Validity coefficients ranged from r=.41 (p<.001) to r=.45 (p<.001). In conclusion, the situational judgment inventory was found an effective vehicle for measuring telephone negotiation skills.

Validation of personality constructs for the selection of insurance industry employees

One hundred and nine employees working in clerical jobs in an insurance company participated in a concurrent criterion-related validity study. The employees were administered the Hogan Personality Inventory which consists of six basic scales and six occupational scales. Five criterion measures were collected: a within-department forced quintile distribution by the manager, and managerial ratings of the job factors of job knowledge, interpersonal skills, organization, and overall performance. The twelve test scores were correlated with each of the five criterion measures. The results revealed the Adjustment, Clerical Aptitude and Managerial Potential scales correlated significantly (p<.05) with each of the five criteria, while the Service Orientation scale correlated significantly with three of the five criteria. The results were discussed in the context of using personality scales to forecast job performance in the insurance industry.