Job Knowledge Tests Research Articles

Computer Predictors of Mechanical Job Performance: Marine Corps Findings

For a long time, psychologists have been interested in creating better tests to predict job performance. Recently, there have been attempts to improve the Armed Services Vocational Aptitude Battery (ASVAB), which is used to select and classify enlisted personnel. This research investigates the ability of the Enhanced Computer-Administered Test (ECAT), a computerized battery of new measures, to increment the validity of the ASVAB in predicting three criteria of mechanics' job performance: a hands-on performance test (HOPT), a job knowledge test (JKT), and final course grade (FCG). Examinees included 698 automotive mechanics and 443 helicopter mechanics. One spatial ability test, assembling objects (AO), was the only promising new predictor of mechanical HOPT performance among those we examined. A 0 added the greatest incremental validity to the ASVAB for predicting HOPT scores of both automotive (.012 or 1.6%) and helicopter mechanics (0.15 or 2.2%). When criteria other than hands-on tests were used, increments were much smaller, and they pinpointed different subtests as incrementing the ASVAB. For the prediction of automotive mechanics' JKT scores, sequential memory, a complex memory test designed to be a nonverbal measure of general intelligence, added .003 (0.4%) to the validity of ASVAB. Spatial Reasoning also had an incremental validity of .003 for predicting JKT scores. For prediction of FCG, the best new subtest added essentially nothing. These results indicate that sole reliance on academic criteria, such as JKT or FCG, can result in choice of different predictors or underestimate the incremental validity of new predictor tests against hands-on performance. The incremental validity of A 0 for predicting hands-on mechanical performance is likely to be between .O1 and .02.

The Validity of Job Knowledge Measures

The validity of written job knowledge tests as predictors of job performance and training success was examined. Based on a sample of 363,528 persons and 502 coefficients, meta‐analyses were performed to investigate the extent of validity generalization and the effects of two hypothesized moderator variables: content similarity and job complexity. Corrected mean validities were 0.45 for studies predicting job performance and 0.47 for studies of training success. Support was found for both moderators since validities were higher for high complexity jobs and when job‐test content similarity was high. The implications of this study for personnel selection and research in the area of work performance are discussed.

Does Choice of a Criterion Matter?

For reasons of objectivity and job relevance, a hands-on performance test (HOPT) has been advocated as the best measure to use as a validation criterion. However, the cost of HOPTs has inspired a search for surrogate measures such as job knowledge tests (JKTs). This study questions whether choice of criteria makes a difference for the purpose of developing an accurate selection composite. Hanser's (1990) research using Army Project A data found that use of surrogates in place of a HOPT made little difference for the selection accuracies of predictor composites of the Armed Services Vocational Aptitude Battery (ASVAB). This study uses Marine Corps Job Performance Measurement (JPM) data to replicate Hanser's results. It also questions whether a criterion is needed at all for developing an accurate selection composite. It demonstrates that predictor composites such as Ree and Earles's (1990a) psychometric g and unit-weighted regression coefficients can be as useful as those developed using an outcome criteri...

An interview approach to work sample criterion measurement.

A variety of measurement techniques have been used to assess job proficiency, including hands-on work sample tests, job knowledge tests, and ratings. The current research effort examined the viability of an interview approach to work sample criterion measurement. Hands-on and interview work sample measures were developed and administered to 1,491 enlisted personnel in eight Air Force specialties. Job-level analyses yielded moderate (.46) to high (.84) correlations but produced significant mean differences (six of eight specialties) between the two techniques. In addition, the patterns of relationships between each work sample test and a series of related performance, aptitude, and experience measures were found to be quite similar, and the magnitudes of these correlations were similar to those in studies reported previously

Setting Standards and Diagnosing Training Needs With Surrogate Job Performance Measures

A surrogate measure is an expedient indicator of job performance that is reliable, valid, and yields results similar to a hands-on performance test (HOPT). It is used in place of a HOPT under certain circumstances. This study used U.S. Marine Corps job performance measurement data to analyze the value of six surrogates-field proficiency ratings, training grade point average (GPA), field conduct ratings, job knowledge tests (JKTs), video marksmanship, and supervisor ratings-for hands-on infantry tests. The study first addressed how useful each surrogate would be for setting classifi- cation standards. Virtually identical standards were computed using the JKT and HOPT; therefore, the JKT is the most appropriate surrogate for setting standards. Next, the JKT was analyzed for its usefulness in diagnosing training needs. The JKT had low reliabilities at the duty-area level, limiting its usefulness for determining training requirements.

Validity of Selection/ Classification Variables for Performance in Three Navy Ratings

This study examined speed of advancement and score on a job knowledge test as job performance criteria in the validation of selection and classification variables for three Navy ratings. Longitudinal samples spanning a 4-year period were drawn for men in radioman, machinist's mate, and hospital corpsman ratings who, within their first enlistment, advanced to E-4 by means of the Navy competitive advancement system. Validity coefficients for Navy enlisted selection and classification variables (Armed Services Vocational Aptitude Battery [ASVAB] and education) were computed for personnel in each rating, subgrouped by entry pay grade. Validity coefficients for the job knowledge test were high by historical standards, and the effects were highest for the subgroups with the highest selection standards. Validity coefficients for the advancement criterion were much lower. Advancement, the criterion that appeared to have the greater face validity, was not as good a measure of job performance as the job knowledge t...

Reliability (Generalizability) of Job Performance Measurements: Navy Machinist Mates

The Armed Services have undertaken a Joint Services Project to develop prototype methods for measuring job performance and, if feasible, to link enlistment standards to on-the-job performance. As one step toward meeting the first purpose of the project, our study assessed the reliability of scores on four measures of job performance of machinist mates in the Navy: a hands-on performance test, a paper-and-pencil job knowledge test, job task perform- ance ratings, and global ratings. Generalizability theory was used to estimate the reliability of the measures and to improve the design of each. The generalizability analyses investigated the consistency of performance scores over observers, tasks (equipment operation, casualty control, preventative maintenance), and types of raters (supervisor, peer, and self). Recommenda- tions about the optimal designs of studies for decision making (e.g., pre- dicting hands-on performance scores from the other measures of job perform- ance) were based on the findings.

More Evidence on Relationships Between the Work Environment and Job Performance

The purposes of this research were: (a) to develop a work environment questionnaire appropriate for assessing the situational constraints and facilitators of performance in the US. Army and (b) to evaluate relationships between individual aspects of the work environment and a wide array of performance measures. The Army Work Environment Questionnaire (AWEQ) was developed and adrninistered to 5,080 first-term Army enlisted personnel working in nine jobs. Also available for each member of this sample were peer and supervisor performance ratings, along with scores on job knowledge and work sample task proficiency tests. The results of factor analyses indicated that a five-factor solution provides a parsimonious explanation of the underlying Army work environment. Correlations of about .20 were found: (a) between ratings on some of the performance dimensions and the environmental factors, Job/'Ttlsk Importance and Supervisor Support; and (b) between performance on task proficiency measures and the environmental construct relating to relevance of training for present job assignment. Discussion focuses on the value of maintaining separate environmental constructs when examining environment-performance relationships and addresses issues related to interpreting the work environment questionnaire responses.

Comparative Validities of Two Cognitive Ability Tests in Predicting Work Performance and Training Success of Psychiatric Aides

Two cognitive ability tests were evaluated using 98 state hospital psychiatric aides to determine if the use of both tests would improve the predictive validity. A verbal ability test used in the selection process correlated .20 (p < .05) with performance ratings and .36 (p < .05) with post-training job knowledge test scores. A vocabulary test given for research purposes correlated .18 (p < .05, one-tail test) with ratings and .49 (p < .01) with training scores. While both tests were valid predictors, the vocabulary test was a significantly better predictor of the training criterion. The multiple correlation of both tests with ratings was .20 (p > .05) and with training scores was .49 (p < .01), neither of which was larger than the correlation with the best single predictor. The results demonstrated the potential use of two cognitive ability tests in improving aide selection without increasing the adverse impact on minorities, but continuing research with such tests and other selection procedures appears important, especially in occupational groups where specific validity data are limited and the jobs less complex.

An Analysis of Black–White Differences in Job Performance.

A meta-analyis of Black and White differences in job performance was conducted in order to investigate the magnitude of differences as a function of the type of criterion used (viz., work samples, job knowledge tests, ratings, and objective data) and to search for potential contaminants. Studies referenced by Schmidt and Hunter (1981) were analyzed. The overall

Ability, training performance, and demographic factors in voluntary turnover among psychiatric aides.

This longitudinal study of voluntary turnover among psychiatric aides in a state hospital compared aides who left during the first year of employment with those who stayed on measures of ability, training performance, and demographic variables. No significant differences between stayers and leavers were found on two tests selecting for general ability and a post-training job knowledge test, but significant differences were found on race and sex (higher turnover among white aides and males). The findings were contrary to those of previous studies with professionals, which indicated that leavers were higher in ability, qualifications, and training. Further research appears needed to assess the relationships between turnover and measures of ability and performance potential as well as research to evaluate the turnover and work performance relationship.

ESTIMATION OF THE ADVERSE IMPACT OF A POLICE PROMOTION EXAMINATION

Sixty‐five male sergeants in the U.S. Park Police, Department of Interior, competed for promotion to lieutenant in the fall of 1978. Candidates were rank‐ordered based on a weighted sum of a written job knowledge test score and an oral interview fitness and merit ranking. Five promotions would be made based upon the rank order; the top nine candidates were white. However, there was no difference in mean scores on either the test or the interview score between the group of 52 whites and the group of 12 blacks. There was also a negative relationship between age of candidate and score on written test.

JOB‐RELATEDNESS OF A POSTTRAINING JOB KNOWLEDGE CRITERION USE TO ASSESS VALIDITY AND TEST FAIRNESS

Significant job‐relatedness was found for a posttraining job knowledge test criterion using an application of Lawshe's content validity method. The aide test was used as a criterion to assess the predictive validity of a vocabulary test and a civil service test with samples of black (N= 43) and white (N= 62) psychiatric aides. Significant validities were found on both tests, but a vocabulary test proved to be the better predictor of the criterion in both samples. The obtained validities were discussed in terms of differential validity, test fairness, and sample size. This study demonstrated that a content validity method could be applied to criteria as well as selection tests. It was concluded that content validity methods may be able to help solve the problem of criterion relevance in validation research by providing quantitative evidence of the job‐relatedness of criteria.

THE IMPACT OF JOB ANALYSIS ON EMPLOYMENT TEST VALIDATION FOR MINORITY AND NONMINORITY ACCOUNTING PERSONNEL

The purposes of the present study were (a) to examine the comparative validity of a written job knowledge test constructed on the basis of a systematic job analysis with that of a commercial employment test selected in the absence of a prior job analysis for accounting positions and (b) to determine the fairness of each test for minority and nonminority job applicants. Results indicated that the job knowledge test was a valid and unbiased predictor of relevant criteria of job performance while the commercial employment test produced adverse impact and lacked validity. Implications of the results for future research studies and test validation efforts involving differential prediction are discussed.

Criterion Referenced Measures of Technical Proficiency in Maintenance Activities

The paper concerns instruments used in the determination of how effeciently maintenance men perform the various tasks of their jobs. Currently, a great deal of reliance is placed on unvalidated job-knowledge tests, theory tests, school marks, supervisors' ratings for such determination. This paper presents a composite of the results of many research and development efforts concerning the empirical validity of such instruments. These results indicate that these measurement instruments have low empirical validity. The limitations of traditional systems effectiveness measures are also discussed. Although job task performance tests (JTPT) have much higher empirical validity, such tests have had very limited use because of their relatively high costs in time, personnel and equipment. As a result of the foregoing findings, the Air Force Human Resources Laboratory (AFHRL) supported a series of efforts to develop better criterion referenced JTPT, and to attempt the development of paper and pencil symbolic substitute tests of high empirical validity. The paper describes the model battery of 48 criterion referenced JTPT, which has been developed to cover all key maintenance activities such as checkout, align/adjust, remove/replace, troubleshooting, test equipment and soldering. During this development many factors were considered including the identification and classification of tasks to be measured, the hierarchal relationship of maintenance tasks, the most effective order of their measurement and the ease of test administration. This battery was developed as a model of JTPT to be used on the job and in training. It was also intended as a battery of criterion tests for the validation of paper and pencil symbolic substitute tests. Batteries of graphic and video symbolic substitute tests were developed and given limited validations. The validation of graphic symbolic substitute tests indicated that the symbolics for all activities, with the exception of soldering, have promise. The paper discusses the requirements for additional refinement and validation for the various graphic tests. An unsuccessful effort to develop video symbolic substitute tests is also mentioned. Suggestions are made for the application of criterion referenced JTPT for the improvement of maintenance efficiency.

Project REALISTIC: Determination of Adult Functional Literacy Skill Levels

DESCRIBES RESEARCH on Project REALISTIC: a project to determine functional literacy levels for selected military jobs having civilian counterparts: Cooks, Vehicle Repairmen, Supply Clerks, and Armor Crewmen. Results of on-the-job interviews are presented which indicate how lack of reading skill may affect the extent to which men will seek job-related information. Relationships between reading ability and performance on adult, jobrelated reading task tests are described. Extensive data are presented which indicate relationships of reading, listening, and arithmetic skill to three indices of job proficiency for nearly 1600 men: performance on Job-Knowledge (multiple-choice) tests, performance on Job-Sample tests (hands-on tests in which each of the 1600 men were individually tested for 4-5 hours in a simulated job-situation, i.e., cooks cooked, repairmen repaired a vehicle, etc.) and Supervisor Ratings. Implications of the results for the establishment of functional literacy levels for adult basic literacy training are discussed.

A comparative evaluation of two approaches to job-knowledge test construction.

A comparative evaluation of two approaches to job-knowledge test construction.