Validity Of Assessment Centers Research Articles

The Relationship Between Assessment Centre Outcomes and Personality Traits

The present study aimed to assess the nomological validity of the Assessment Centre (AC). More specifically, the authors investigated the relationship between theoretically related personality variables and AC ratings in a personnel selection procedure. Previous studies have reported contradictory findings regarding the nomological validity of the Assessment Centre. We identified shortcomings in the previous studies and demonstrated the nomological validity of the AC in a relatively large sample of line manager candidates (N = 314). Our results are in harmony with trait activation theory, and this confirms the nomological validity of Assessment Centres. The theoretical and practical implications of these findings are discussed.

The Impact of Item Parceling Ratios and Strategies on the Internal Structure of Assessment Center Ratings

Abstract. The aim of the present study was to investigate whether using item parcels instead of single indicators would increase support for the factorial validity of assessment center (AC) ratings in factor analytic applications. Factor analytic analyses of AC ratings are often plagued by poor model fit as well as admissibility and termination problems. In the present study, three purposive item parceling strategies, in conjunction with three parceling approaches (specifying different ratios of indicators to dimensions), were investigated in relation to five confirmatory factor analysis specifications of AC ratings across two AC samples (Sample 1: N = 244; Sample 2: N = 320). The findings were equivocal across the two samples. Nonetheless, a three-parcel approach using a factorial allocation strategy performed better that a one-parcel approach (akin to the postexercise dimension rating).

How Different Indicator-Dimension Ratios in Assessment Center Ratings Affect Evidence for Dimension Factors.

Previous research on the construct validity of assessment center (AC) ratings has usually struggled to find support for dimension factors as an underlying source of variance of these ratings. Confirmatory factor analysis (CFA) remains the most widely used method to specify and validate the internal structure of AC ratings. However, the research support for dimension effects in AC ratings remains mixed. In addition, competing CFA models (e.g., correlated dimensions-correlated exercises models) are often plagued by non-convergence and estimation problems. Recently, it has been proposed that increasing the number of indicators per dimension and exercise combination might help to find support for dimension factors, in addition to exercise factors, in CFAs of AC ratings. Furthermore, it was also suggested that the increased ratio of indicators to dimensions may also solve some of the methodological problems associated with CFA models used to model AC ratings. However, in this research it remained unclear whether the support for dimension factors was solely due to the use of a larger indicator-dimension ratio or due to parceling that combines several behavioral indicators per dimension and exercise combination into more reliable measures of the targeted dimension. These are important empirical questions that have been left unanswered in the literature but can be potentially meaningful in seeking more balanced support for dimension effects in AC research. Using data from N = 213 participants from a 1-day AC, we aimed to investigate the impact of using different indicator-dimension ratios when specifying CFA models of AC ratings. Therefore, we investigated the impact of using different indicator-dimension ratios in the form of item parcels with data from an operational AC. On average, using three parcels eventually led to support for dimension factors in CFAs. However, exercise-based CFA models still performed better than dimension-based models. Thus, the present results point out potential limits concerning the generalizability of recent results that provided support for dimension factors in ACs.

Investigating the construct validity of an electronic in-basket exercise using bias-corrected bootstrapping and Monte Carlo re-sampling techniques

Orientation: Technology-based simulation exercises are popular assessment measures for the selection and development of human resources.Research purpose: The primary goal of this study was to investigate the construct validity of an electronic in-basket exercise using computer-based simulation technology. The secondary goal of the study was to investigate how re-sampling techniques can be used to recover model parameters using small samples.Motivation for the study: Although computer-based simulations are becoming more popular in the applied context, relatively little is known about the construct validity of these measures.Research approach/design and method: A quantitative ex post facto correlational design was used in the current study with a convenience sample (N = 89). The internal structure of the simulation exercise was assessed using a confirmatory factor analytical approach. In addition, bias-corrected bootstrapping and Monte Carlo simulation strategies were used to assess the confidence intervals around model parameters.Main findings: Support was not found for the entire model, but only for one of the dimensions, namely, the Interaction dimension. Multicollinearity was found between most of the dimensions that were problematic for factor analyses.Practical/managerial implications: This study holds important implications for assessment practitioners who hope to develop unproctored simulation exercises.Contribution/value-add: This study aims to contribute to the existing debate regarding the validity and utility of assessment centres (ACs), as well as to the literature concerning the use of technology-driven ACs. In addition, the study aims to make a methodological contribution by demonstrating how re-sampling techniques can be used in small AC samples.

Are assessment center behaviors' meanings consistent across exercises? A measurement invariance approach

To examine the appropriateness of a Multi‐Trait–Multi‐Method framework for testing construct validity of Assessment Centers (ACs) and get practical implications for the improved AC design, degree to which the AC dimension‐related performance behaviors consistently manifest across multiple AC rating situations was investigated. The present study used a large sample (N = 5,006) to apply a measurement invariance analysis. AC rating situations generally produced consistent factor loadings for items on AC dimensions, item residuals, dimension factor variances, and covariance between dimensions. The AC rating situation of interview tended to produce higher ratings and less item residuals. These findings support the consistency in constructs assessed across different AC rating situations, while some exercises may be better for teasing apart particular dimensions than others.

Situational bandwidth and the criterion-related validity of assessment center ratings: Is cross-exercise convergence always desirable?

This research examines the relationship between the construct and criterion-related validity of assessment centers (ACs) based on how convergence of dimension ratings across AC exercises affects their ability to predict managerial performance. According to traditional multitrait-multimethod perspective, a high degree of convergence represents more reliable measurement and has the potential for better validity. In contrast, the concept of situational bandwidth suggests that behavior assessed under a dissimilar set of circumstances should result in a more comprehensive assessment of a candidate's tendencies even though ratings are less likely to show high convergence. To test these opposing viewpoints, data from 3 operational ACs were obtained along with experts' evaluations of exercise characteristics and supervisors' ratings of candidates' managerial performance. Across the 3 samples, AC ratings taken from exercises with dissimilar demands had higher estimates of criterion-related validity than ratings taken from similar exercises, even though the same dimension-different exercise correlations were substantially higher between similar exercises. Composites of ratings high in convergence did not emerge as better predictors of managerial performance, and validity particularly suffered when derived from ratings that converged as a result of exercises with similar demands. Implications for AC design are discussed.

Situation assessment as an ignored factor in the behavioral consistency paradigm underlying the validity of personnel selection procedures.

This study contributes to the literature on why selection procedures that are based on the behavioral consistency logic (e.g., structured interviews and assessment centers) are valid predictors of job performance. We rely on interactionist theories to propose that individual differences in assessing situational demands explain true variance in performance in selection procedures and on the job. Results from 124 individuals in a simulated selection process showed that the assessment of situational demands was related to both selection and job performance. Individual differences in assessing situational demands also contributed to the criterion-related validity of assessment center and structured interview ratings, offering a complementary explanation as to why selection procedures based on the notion of behavioral consistency predict job performance.

Assessment Center Dimensions Predict Performance-Based Bonus in Business Management Settings

This study sought to add to the literature on the validity of Assessment centers (ACs) by first examining the factorial structure emerging from observers’ dimension ratings and then examining their predictive validity using a performance criterion often unavailable to researchers—performance-based bonus payment. A series of ACs specially designed for the selection of candidates for entry-mid tier management positions in a large financial corporate (n = 180) was used as the sampling frame. For candidates who were promoted to managerial position we gathered bonus information within 6 - 12 months of their promotion (n = 75). The dimension ratings and factorial structure of the AC were examined to reveal a 2-factor structure pertaining to cognitive and interpersonal aspects of performance. Both the original dimensions and the two factorial grades showed moderate predictive validity using performance-based bonus as the criterion: The ‘organizational commitment’ dimension best predicted bonus payment (r = .38; p < .01) and the interpersonal factorial grade best predicted bonus (standardized b = .22 p < .01), followed by the cognitive factor, after controlling for gender and tenure. The theoretical and practical implications of the findings are briefly discussed.

The Predictive Validity of Assessment Centers in German-Speaking Regions

As previous meta-analyses have focused almost solely on English-speaking regions, this study presents the first systematic meta-analytical examination of the predictive validity of assessment centers (ACs) conducted in German-speaking regions. It summarizes 24 validity coefficients taken from 19 studies (N = 3,556), yielding a mean corrected validity of ρ = .396 (80% credibility interval .235 ≤ ρ ≤ .558). ACs with different purposes and different kinds of criterion measures were analyzed separately. Furthermore, target group (internal vs. external candidates), average age of the assessees, inclusion of intelligence measures, number of instruments used, AC duration, as well as time elapsed between AC and criterion assessment were found to moderate the validity.

Reliability and Validity of Nested-Designed Assessment Center

Reliability and Validity of Nested-Designed Assessment Center

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.

Assessment center construct-related validity: Stepping beyond the MTMM matrix

Recent Monte Carlo research has illustrated that the traditional method for assessing the construct-related validity of assessment center (AC) post-exercise dimension ratings (PEDRs), an application of confirmatory factor analysis (CFA) to a multitrait–multimethod matrix, produces inconsistent results [Lance, C. E., Woehr, D. J., & Meade, A. W. (2007). Case study: A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods, 10, 430–448]. To avoid this shortcoming, a variance partitioning procedure was applied to the examination of the PEDRs of 193 individuals. Overall, results indicated that the person, dimension, and person by dimension interaction effects together accounted for approximately 32% of the total variance in AC ratings. However, despite no apparent exercise effect, the person by exercise interaction accounted for approximately 28% of the total variance. Although these results are drawn from a single AC, they nevertheless provide general support for the overall functioning of ACs and encourage continued application of variance partitioning approaches to AC research. Implications for AC design and research are discussed.

Validity of assessment centers for personnel selection

Research and practice in the application of assessment centers (AC) for personnel selection are reviewed and critiqued. Several examples of the use of ACs for external screening, internal promotion, and certification are described. Several types of evidence of validity of ACs for selection are reviewed, including representativeness of the content of dimensions and exercises in relation to job requirements, relationships among ratings within an AC, relationships of AC ratings and criteria of work effectiveness, and consequences of assessments including candidates' reactions to assessments and sub-group differences in ratings. Several controversies in research findings and practices of ACs are noted. Further research to address these controversies and new research to study emerging issues are suggested. Conclusions about the validity, fairness, and legal defensibility of ACs for personnel selection are offered.

Improving Assessment Centers by the Trimodal Concept of Personnel Assessment

Lance (2008) attacks the ‘‘assessment center validity paradox,’’ arguing that construct validity is not as low as a number of researchers seem to have agreed about. But he neglects the second part of the supposed paradox, predictive validity. Assessment center predictive validity is not as high as is circulated by many researchers—and all the more by most of the users. This seems to me the even more important deficit. My suggestion is to solve the theoretical problem of construct validity as well as the pragmatic problem of predictive validity by (re)conceiving the assessment center as a truly multimodal procedure. Assessment centers used to be valuable diagnostic tools—at times when they were run as procedures integrating a broad methodological diversity. One of the icons of assessment center research, the extensive ATT that is, there is a rather continuous decline of assessment center validity over the past 40 years. There are several possible explanations for this decline. My suggestion is that assessment centers often perform poorly because too simplistic methods are employed. Their attractiveness for managers and practitioners in personnel departments is connected with concentrating on ‘‘exercises’’such as a group discussion, roleplay, and presentation, which allow for behavioral observations and a lively personal impression formation but are essentially nonpsychometric tools. Correspondence concerning this article should be addressed to Heinz Schuler. E-mail: schuler@unihohenheim.de Address: Department of Psychology (540F), University of Hohenheim, D-70593 Stuttgart, Germany Heinz Schuler, Department of Psychology (540F), University of Hohenheim Industrial and Organizational Psychology, 1 (2008), 128–130. Copyright a 2008 Society for Industrial and Organizational Psychology. 1754-9426/08

The Validity of Assessment Centres for the Prediction of Supervisory Performance Ratings: A meta‐analysis

The current meta‐analysis of the selection validity of assessment centres aims to update an earlier meta‐analysis of assessment centre validity. To this end, we retrieved 26 studies and 27 validity coefficients (N=5850) relating the Overall Assessment Rating (OAR) to supervisory performance ratings. The current study obtained a corrected correlation of .28 between the OAR and supervisory job performance ratings (95% confidence interval .24≤ρ≤.32). It is further suggested that this validity estimate is likely to be conservative given that assessment centre validities tend to be affected by indirect range restriction.

Do within-dimension ratings in assessment centers really lead to improved construct validity?

Abstract. In their meta-analysis on the construct validity of assessment centers (ACs), Woehr and Arthur (2003) found that within-dimension ratings of candidates’ performance had better construct validity than within-exercise ratings. However, this finding conflicts with previous experimental studies. In these studies, comparable results were only obtained when the rating approach was confounded with other factors, which are also likely to affect construct validity but which were not taken into account by Woehr and Arthur. In the present investigation, we identified factors that are often associated with a within-dimension rating approach. Most of these factors were also confounded with the rating approach in the studies from Woehr and Arthur’s meta-analysis. Meta-analytic moderator analyses showed that the time at which assessors rated candidates, the exchange of information between assessors, and assessor rotation moderate AC construct validity. The effects of the rating approach can probably be attributed to these confounding factors.

Extending the Nomological Network of Assessment Center Construct Validity: Prediction of Cross-Situationally Consistent and Specific Aspects of Assessment Center Performance

The purpose of this study was to expand the nomological validity of assessment centers (ACs) by investigating predictors of cross-situationally consistent versus specific aspects of AC performance. Consistent with hypotheses, (a) Big Five personality factors predicted AC performance as it related to a cross-situationally consistent general performance factor but not as it related to exercise (i.e., situationally specific) factors, and (b) job knowledge predicted performance as it related to both the general performance factor and exercise-specific factors. Results are interpreted as they relate to the growing literature on AC construct validity.

Is Personality Related to Assessment Center Performance? That Depends on How Old You Are

Despite compelling arguments that performance in the assessment center (AC) should be related to personality, past research has failed to demonstrate a consistent link between personality and AC performance. This study investigated whether managerial candidates’ age could be moderating, and thus obscuring, specific personality–AC performance relations. A large sample of managers (N = 371) participated in an AC and also completed the leadership relevant personality scales of achievement, dominance, and exhibition from the Personality Research Form-E (PRF-E; Jackson, D. N. (1984). Manual for the personality research form. Port Huron, MI: Research Psychologists Press). Consistent with hypotheses, age moderated the relations of dominance and exhibition with AC performance (p < .05), such that dominance and exhibition were more strongly related to AC performance for older as opposed to younger managers. Results were discussed in terms of their implications for the criterion-related and construct validity of ACs.

The Validity of Assessment Center Ratings and 16PF Personality Trait Scores in Police Sergeant Promotions: A Case of Incremental Validity

Assessment centers (AC) have become a mainstay of promotion systems for both the private and public sectors--police agencies included. (1) While costly to develop and implement, their track record of validity and legal defensibility has justified their increased use. (2) Similarly, the use of personality trait measurement in personnel decisions has also shown a substantial increase in use based on its validity in predicting a range of employee and managerial job behaviors. (3) Unlike ACs, police agencies have used personality assessment primarily for entry level officer screening and fitness for duty assessments. (4) The use of personality assessment in police officer promotional decisions has rarely been reported. Personality trait scores have shown incremental validity, however, when combined with cognitive ability tests in predicting a range of employee and managerial behaviors, (5) but not for state police recruits. (6) In addition, incremental validity has been found for personality measures when combined with biodata (7) and ACs. (8) Other than the attempt of Cortina, Coherty, Schmitt, Kaufmann and Smith, (9) who found no evidence of incremental validity, there have been no other reported studies of incremental validity for personality trait scores when used in combination with any other predictor of police officer or supervisor performance. Evidence of Assessment Center Validity There is substantial literature which documents the validity of ACs in predicting on-the-job performance, especially for managers. (10) In addition to its proven validity, based on candidate reactions and court challenges, the AC method has been labeled fair and legally defensible. (11) The validity of ACs used for police officer selection and promotion has been shown to be commensurate with that found in other organizational settings. For example, AC performance ratings have been shown to be significantly related to supervisor ratings of officer performance, (12) valid for high ranking police officials (13) and accurate in prediction over a 19-year time span. (14) Evidence of Validity for Personality Measurement While early reviews of the literature indicated that personality assessment yielded poor validity in predicting job performance, (15) more recent work has established personality as a bona fide predictor of a range of job-related behaviors across a variety of organizational settings and employee groups, including police officers. (16) A number of meta-analyses have provided reasonable support for the use of personality assessment in selection. (17) The decision to use personality assessment by itself within an employee selection context should be made with caution, given the greater validity shown by other predictor measures, such as cognitive ability tests and assessment centers. The resurgence of personality assessment as a valid predictor of job performance is due in large part to the fact that: 1. Personality has been shown to be a valid predictor of work-related outcomes; 2. Personality measures do not generally display adverse impact against demographic subgroups (e.g., racial, gender, ethnic, etc.); 3. The validity of personality measures is not affected by intentional faking; (18) and 4. The construct of personality has become more structured with the acceptance of the personality traits (e.g., conscientiousness and extraversion) and the consistency with which they are measured. (19) Many studies have dealt specifically with the ability of personality trait scores to predict police officer performance, both on the job and while in training. Personality traits have been measured through traditional clinical self report inventories as well as instruments designed to assess the Big Five dimensions (e.g., the Minnesota Multiphasic Personality Inventory; (20) the Inwald Personality Inventory; (21) the California Psychological Inventory; (22) the Eyesenck Personality Inventory, the SCL-90, and the Bender-Gestalt; (23) the 16PF; (24) and the NEO Personality Inventory (25)). …

Comment évaluer les compétences clés dans le domaine professionnel ?

The article attempts to clarify the question of defining and evaluating key competencies in the professional domain, by reviewing existing literature in the domains of cognitive differential psychology, psychometrics and industrial and organisational psychology. A hierarchical model of competencies is favoured, in which the key competencies are general across professional situations and transferable from one situation to another. The different methods of evaluating key competencies are reviewed. Particular attention is paid to the validity of assessment centres and intelligence tests. The crucial role of general intelligence, depending on the complexity of professional tasks, is demonstrated through an analysis of professional profiles. We review writings on the psychology of intelligence in order to show the importance of the g factor as principle predictor of work performance and the acquisition of key competencies. Finally, the judicious combination of different validated and economical psychometric tools is proposed as the most effective and equitable solution for evaluating key competencies in the three paradigmatic social situations which require it: vocational orientation, recruitment of personnel and access to continuing training.