Use Of Situational Judgment Tests Research Articles

Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.

Situational judgment tests: the future of medical selection?

Fiona Patterson and Vicki Ashworth of the Work Psychology Group (www.workpsychologygroup.com) look at the growing use of situational judgment tests

Assessing managers' common sense using situational judgment tests

PurposeIn industrial‐organizational psychology, research and practice has focused on the use of situational judgment tests to predict managerial job performance. Although there is considerable controversy over what these tests actually measure, many have argued that they at least partially measure practical intelligence or “common sense.” Therefore the purpose of this article is to discuss situational judgment tests as assessments of common sense.Design/methodology/approachReviewing the relevant industrial‐organizational psychology literature, this article considers the history and development of situational judgment tests as well as how the field has defined situational judgment. It also reviews the empirical and theoretical literature on their effectiveness for assessing managerial decision‐making competence.FindingsSituational judgment tests have been found to be effective predictors of many organizational outcomes in a variety of job settings and positions. In particular, they have been found to incrementally predict future performance on‐the‐job beyond other typical predictors. Situational judgment tests are also relatively easy to administer and score and are often well received by managers and job applicants.Originality/valueThis article suggests situational judgment tests offer an efficient method of assessing common sense that can easily be adapted to many employment settings.

Threats to the Operational Use of Situational Judgment Tests in the College Admission Process

This study examined the coachability of two situational judgment tests, the College Student Questionnaire (CSQ) and the Situational Judgment Inventory (SJI), developed for consideration as selection instruments in the college admission process. Strategies for raising scores on each test were generated, and undergraduates were trained in the use of the strategies using a video-based training program. Results indicated that the CSQ was susceptible to coaching. In addition, the scoring format of the CSQ was found to be easily exploited, such that trainees could increase their scores by greater than 1 SD simply by avoiding extreme responses on that test. The results as a whole sounded a note of caution for the potential use of the CSQ in the college admission process.

What Would You Do? The Use of Situational Judgment Tests in Organizational Selection

What Would You Do? The Use of Situational Judgment Tests in Organizational Selection

Situational Judgment Tests and their Predictiveness of College Students’ Success: The Influence of Faking

There is increasing interest in using situational judgment tests (SJTs) to supplement traditional student admission procedures. An important unexplored issue is whether students can intentionally distort or fake their responses on SJTs. This study examined the fakability of an SJT of college students’ performance. Two hundred ninety-three psychology students completed a cognitive test, a personality measure, and an SJT. Only for the SJT, the students were assigned to either an honest or a fake condition. The scores of students in the fake condition were significantly higher than those of students in the honest condition ( d = .89). Furthermore, faking had a negative effect on the criterion-related validity (there was a significant drop from r = .33 to r = .09) and the incremental validity of the SJT over cognitive ability and personality. These results are discussed in terms of the use of SJTs in high-stakes testing programs.

Use of situational judgment tests to predict job performance: a clarification of the literature.

Although situational judgment tests have a long history in the psychological assessment literature and continue to be frequently used in employment contexts, there has been virtually no summarization of this literature. The purpose of this article is to review the history of such tests and present the results of a meta-analysis on criterion-related and construct validity. On the basis of 102 coefficients and 10,640 people, situational judgment tests showed useful levels of validity (rho = .34) that were generalizable. A review of 79 correlations between situational judgment tests and general cognitive ability involving 16,984 people indicated that situational judgment tests typically evidence relationships with cognitive ability (rho = .46). On the basis of the literature review and meta-analytic findings, implications for the continued use of situational judgment tests are discussed, particularly in terms of recent investigations into tacit knowledge.

Use of situational judgment tests to predict job performance: A clarification of the literature.

Use of situational judgment tests to predict job performance: A clarification of the literature.