Validity Of Situational Judgment Tests Research Articles

Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews

BackgroundSituational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context.MethodsThirty participants—15 student pharmacists and 15 practicing pharmacists—completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes.ResultsThe findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item.ConclusionsThis study provides additional SJT validity evidence by exploring participants’ response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.

The role of Situations in Situational Judgment Tests: Effects on construct saturation, predictive validity, and applicant perceptions.

Recent theorizing and empirical evidence suggesting that Situational Judgment Tests (SJTs) are more context-independent than previously thought has sparked a debate about the role of situation descriptions in SJTs. To contribute to this debate and add to our understanding of how SJTs work, this article conceptually embeds SJT performance in a situation construal model and examines the effects of situation descriptions on the construct saturation and predictive validity of SJT scores, as well as on applicant perceptions. Across two studies (N = 1,092 and 578) and different SJTs, personality and cognitive ability were equally important determinants of SJT performance regardless of whether situation descriptions were presented or omitted. The effects of removing situation descriptions on the criterion-related validity of SJT scores differed depending on the breadth of the criteria. For predicting global job performance criteria (in-role performance and organizational citizenship behavior), SJT validity was not significantly affected, whereas it decreased for predicting more specific criteria (interpersonal adaptability, efficacy for teamwork). Finally, the effects of omitting situation descriptions in SJTs on applicant perceptions were either negligible or small. Implications for SJT theory, research, and design are discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Prosocial implicit trait policies underlie performance on different situational judgment tests with interpersonal content

ABSTRACTThis study tests the hypothesis that situational judgment tests (SJTs) with interpersonal content reflect implicit beliefs about the utility of prosocial action for job effectiveness and that agreeable people are more likely to believe that prosocial action is effective. Two hundred ninety-four undergraduates completed four different SJTs with interpersonal content and a measure of Agreeableness. Results show that the effectiveness of response options in these SJTs is positively correlated with the level of prosociality they express and that because of their prosocial elements, scores on different SJTs are correlated with one another and with Agreeableness. These results shed light on the construct-related validity of SJTs with interpersonal content and point to the possibility that they can assess prosociality in job settings different from those described in their items.

Optimizing the validity of situational judgment tests: The importance of scoring methods

In recent years, situational judgment tests (SJTs) have made strong inroads in assessment practices. Despite the importance of scoring for the validity of SJTs, little attention has been paid to different SJT scoring methods. This study investigated the influence of scoring methods on the criterion-related validity of SJTs. We examined five different consensus scoring methods (i.e., raw, standardized, dichotomous, mode, and proportion scoring) and several integrated scoring methods for scoring the same SJT. Results showed that one of the most popular scoring approaches (raw consensus scoring) is associated with an extreme response tendency and yields the lowest scale validity of all scoring approaches examined. Moreover, the mean item validity of midrange items was good only when they were scored by the mode consensus method. Thus, this study extends previous work (McDaniel et al., 2011) by deepening our understanding of how different scoring methods improve the validities of SJTs. Our findings suggest that using scoring methods that control the influence of extreme response tendency on the scores of SJTs yields higher validities. Finally, this study is the first to suggest that scoring SJTs with integrated methods yielded higher mean item validities than using any single method.

The Predictive Validity of a Text-Based Situational Judgment Test in Undergraduate Medical and Dental School Admissions.

Situational judgment tests (SJTs) can be used to assess the nonacademic attributes necessary for medical and dental trainees to become successful practitioners. Evidence for SJTs' predictive validity, however, relates predominantly to selection in postgraduate settings or using video-based SJTs at the undergraduate level; it may not be directly transferable to text-based SJTs in undergraduate medical and dental school selection. This preliminary study aimed to address these gaps by assessing the validity of the UK Clinical Aptitude Test (UKCAT) text-based SJT. Study participants were 218 first-year medical and dental students from four UK undergraduate schools who completed the first UKCAT text-based SJT in 2013. Outcome measures were educational supervisor ratings of in-role performance in problem-based learning tutorial sessions-mean rating across the three domains measured by the SJT (integrity, perspective taking, and team involvement) and an overall judgment of performance-collected in 2015. There were significant correlations between SJT scores and both mean supervisor ratings (uncorrected r = 0.24, P < .001; corrected r = 0.34) and overall judgments (uncorrected rs = 0.16, P < .05; corrected rs = 0.20). SJT scores predicted 6% of variance in mean supervisor ratings across the three nonacademic domains. The results provide evidence that a well-designed text-based SJT can be appropriately integrated, and add value to, the selection process for undergraduate medical and dental school. More evidence is needed regarding the longitudinal predictive validity of SJTs throughout medical and dental training pathways, with appropriate outcome criteria.

Knowing What NOT To Do Is a Critical Job Skill: Evidence from 10 different scoring methods

Situational judgment tests (SJTs) have become an increasingly important tool for predicting employee performance; however, at least two key areas warrant further investigation. First, prior studies of SJTs have generally relied on samples from the western world, leaving open the question of the validity of using SJTs in the developing world where the majority of the world's workforce resides. Second, there is currently no standardized, theoretically-based method for the development and scoring of SJTs. Therefore, SJTs are highly domain-specific and must be developed anew for each new context. We report the results of three studies, conducted in India, that aim to: (1) test the cross-cultural validity of SJTs in a non-western context, and (2) examine the differential validity of 10 different approaches to scoring SJTs, some of which have the potential to resolve the problem of developing a theoretically-infused, standardized approach to scoring and future development.

The “Hot Mess” of Situational Judgment Test Construct Validity and Other Issues

The construct validity of situational judgment tests (SJTs) is a “hot mess.” The suggestions of Lievens and Motowidlo (2016) concerning a strategy to make the constructs assessed by an SJT more “clear and explicit” (p. 5) are worthy of serious consideration. In this commentary, we highlight two challenges that will likely need to be addressed before one can develop SJTs with clear and explicit constructs. We also offer critiques of four positions presented by Lievens and Motowidlo that are not well supported by evidence.

Effects of Organizationally Endorsed Coaching on Performance and Validity of Situational Judgment Tests

There is growing interest in organizationally provided or organizationally endorsed coaching. However, little is known about the effects of such coaching on test scores in operational settings. This study reports on an examination of such a program in the context of the use of a situational judgment test (SJT) for medical school admissions. We examine the effects of multiple types of coaching methods on SJT scores and on their construct‐related and predictive validities. Results suggest that (1) commercial coaching techniques may not be as effective as previously thought, whereas organizationally provided methods may be more effective, and that (2) the criterion‐related validity of the SJT scores is not degraded by the availability of coaching. Generally, this study illustrates that concerns about potential unfairness of coaching can be countered by making effective coaching available to all examinees, in the form of organizationally endorsed coaching.

Situational Judgment Test Validity: A Matter of Approach

Previous research has examined how the method used to score a situational judgment test (SJT) affects the validity of the SJT (Bergman, Drasgow, Donovan, Henning, & Juraska, 2006). We seek to add a...

On the Limitations of Using Situational Judgment Tests to Measure Interpersonal Skills: The Moderating Influence of Employee Anger

Many authors have suggested that situational judgment tests (SJTs) are useful tools for assessing applicants because SJT items can be written to assess a number of job‐related knowledges, skills, abilities and other characteristics (KSAOs). However, SJTs may not be appropriate for measuring certain KSAOs for some applicants. We posit that using SJTs to measure interpersonal skills may lead to invalid inferences about applicants with higher levels of angry hostility (AH), and thus, AH should moderate the relation between interpersonally oriented SJTs and job performance. Three studies, using samples of healthcare workers (n = 225), police officers (n = 54), and medical doctors (n = 92), provided support for hypotheses in that that relations between SJT scores and performance criteria were significantly weaker among employees higher in AH compared to those lower in AH. In addition, none of the other facets of neuroticism tested (self‐consciousness, anxiety, depression, immoderation, or vulnerability to stress) consistently moderated SJT validity, providing support for the uniqueness of AH. Implications for practice, and for future research studying the relations between interpersonal skills as measured by SJTs and job performance, are discussed.

Situational judgment tests: a review of recent research

PurposeThe purpose of this paper is to give an empirically‐based review of the strengths and weaknesses of situational judgment tests (SJTs).Design/methodology/approachThe features, history, and development of SJTs are discussed. Next, a computerized search (from 1990‐2007) is conducted to retrieve empirical studies related to SJTs. The effectiveness of SJTs is discussed in terms of reliability, criterion‐related validity, incremental validity, construct‐related validity, utility, adverse impact, applicant perceptions, fakability, and susceptibility to practice and coaching effects.FindingsStrengths of SJTs are that they show criterion‐related validity and incremental validity above cognitive ability and personality tests. SJTs have also less adverse impact towards minorities (especially if the cognitive loading of the SJT is low). Furthermore, applicant reactions towards SJTs are positive and SJTs enable to test large applicant groups at once (through the Internet). In terms of weaknesses, SJTs might be prone to faking, practice, and coaching effects. There is also debate about what constructs are measured by SJTs.Research limitations/implicationsFive avenues for future research are discussed: construct‐related validity of SJTs, utility of SJTs vis‐à‐vis other predictors, impact of SJT features on validity and adverse impact, examination of alternative stimulus and response formats, and cross‐cultural transportability of SJTs.Practical implicationsPractitioners receive evidence‐based information about the features, development, and strengths and weaknesses of SJTs.Originality/valueApart from the USA, SJTs have not made strong inroads in selection practice in Europe and other parts of the world. This evidence‐based paper might highlight the value of SJTs.

Incremental Validity of Situational Judgment Tests for Task and Contextual Job Performance

This paper has three goals. First, it responds to calls for additional research on subgroup differences in situational judgment tests. Second, it expands the cumulative knowledge on the incremental validity of situational judgment tests beyond cognitive ability and personality. Third, it examines the validity and incremental validity of various predictors for both task and contextual performance.

Situational judgment test research: Informing the debate on practical intelligence theory

[Gottfredson, L.S. (2003). Dissecting practical intelligence theory: Its claims and evidence. Intelligence, 31, 343–397.] provided a detailed critique of Sternberg's [Sternberg, R.J., Fotsythe, G.B., Hedlund, J., Horvath, J.A., Wagner, R.K., Williams, W.M., Snook, S.A., Grigorenko, E.L. (2000). Practical intelligence in everyday life. New York: Cambridge University Press.] practical intelligence theory. The current paper seeks to supplement Gottfredson's extensive critique using research and practice in the situational judgment literature in industrial/organizational psychology to inform the debate concerning practical intelligence. This paper makes four major points. First, there is a research and practice tradition in personnel selection that has used the item type that Sternberg uses to assess what he calls tacit knowledge. This item type is called situational judgment in the personnel selection field. There is an extensive literature on situational judgment tests that can inform debate concerning practical intelligence. Second, we present evidence that situational judgment tests do not measure a general factor, whether labeled practical intelligence, or something else. Both the individual test items and the tests are factorially complex and measure multiple known constructs. Third, since the measures assess multiple constructs ( g and personality), they are best viewed as measurement methods. Fourth, we compare the validity of the measures with g for the prediction of job performance and examine the incremental validity of situational judgment tests. We conclude that this research can help guide a revision to practical intelligence theory.

Situational Judgment and Job Performance

Data from 160 civil service employees demonstrate the validity of a situational judgment test in predicting overall job performance as well as three performance dimensions: task performance (core technical proficiency), motivational contextual performance (job dedication), and interpersonal contextual performance (interpersonal facilitation). Situational judgment also provided incremental validity over the prediction provided jointly by cognitive ability, the Big Five personality traits, and job experience. These findings extended the work of Clevenger, Pereira, Wiechmann, Schmitt, and Harvey (2001) on the incremental validity of situational judgment tests as well as the meta-analytic results reported by McDaniel, Morgeson, Finnegan, Campion, and Braverman (2001). Implications are discussed in terms of research on the prediction and understanding of job performance.

Incremental validation of situational judgment tests.

Using 3 different samples, the authors assessed the incremental validity of situational judgment inventories (SJIs), relative to job knowledge, cognitive ability, job experience, and conscientiousness, in the prediction of job performance. The SJI was a valid predictor in all 3 samples and incrementally so in 2 samples. Relative to the other predictors, SJI's partial correlation with performance, controlling for the other 4 predictors, was superior in most comparisons. Subgroup differences on the SJI also appear to be less than those for cognitive ability and job knowledge, but greater than differences in conscientiousness. The SJI should prove to be a valuable additional measure in the prediction of job performance, but several additional areas of research are suggested.

Situational Judgment Tests: A Review of Practice and Constructs Assessed

In this article, we seek to summarize current practice concerning situational judgment tests in personnel selection. We begin by describing the manner in which situational judgment tests are developed and examining the diverse ways in which situational items are presented and scored. We then offer speculation concerning constructs assessed by situational judgment tests as well as discuss the legal aspect of situational judgment measures. We also review meta‐analytic evidence concerning the construct validity of situational judgment tests and offer several new meta‐analytic findings. Situational judgment tests are shown to be typically correlated moderately with general mental ability. Their primary personality correlates are emotional stability, conscientiousness, and agreeableness. Situational test scores also tend to increase with increasing years of job experience. The article concludes with a list of areas that need addressed in future research.

Incremental validity of situational judgment tests.

Incremental validity of situational judgment tests.