Situational Judgment Test Items Research Articles

The influence of situational strength on the relation of personality and situational judgment test performance

AbstractSituational strength theory has been used as a theoretical underpinning of person–situation processes that are linked to job performance. Accordingly, the link between personality traits and job performance increases in weak situations. Building on this research, similar mechanisms have been proposed for simulation‐based selection tools, such as Situational Judgment Tests (SJTs), to explain how these measures work as predictors of job performance. However, underlying processes of SJT performance are subject to debate with some scholars arguing in favor of context‐independent processes while others maintain that situations play an essential role. This study (N = 707) examined whether the strength of situations in SJT items moderated the relation between personality and SJT performance. Results did not support the notion that personality is more strongly related to SJT performance when situations are weak. In fact, for some traits, the opposite may be true as more situational constraints led to an increase in the relation of extraversion, emotional stability, and SJT performance. The results add to an increasing body of research about psychological processes in SJTs. Limitations and implications for research and practice are discussed.

Use of situational judgment tests for assessing non-cognitive attributes of final year dental students.

Situational judgment tests (SJT) have been previously used in the admission process of medical and dental undergraduate programs, but their utilization as an assessment tool is fairly new. The objective of this study was to explore essential non-cognitive attributes of dental graduates. We also aimed to assess the attributes of final year dental students utilizing SJT and later collected participants' reactions on the test. A mixed-method exploratory sequential study was conducted. One-on-one and group discussions were done with subject matter experts (SMEs) to explore the essential attributes of fresh dental graduates. Based on the thematic analysis, 10 SJT items per domain were constructed by one group of SMEs and expert validated by a different group of SMEs and vice versa. The final dental-SJT was piloted online on final year dental students and their perceptions were collected using a five-point Likert scale questionnaire. Descriptive statistics and Cronbach's alpha were calculated for the test. The consultation with SMEs generated three main themes, namely, Conventional skills, Tribal skills, and Occupational skills. The 70-item dental-SJT piloted on 150 final year dental students revealed data were normally distributed. The internal reliability of test was good (0.738). Students scored highest in "management skills" (>70%) and lowest in "professional ethics" (56%) and "diverse experiences" (58%). The overall student perception about the test was good. SJT is an effective tool to assess non-cognitive attributes of dental students. It has an overall positive perception about perceived predictive validity, perceived fairness, face and content validity.

Early detection measures: addressing Unemployability using Modified Angoff in situational judgment test

This is the first standard setting study in Malaysia aimed at developing Performance-Level Descriptors (PLDs) and determining cut scores that differentiate undergraduates with low, moderate and high employability skills using a newly developed Situational judgment Test (SJT). SJT items from five domains critical for future job employment: 1) critical thinking and problem solving, 2) enterprising, 3) integrity, 4) leadership and 5) teamwork were developed. Modified Angoff method involving PLDs and cut scores were developed by five subject-matter experts. Two test instruments were used: 10 practice and 36 original bilingual Malay-English SJT items. The PLDs and two cut scores to distinguish low, moderate and high employability skills were validly developed, without compromising consequential validity. The findings benefit undergraduates with low employability as they can be trained with current global job readiness skills by the universities, which also forms the practice implication of this study. Another practical implication is academicians can be further encouraged to understand the theoretical framework of establishing standard setting for employability.

Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

AbstractIt is common practice to rely on a convenience sample of subject matter experts (SMEs) when developing scoring keys for situational judgment tests (SJTs). However, the defining characteristics of what constitutes a SME are often ambiguous and inconsistent. Sampling SMEs can also impose considerable costs. Other research fields have adopted crowdsourcing methods to replace or reproduce judgments thought to require subject matter expertise. Therefore, we conducted the current study to compare crowdsourced scoring keys to SME‐based scoring keys for three SJTs designed for three different job domains: Medicine, Communication, and Military. Our results indicate that scoring keys derived from crowdsourced samples are likely to converge with keys based on SME judgment, regardless of test content (r = .88 to .94 between keys). We observed the weakest agreement among individual MTurk and SME ratings for the Medical SJT (classification consistency = 61%) relative to the Military and Communication SJTs (80% and 85%). Although general mental ability and conscientiousness were each related to greater expert similarity among MTurk raters, the average crowd rating outperformed nearly all individual MTurk raters. Using randomly‐drawn bootstrapped samples of MTurk ratings in each of the three samples, we found that as few as 30–40 raters may provide adequate estimates of SME judgments of most SJT items. These findings suggest the potential usefulness of crowdsourcing as an alternative or supplement to SME‐generated scoring keys.

Pre-service Science and Mathematics teachers’ reasoning: a think-aloud study

ABSTRACT Teacher education programmes struggle to attract, develop, and retain potential teachers in Science and Mathematics. The development of both academic and non-academic attributes (e.g. adaptability, empathy) during a programme can influence not only retention but the profession-readiness of graduates. This is especially important in underserved schools where high attrition rates are more likely with Science and Mathematics teachers in their first few years of teaching. Yet most programmes lack intentional opportunities for non-academic development. Using a qualitative case study approach with an established think-aloud protocol, nine pre-service teachers (Science or Mathematics) were presented with situational judgment test items. A three-step analytical strategy revealed participants’ reasoning with a range of motivational beliefs and values. Results point to four broad reasoning profiles (solution, consequence, student, and value-focused). By exploring the implicit and explicit beliefs and values that contribute to prospective teachers’ reasoning processes, we offer implications for professional development and future research.

Development of a situational judgment test for global engineering competency

AbstractBackgroundAs globalization continues to impact the engineering profession, many programs aim to prepare current and future engineers to work across national and cultural boundaries. Yet there remains a lack of quality tools for assessing global competency among engineers and other technical professionals, including their behavioral tendencies in global work situations.PurposeWe introduce development of a situational judgment test (SJT) covering three dimensions of global engineering competency (GEC) in Chinese national/cultural context. The main aim of this article is to describe how the SJT was developed through a systematic multistep process. Secondarily, we explore relationships between SJT performance and other theoretically relevant variables.MethodsAfter generating a large initial pool of SJT scenarios and behavioral response items, we used ratings from subject matter experts (SMEs) to select six SJT scenarios and create scoring keys for 26 response items. To further explore the instrument's validity, we deployed the SJT items, other relevant measures, and a demographic survey to a sample of practicing engineers (n = 400).ResultsSME ratings provide strong evidence for the content relevance of the GEC–SJT tool. Survey results also suggest positive relationships between SJT performance and Chinese cultural knowledge, age, and years of work experience. However, more validity and reliability evidence is needed before recommending wider use of the instrument.ConclusionsOur findings suggest the SJT format as a promising behavior‐based approach to measuring global competency and other professional attributes in engineering. We also discuss directions for future research and training efforts related to assessing and developing global competency.

Performance on Video-Based Situational Judgment Test Items: Simulated Interracial Interactions

Individuals are known to categorize others into social groups based on cues like race and gender and to experience relative discomfort when interacting with “outgroup” members. Two experimental studies were used to examine whether actor demographic cues in situational judgment assessment items completed by test takers in a simulated employee selection context may lead to differences in their performance and reactions to the hiring organization. In both studies, test takers assumed the perspective of actors shown in video-based scenarios and indicated how they would respond to interaction partners (IPs) to whom they were racially similar or dissimilar. In Study 1, a given test taker responded to IPs of a constant gender; in Study 2, IPs’ gender varied across scenarios within each condition. In Study 1, Black test takers spent more time and scored better on two of the four scenarios when responding to racially similar IPs. These effects were not found in Study 2, but demographic cues showed new interactive effects on performance and reactions. We discuss the implications of different findings across the two studies.

Measuring social and emotional skills in elementary students: Development of self-report Likert, situational judgment test, and forced choice items

As social and emotional learning (SEL) continues to gain popularity, the need for high-quality social and emotional skill assessments also increases. We conducted two studies to develop and validate items to measure social and emotional skills in third, fourth, and fifth grade students. The Big Five personality framework served as an assessment framework for image-enhanced Likert items, situational judgment test items, and forced choice items. Results from Study 1 (n = 1047) provided promising reliability and validity evidence, as well as concrete recommendations for item revisions. Study 2 (n = 826) was conducted with a revised item pool and demonstrated improved reliability and validity. Taken together, results provided initial support that social and emotional skills can be validly and reliably measured in elementary-aged students using innovative item types.

Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance

AbstractRecent research challenges the importance of situation descriptions for situational judgment test (SJT) performance. This study contributes to resolving the ongoing debate on whether SJTs are situational measures, by incorporating findings on person × situation interactions into SJT research. Specifically, across three studies (NTotal = 1,239), we first tested whether situation construal (i.e., the individual perception of situations in SJTs) predicts responses to SJT items. Second, we assessed whether the relevance of situation construal for SJT performance depends on test elements (i.e., situation descriptions and response options) and item features (i.e., description‐dependent vs. description‐independent SJT items). Lastly, we determined whether situation construal has incremental validity for job‐related criteria over and above SJT performance. The results showed that, for most SJT items, situation construal significantly contributed to SJT performance, even if only response options were available. This was also true for SJT items that are significantly more difficult to solve when situation descriptions are omitted (i.e., description‐dependent SJT items). Finally, situation construal explained variance in relevant criteria over and above SJT performance. Despite recent efforts to reconceptualize SJTs, our results suggest that they can still be viewed as situational measures. However, situation descriptions may be less crucial for these underlying situational processes. Theoretical and practical implications are discussed.

The Undergraduate Situational Judgment Test: An Active Learning Project for Teaching Selection Tool Effectiveness

This article outlines an active learning project that gives students hands-on experience in developing an undergraduate situational judgment test. The five-part activity models the process for constructing a situational judgment test—a tool commonly used for employee selection in organizations. The project is designed to help students assimilate concepts associated with the development of and decision to use different selection tools. Student reactions indicate that the exercise promotes student attention and engagement, stimulates learning about employee selection tools and concepts, and helps students understand the process of constructing and validating selection methods. Student development is a positive side effect of the exercise as constructing undergraduate situational judgment test items requires participants to explore and evaluate solutions to the problems faced by the typical undergraduate.

A process model of situational judgment test responding

Better understanding respondents' cognitions as they respond to situational judgment test (SJT) items and isolating which elements of knowledge they measure may allow psychologists to develop more predictive SJT items with greater ease. Consequently, we present a theoretical framework outlining the thought processes individuals engage in as they respond to SJT items. We review interactionist theories explaining how these models have shaped the understanding of how personality traits affect behavior and discuss the recent scholarly debate regarding the role of the situation in SJTs. We then describe our proposed tripartite model of the psychological processes test takers may engage in as they respond to SJT items. Finally, we conclude by discussing directions for future research and potential avenues for expanding the proposed model.

Removing situation descriptions from situational judgment test items: Does the impact differ for video‐based versus text‐based formats?

Recent research has shown that many text‐based situational judgment test (SJT) items can be solved even when the situational descriptions in the item stems are not presented to test takers. This finding challenges the traditional view of SJTs as low‐fidelity simulations that rely on ‘situational’ (context‐dependent) judgment. However, media richness theory and construal level theory suggest that situation descriptions presented in a richer and more concrete format (video format) will reduce uncertainty about inherent requirements and facilitate the perception that the situation is taking place in the here and now. Therefore, we hypothesized that situational judgment would be more important in video situation descriptions than in text situation descriptions. We adapted a leadership SJT to realize a 3 (situation description in the item stem: video vs. text vs. none) × 2 (response format: video response options vs. text response options) between‐subjects design (N = 279). Participants were randomly assigned to one of the six conditions. The removal of video‐based situation descriptions in item stems led to an equivalent decrease in SJT scores as the removal of text‐based situation descriptions in item stems (video‐based version: Cohen's d = 0.535 vs. text‐based version: Cohen's d = 0.531). SJT scores were also contingent on the presentation format of both situation descriptions and response options: The highest scores were observed when situation descriptions and response options were presented in the same format. Implications for SJT theory and research are discussed.Practitioner points The presentation format did not moderate the effect of omitting situation descriptions in SJTs – that is, the context dependency of SJT performance did not increase when the SJT was administered in a video‐based rather than a text‐based format. The elimination of situation descriptions in item stems had a medium effect on overall test scores: SJT scores were significant lower without situation descriptions in comparison to SJT scores with situation descriptions (video‐based version: Cohen's d = 0.535 vs. text‐based version: Cohen's d = 0.531). It is important to match the stimulus and response formats in SJTs.

Situational Judgment Tests as Measures of 21st Century Skills: Evidence across Europe and Latin America

Over the years, various governmental, employment, and academic organizations have identified a list of skills to successfully master the challenges of the 21st century. So far, an adequate assessment of these skills across countries has remained challenging. Limitations inherent in the use of self-reports (e.g., lack of self-insight, socially desirable responding, response style bias, reference group bias, etc.) have spurred on the search for methods that could complement or even substitute self-report inventories. Situational judgment tests (SJTs) have been proposed as one of the complements/alternatives to the traditional self-report inventories. SJTs are low-fidelity simulations that confront participants with multiple domain-relevant situations and request to choose from a set of predefined responses. Our objectives are twofold: (a) outlining how a combined emic-etic approach can be used for developing SJT items that can be used across geographical regions and (b) investigating whether SJT scores can be compared across regions. Our data come from Laureate International Universities (N = 5,790) and comprise test-takers from Europe and Latin America who completed five different SJTs that were developed in line with a combined emic-etic approach. Results showed evidence for metric measurement invariance across participants from Europe and Latin America for all five SJTs. Implications for the use of SJTs as measures of 21st century skills are discussed.

Unintended Consequences: Narrowing SJT Usage and Losing Credibility With Applicants

Lievens and Motowidlo (2016) present a convincing case for why situational judgment tests (SJTs) should be developed specifically to measure general domain knowledge, but I have two concerns regarding the use of SJTs in psychological research and practical settings if the reconceptualization of SJTs offered by the authors is adopted to the exclusion of other current approaches. The first concerns abandoning SJTs as a helpful job analysis tool if we encourage intentional conflation of trait expression with job effectiveness in the development of SJT items. Second, diluting the situational component of SJTs may reduce their acceptance as selection tools by job applicants and practitioners in organizations.

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States.

Purpose:The situational judgment test (SJT) shows promise for assessing the non-cognitive skills of medical school applicants, but has only been used in Europe. Since the admissions processes and education levels of applicants to medical school are different in the United States and in Europe, it is necessary to obtain validity evidence of the SJT based on a sample of United States applicants.Methods:Ninety SJT items were developed and Kane’s validity framework was used to create a test blueprint. A total of 489 applicants selected for assessment/interview day at the University of Utah School of Medicine during the 2014-2015 admissions cycle completed one of five SJTs, which assessed professionalism, coping with pressure, communication, patient focus, and teamwork. Item difficulty, each item’s discrimination index, internal consistency, and the categorization of items by two experts were used to create the test blueprint.Results:The majority of item scores were within an acceptable range of difficulty, as measured by the difficulty index (0.50-0.85) and had fair to good discrimination. However, internal consistency was low for each domain, and 63% of items appeared to assess multiple domains. The concordance of categorization between the two educational experts ranged from 24% to 76% across the five domains.Conclusion:The results of this study will help medical school admissions departments determine how to begin constructing a SJT. Further testing with a more representative sample is needed to determine if the SJT is a useful assessment tool for measuring the non-cognitive skills of medical school applicants.

Putting judging situations into situational judgment tests: evidence from intercultural multimedia SJTs.

Although the term situational judgment test (SJT) implies judging situations, existing SJTs focus more on judging the effectiveness of different response options (i.e., response judgment) and less on how people perceive and interpret situations (i.e., situational judgment). We expand the traditional SJT paradigm and propose that adding explicit assessments of situational judgment to SJTs will provide incremental information beyond that provided by response judgment. We test this hypothesis across 4 studies using intercultural multimedia SJTs. Study 1 uses verbal protocol analysis to discover the situational judgments people make when responding to SJT items. Study 2 shows situational judgment predicts time-lagged, peer-rated task performance and interpersonal citizenship among undergraduate seniors over and above response judgment and other established predictors. Study 3 shows providing situational judgment did not affect the predictive validity of response judgment. Study 4 replicates Study 2 in a working adult sample. We discuss implications for SJT theory as well as the practical implications of putting judging situations back into SJTs.

On the Limitations of Using Situational Judgment Tests to Measure Interpersonal Skills: The Moderating Influence of Employee Anger

Many authors have suggested that situational judgment tests (SJTs) are useful tools for assessing applicants because SJT items can be written to assess a number of job‐related knowledges, skills, abilities and other characteristics (KSAOs). However, SJTs may not be appropriate for measuring certain KSAOs for some applicants. We posit that using SJTs to measure interpersonal skills may lead to invalid inferences about applicants with higher levels of angry hostility (AH), and thus, AH should moderate the relation between interpersonally oriented SJTs and job performance. Three studies, using samples of healthcare workers (n = 225), police officers (n = 54), and medical doctors (n = 92), provided support for hypotheses in that that relations between SJT scores and performance criteria were significantly weaker among employees higher in AH compared to those lower in AH. In addition, none of the other facets of neuroticism tested (self‐consciousness, anxiety, depression, immoderation, or vulnerability to stress) consistently moderated SJT validity, providing support for the uniqueness of AH. Implications for practice, and for future research studying the relations between interpersonal skills as measured by SJTs and job performance, are discussed.

Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.

Estimating Trait and Situational Variance in a Situational Judgment Test

In organizational research, situational judgment tests (SJTs) consistently demonstrate incremental validity, yet our theoretical understanding of SJTs is limited. Our knowledge could be advanced by decomposing the variance of SJT items into trait variance and situation variance; we do that by applying statistical methods used to analyze multitrait–multimethod matrices. A college-student sample (N = 2,747) was administered an SJT of goal orientation traits (i.e., mastery, performance-approach, and performance-avoid). Structural equation modeling was used to estimate the proportions of item variance to attributable to situational differences (across students) and to trait-based differences in students (across situations). Situation factors accounted for over three times the amount of variance as did individual difference factors. We conclude with general implications for the design of SJTs in organizational research.

Impact of Elaboration on Responding to Situational Judgment Test Items

Although faking has been identified as a potential problem in situational judgment tests (SJTs), no studies have investigated proactive approaches for controlling faking in SJTs. Therefore, this study examined the impact of elaboration on responding to SJT items. Elaboration was operationalized as reason‐giving. Two hundred and forty‐seven master students were assigned to either an honest or a fake condition, and to a non‐elaboration or an elaboration condition. Results showed that elaboration decreased the effect of faking for items with high familiarity. Elaboration on familiar items also decreased the percentage of fakers in the top of the distribution. Next, participants in the elaboration condition rated the SJT significantly higher in terms of allowing them to present themselves more realistically and to demonstrate their knowledge, skills, and abilities. Finally, there were no significant differences in participants' satisfaction with the SJT across the elaboration and non‐elaboration condition.