Situational Judgment Tests Scores Research Articles

Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders

Intercultural competence is a goal of many educational and training programs for military leaders who, when deployed overseas, are required to think critically and make strategic decisions in culturally complex environments. Critical thinking skills are not only essential leadership tools, but are also the keys to development of intercultural competence. The primary objective of this study is to gain a better understanding of the relationship between critical thinking and intercultural competence. Another objective is to determine which intercultural competencies learners find most challenging. Research methodology included a thorough examination of the curriculum, content, testing data, and end-of-course survey results of a non-credit, self-paced, instructorless online course for 2241 mid-career Air Force officers. Assessing critical thinking and intercultural competence development is a challenge in any educational context, but especially in online, self-paced courses without the benefit of teacher or student interaction. Situational judgment tests (SJTs) are a practical technique for assessing progress. Results revealed a significant relationship between critical thinking and intercultural competence SJT scores. Item analysis indicated that some scenarios were more challenging than others from an intercultural perspective. According to the findings, certain cultural differences challenge the decision-making ability of military officers in high-stress, high-visibility situations. These include culture-general knowledge (Mindset), empathic communication skills (Skillset), and a curious, open, and nonjudgmental attitude (Heartset). Participants scored significantly higher, however, when analyzing SJTs situated in more mundane intercultural workplace situations. The study’s findings support the efficacy of utilizing SJTs to develop intercultural competence, especially in online learning environments.

Effects of Organizationally Endorsed Coaching on Performance and Validity of Situational Judgment Tests

There is growing interest in organizationally provided or organizationally endorsed coaching. However, little is known about the effects of such coaching on test scores in operational settings. This study reports on an examination of such a program in the context of the use of a situational judgment test (SJT) for medical school admissions. We examine the effects of multiple types of coaching methods on SJT scores and on their construct‐related and predictive validities. Results suggest that (1) commercial coaching techniques may not be as effective as previously thought, whereas organizationally provided methods may be more effective, and that (2) the criterion‐related validity of the SJT scores is not degraded by the availability of coaching. Generally, this study illustrates that concerns about potential unfairness of coaching can be countered by making effective coaching available to all examinees, in the form of organizationally endorsed coaching.

Does a high ranking mean success in the Situational Judgement Test?

The selection of medical students to the foundation programme has undergone several changes since its introduction in 2005, with the latest being the introduction of the Situational Judgement Test (SJT) in 2013. The SJT, a 2-hour exam that uses a multiple-choice format to assess an individual's judgement when presented with clinically related scenarios, now accounts for 50 per cent of the application process. The remaining 50 per cent is made up of the Educational Performance Measure (EPM) score, which includes the medical student's performance at medical school, allocated by points attributed according to decile ranking. Coming out of its pilot year, there is little research into the results of the SJT. This project aims to discover whether there is a correlation between high-performing students who succeeded at medical school (i.e. by decile ranking) and those who scored high in the SJT. This project aims to discover whether there is a correlation between high- performing students and those whoscored high in the SJT METHOD: A survey was circulated to fifth-year medical students across the UK applying to the foundation programme in 2013, and who had sat the pilot year of the SJT. Students were asked to enter numerical values for their EPM and SJT scores. A total of 239 students from 12 UK medical schools responded to the questionnaire. In addition to their EPM, academic and SJT scores, all participants provided data on gender, duration of course (i.e. 4 or 5 years) and whether or not additional resources were used for the SJT. There was no correlation between the SJT and the academic scores (Spearman's = 0.0458; probability of any relationship = 0.4865).

Are predictions based on situational judgement tests precise enough for feedback in leadership development?

Situational judgement tests (SJTs) have much to recommend their use for personnel selection, but partly because of their low reliability, the role of SJTs in behavioural training is largely unexplored. However, research showing that SJTs cannot measure homogenous constructs very well is based exclusively on internal analyses, for example, alpha reliability and factor analysis. In this study, we investigated whether patterns of correlations with external criteria could be used to show that SJT dimension scores are homogenous enough for feedback purposes in leadership development. A multidimensional SJT was designed for 268 high-potential leaders on a development programme and used in conjunction with a multisource feedback instrument that measured the same competency framework. The SJT was criterion-keyed against the multisource feedback instrument using an N-fold cross validation strategy. Convergent and divergent correlations between the SJT scores and corresponding multisource dimension scores suggested that SJT scores can be constructed in a way that permits dimension-level feedback that would be useful in leadership development.

On the Limitations of Using Situational Judgment Tests to Measure Interpersonal Skills: The Moderating Influence of Employee Anger

Many authors have suggested that situational judgment tests (SJTs) are useful tools for assessing applicants because SJT items can be written to assess a number of job‐related knowledges, skills, abilities and other characteristics (KSAOs). However, SJTs may not be appropriate for measuring certain KSAOs for some applicants. We posit that using SJTs to measure interpersonal skills may lead to invalid inferences about applicants with higher levels of angry hostility (AH), and thus, AH should moderate the relation between interpersonally oriented SJTs and job performance. Three studies, using samples of healthcare workers (n = 225), police officers (n = 54), and medical doctors (n = 92), provided support for hypotheses in that that relations between SJT scores and performance criteria were significantly weaker among employees higher in AH compared to those lower in AH. In addition, none of the other facets of neuroticism tested (self‐consciousness, anxiety, depression, immoderation, or vulnerability to stress) consistently moderated SJT validity, providing support for the uniqueness of AH. Implications for practice, and for future research studying the relations between interpersonal skills as measured by SJTs and job performance, are discussed.

Foundation allocation error affects thousands of medical students

Final year medical students have been left confused and distressed after learning a day after they had received their foundation school allocation that the offers may be withdrawn. Over 7200 medical students are having their situational judgment test (SJT) scores rechecked after an error in the marking of these papers was discovered.1 Students received details of their foundation school allocations on 25 February and were told about the error on 26 February. Scores on the SJT, a multiple choice examination that requires candidates to rank clinical scenarios, are combined with an educational performance score to decide foundation school allocation. Students will be given new SJT scores and foundation school allocations at 9 am on Friday 8 March. Clare Smyth, a final year medical student at the University of Glasgow, said that there had …

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.

Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Situational judgment tests (SJTs) pose unique cognitive demands on test takers in that, when presented in written form, they require a great deal of reading and cognitive effort. Because of this cognitive demand, responses to test items toward the end of the test may be influenced by an order effect produced by responding to a large quantity of previous test items. This construct‐irrelevant order effect may increase measurement error and threaten the validity of SJT scores. To test this phenomenon, data were obtained from 1,089 applicants who had completed a lengthy SJT as part of a selection process for an hourly safety and surveillance job at a large international corporation. Results showed that local item dependence, item difficulty, and the rate of omitted responses all increased when items were placed toward the end. The order effect alone was not strong enough to influence subgroup mean score differences in the second half of the test. However, this effect did vary by race: African‐Americans were most strongly affected by the order effect, followed by Caucasians, in their number of omitted responses. Implications and future research of this effect for SJTs and similar types of assessments are discussed.

Contextual Effects on SJT Responses: An Examination of Construct Validity and Mean Differences Across Applicant and Incumbent Contexts

This study examines two important issues with situational judgment tests (SJTs) in operational contexts. First, we examine whether the construct validity of SJTs changes across applicant and incumbent contexts. Second, we examine whether means differ across contexts. Using a model of the SJT response process as a guide, results from 12 predictive and concurrent criterion-related studies show that the construct validity inferences of SJTs differ somewhat by context, such that cognitive ability was more strongly related to SJT scores in incumbent contexts than applicant contexts. No significant contextual differences were found with personality. However, in contrast to most noncognitive predictors, applicants score lower on SJTs than incumbents. Cumulatively, these results suggest that the response processes used by respondents may differ by context, and inferences of SJT construct validity may be partly context specific.

Effects of Response Instructions on Faking a Situational Judgment Test

A situational judgment test (SJT) and a Big 5 personality test were administered to 203 participants under instructions to respond honestly and to fake good using a within-subjects design. Participants indicated both the best and worst response (i.e., Knowledge) and the most likely and least likely response (i.e., Behavioral Tendency) to each situation. Faking effect size for the SJT Behavioral Tendency response format was (d=.34) when participants responded first under honest instructions and (d=.15) when they responded first under faking instructions. Those for the Big 5 dimensions ranged from d=.26 to d=1.0. For the Knowledge response format results were inconsistent. Honest condition Knowledge SJT scores were more highly correlated with cognitive ability (r=.56) than were Behavioral Tendency SJT scores (r=.38). Implications for researchers and practitioners are discussed.

The Development and Validation of a Situational Judgement Test for Security Officers

This study reports on a novel tool for improving medical center security officer selection decisions. A low-fidelity situational judgement test (SJT) was developed and content/criterion-validated. A critical incident job analysis surveying 18 different organizations was conducted, and used to develop both the 56-item SJT and the eight-dimension performance appraisal. These instruments showed acceptable reliability, and security experts were in high agreement as to the relevance of the SJT incident items and the effectiveness of incident response options. Using a sample of 163 medical center security officers from the 18 organizations across the United States, a concurrent validation study was conducted. Results showed that total SJT scores correlated significantly with overall performance scores and were found to be significantly associated with each performance appraisal dimension rating. Racial group disparity in test performance favoring White persons and Hispanics was discovered. Finally, the utility of the test and the limitations of the study are discussed.