Situational Judgment Tests Scores Research Articles

DiffErential attainment and Factors AssoCiated with Training applications and Outcomes (DE FACTO) study: Trauma & Orthopaedic surgery in the UK.

The aims of this study were to describe the demographic, socioeconomic, and educational factors associated with core surgical trainees (CSTs) who apply to and receive offers for higher surgical training (ST3) posts in Trauma & Orthopaedics (T&O). Data collected by the UK Medical Education Database (UKMED) between 1 January 2014 and 31 December 2019 were used in this retrospective longitudinal cohort study comprising 1,960 CSTs eligible for ST3. The primary outcome measures were whether CSTs applied for a T&O ST3 post and if they were subsequently offered a post. A directed acyclic graph was used for detecting confounders and adjusting logistic regression models to calculate odds ratios (ORs), which assessed the association between the primary outcomes and relevant exposures of interest, including: age, sex, ethnicity, parental socioeconomic status (SES), domiciliary status, category of medical school, Situational Judgement Test (SJT) scores at medical school, and success in postgraduate examinations. This study followed STROBE guidelines. Compared to the overall cohort of CSTs, females were significantly less likely to apply to T&O (OR 0.37, 95% CI 0.30 to 0.46; n = 155/720 female vs n = 535/1,240 male; p < 0.001). CSTs who were not UK-domiciled prior to university were nearly twice as likely to apply to T&O (OR 1.99, 95% CI 1.39 to 2.85; n = 50/205 vs not UK-domiciled vs n = 585/1,580 UK-domiciled; p < 0.001). Age, ethnicity, SES, and medical school category were not associated with applying to T&O. Applicants who identified as 'black and minority ethnic' (BME) were significantly less likely to be offered a T&O ST3 post (OR 0.70, 95% CI 0.51 to 0.97; n = 165/265 BME vs n = 265/385 white; p = 0.034). Differences in age, sex, SES, medical school category, and SJT scores were not significantly associated with being offered a T&O ST3 post. There is an evident disparity in sex between T&O applicants and an ethnic disparity between those who receive offers on their first attempt. Further high-quality, prospective research in the post-COVID-19 pandemic period is needed to improve equality, diversity, and inclusion in T&O training.

“Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission

We explored the transformative impact of ChatGPT on applicants' responses and performance in situational judgement tests (SJTs), as well as the role played by faking-prevention mechanisms, in two complementary studies. Study 1 examined how the availability of ChatGPT influenced response content and performance of real applicants (N = 107,805), who completed an SJT for admission before vs. after the release of the technology. We found only small differences in content (e.g., slightly less “authentic” words used) and performance (slight score improvements when controlling for response length, no differences otherwise). In Study 2, we used an experimental approach with (N = 138) Prolific participants completing a mock SJT, while being instructed to use ChatGPT when responding (vs. use online resources or no resources). We found only slightly higher SJT scores for the ChatGPT users, but no difference in response content. Additionally, GPTZero (i.e., a popular AI detection tool) struggled to detect ChatGPT content, and generated many false positives, in both studies. This research advances our understanding of how the release and popularization of ChatGPT can influence applicant behaviors. Given the “arms race” nature of applicant selection, they also highlight the importance of designing assessments to prevent or limit faking. Yet, the ever-evolving nature of AI calls for continuous research on the topic.

Exploring the Utility and Fairness of a Multi-institutional Situational Judgment Test

ObjectiveTo pilot a situational judgment test (SJT) developed to assess empathy, integrity, and teamwork and explore differences in performance between first-year (P1) and fourth-year (P4) student pharmacists based on gender, race or ethnicity, and geographical region. MethodsAn SJT was developed to assess empathy, integrity, and teamwork, using best practices. The SJT was piloted tested with P1 and P4 student pharmacists at 5 institutions. Scores were adjusted for elevation and scatter to minimize the subgroup differences in response styles. ResultsThe unadjusted mean SJT scores for P1 and P4 student pharmacists were 79.7 % and 80.7 %, respectively. The adjusted scores for P1 and P4 student pharmacists were 86.3 % and 87.2 %, respectively. Small differences were observed across institutions which can likely explained by multiple factors. The mean scores were higher among students who were older, female, identified as White, and spoke English as their first language. The subgroup differences persisted even after adjusting scores for elevation and scatter. ConclusionPerformance on an SJT designed to assess empathy, integrity, and teamwork was similar to others in the literature; however, differences in performance were observed across subgroups. Additional research is needed to determine whether these differences reflect test bias or other possible factors.

Predicting Medical Student Performance With a Situational Judgment Test in Admissions.

To explore correlations between AAMC situational judgment test (SJT) scores, other admissions data, and learners' medical school performance. First- and second-year medical students from 8 U.S. MD-granting medical schools completed a prototype version of the AAMC SJT in 2017. Outcomes included research-only faculty ratings of student performance, final course grades, and faculty evaluations of student performance, 2017-2018 and 2018-2019 academic years. Bivariate correlations were used to investigate the relationship between SJT scores and student performance outcomes and hierarchical regressions to investigate whether SJT scores provided incremental validity over MCAT total scores and cumulative undergraduate grade point averages (UGPAs) for predicting student performance outcomes. In general, there were small positive correlations with research-only faculty ratings from the first year of medical school, with the highest for social skills/service orientation ( rcorrected = .33, P < .05). Correlations were higher, with the highest for social skills/service orientation and cultural competence ( rcorrected = .33 and .36, respectively, P < .05) in the second year in medical school. SJT scores improved prediction of research-only faculty ratings over MCAT total scores and UGPAs for reliability and dependability/capacity for improvement, cultural competence, social skills/service orientation, and the overall composite score in the first year and for resilience and adaptability, social skills/service orientation, cultural competence, and the overall composite score in the second year. SJT scores demonstrated small correlations with course grades ( rsample-weighted = .10, P = ns) and faculty evaluations related to professionalism skills ( rsample-weighted = .14, P < .05); however, MCAT total scores explained most of the variance associated with course outcomes. These studies provide initial evidence that SJT scores may add value to the medical school admissions process because scores were related to faculty ratings of professional behaviors and provided unique information relative to MCAT scores and UGPAs.

Designing a Situational Judgment Test for Use in Medical School Admissions.

It has long been acknowledged that professional competencies are required for success in medical school, residency training, and medical practice. Over the last decade, medical schools have begun to introduce standardized assessments of professional competencies, but many still rely on interviews to assess these competencies, which occur after about half of the applicant pool has already been screened out. In this article, the authors discuss the development, evaluation, and launch of the Association of American Medical Colleges (AAMC) situational judgment test (SJT) for use in medical school admissions. The AAMC SJT is designed to assess an examinee's understanding of effective and ineffective behaviors related to the core competencies for entering medical students, including service orientation, social skills, cultural competence, teamwork, ethical responsibility to self and others, reliability and dependability, resilience and adaptability, and capacity for improvement. The authors evaluate the evidence for the need for SJTs in medical school admissions by exploring common derailers in medical school, gaps in the admissions process regarding information about professional competencies, and the challenge of conducting holistic review in a high-volume context. They summarize existing research from the employment, international medical education, and residency selection contexts suggesting that SJT scores are positively associated with subsequent performance and may add value to the admissions process. The authors discuss 5 goals that were the foundation for developing the AAMC SJT: (1) assess the professional competencies needed for success in medical school using a proven method, (2) enable holistic review in a high-volume admissions context, (3) create and share a program of research to support the appropriate use of SJT scores, (4) signal the need for preparation in professionalism to learners, and (5) balance the need for a new assessment with minimizing the burden and risk for applicants.

422 Utilizing Situational Judgment Test in Neurosurgery Residency Online Interviews

INTRODUCTION: Neurosurgery residency selection is heavily reliant on pre-residency factors and unstructured interviews. These metrics, however, have not been shown to reliably predict residency performance, and have potential to introduce bias and discrimination. Outside of medicine, the science of selection is much better developed, and methods such as structured interviews, situational judgment tests (SJT), and personality assessments are routinely used with robust evidence for efficacy and reliability at predicting job performance. METHODS: We designed a set of 4 SJT questions mimicking common situations a neurosurgery resident would encounter during residency. Each question had individualized grading rubrics scored on a scale of 1 to 4. For the 2021 residency application cycle, our program incorporated these SJT questions as part of the online interview process. Four volunteer faculty interview pairs delivered one question each for every candidate, and recorded the scores. Other traditional evaluation metrics from the candidate’s application and interview were also collected post interview. RESULTS: All 44 interview candidates applying to our program during the 2021 cycle were assessed. Due to time constraints, the number of questions each candidate responded to varied from 2 to 3. The average score was 3.6 (SD 0.44, range 2-4). The SJT score was not associated with USMLE Step 1 score (p = 0.31), number of publications (p = 0.42), or letter of recommendation score (p = 0.21). The SJT scores had a moderate positive correlation with personality on interview (r = 0.42, p &lt; 0.01), and overall interview score (r = 0.30, p = 0.04). CONCLUSIONS: It is feasible to incorporate SJT questions into the neurosurgery resident online interview process. They may provide objective metrics independent of academic and research abilities. Further investigations with long term follow up is needed to see if SJT can be validated and correlated with residency performance.

The Relationship Between a Multiple Mini-interview and Situational Judgment Test for Admissions

Objective. To explore the relationship between a multiple mini-interview (MMI) and situational judgment test (SJT) designed to evaluate nonacademic constructs.Methods. A 30-question ranked-item SJT was developed to test three constructs also measured by MMIs during a pharmacy school's admissions process. First-year pharmacy students were invited to complete the SJT in fall 2020. One hundred four students took the SJT (82.5% response rate), with 97 (77% of possible participants) having MMI scores from the admissions process. Descriptive statistics and other statistical analyses were used to explore the psychometric properties of the SJT and its relationship to MMI scores.Results. Seventy-four percent of students identified as female (n=72), and 11.3% identified with an underrepresented racial identity (n=11). The average age, in mean (SD), was 21.8 (2.1) years. Students' mean (SD) scores were 85.5 (3.1) (out of 100 points) on the SJT and 6.1 (1.0) (out of 10 points) on the MMI. Principal components analysis indicated that the SJT lacked construct validity and internal reliability. However, reliability of the entire SJT instrument provided support for using the total SJT score for analysis (α=.63). Correlations between total SJT and MMI scores were weak (r p<0.29).Conclusion. Results of this study suggest that an SJT may not be a good replacement for the MMI to measure distinct constructs during the admissions process. However, the SJT may provide useful supplemental information during admissions or as part of formative feedback once students are enrolled in a program.

Time for preference-informed foundation allocation?

Successful completion of year 1 of the UK Foundation Programme is a General Medical Council requirement that newly qualified doctors must achieve in order to gain full registration for licence to practise in the UK. We present compelling evidence that both sections of the UK Foundation Programme allocation process, consisting of the Educational Performance Measure and Situational Judgement Test scores, are not fit for purpose. The ranking process drives competitive behaviours among medical students and undermines NHS teamworking values. Furthermore, data from 2013-2020 show that UK minority ethnic students consistently receive significantly lower SJT scores than White students. The current process in the UK allocates lower ranked students, who often need more academic and social support, to undersubscribed regions. This can lead to vacancies in less popular regions, ultimately worsening health inequality. A preference-informed allocation process will improve trainee access to support and help retain trainees in underserved regions. We aim to summarise the flaws of the current system and report a potential radical solution.

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study.

To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods. In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods. Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P &lt; .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear. Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

Assessing distinguishable social skills in medical admission: does construct-driven development solve validity issues of situational judgment tests?

BackgroundSocial skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJTs) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs.MethodThe final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior.ResultsThe proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures.ConclusionsResults suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended.

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

BackgroundInternationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile.MethodsThirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508).ResultsThe SJT demonstrated good internal consistency (Cronbach’s alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity.ConclusionsThis is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

Situational judgement test performance and subsequent misconduct in medical students.

IntroductionSituational judgement tests (SJTs) have been widely adopted, internationally, into medical selection. It was hoped that such assessments could identify candidates likely to exhibit future professional behaviours. Understanding how performance on such tests may predict the risk of disciplinary action during medical school would provide evidence for the validity of such SJTs within student selection. It would also inform the implementation of such tests within student recruitment.MethodsThis cohort study used data for 6910 medical students from 36 UK medical schools who sat the University Clinical Aptitude Test (UCAT) SJT in 2013. The relationship between SJT scores at application and the risk of subsequent disciplinary action during their studies was modelled. The incremental ability of the SJT scores to predict the risk of disciplinary action, above that already provided by UCAT cognitive test scores and secondary (high) school achievement, was also evaluated in 5535 of the students with information available on this latter metric.ResultsTwo hundred and ten (3.05%) of the students in the cohort experienced disciplinary action. The risk of disciplinary action reduced with increasing performance on the admissions SJT (odds ratio (OR) 0.80, 95% confidence interval (CI) 0.69 to 0.92, p = 0.002). This effect remained similar after adjusting for cognitive performance and prior academic attainment (OR 0.77, 95% CI 0.65 to 0.92, p = 0.004). The overall estimated effect‐size was small (Cohen's d = 0.08) and no evidence of ‘threshold’ effects were observed for the SJT scores and risk of disciplinary action.ConclusionsPerformance on admissions SJTs can, at least modestly, incrementally predict the risk of subsequent disciplinary action, supporting their use in this context. However, for this SJT and outcome, there did not seem a distinct threshold score above which the risk of disciplinary action disproportionately increased. This should be considered when using the scores within medical selection.

Is Academic Attainment or Situational Judgment Test Performance in Medical School Associated With the Likelihood of Disciplinary Action? A National Retrospective Cohort Study.

Disciplinary action imposed on physicians indicates their fitness to practice medicine is impaired and patient safety is potentially at risk. This national retrospective cohort study sought to examine whether there was an association between academic attainment or performance on a situational judgment test (SJT) in medical school and the risk of receiving disciplinary action within the first 5 years of professional practice in the United Kingdom. The authors included data from the UK Medical Education Database for 34,865 physicians from 33 U.K. medical schools that started the UK Foundation Programme (similar to internship) between 2014 and 2018. They analyzed data from 2 undergraduate medical assessments used in the United Kingdom: the Educational Performance Measure (EPM), which is based on academic attainment, and SJT, which is an assessment of professional attributes. The authors calculated hazard ratios (HRs) for EPM and SJT scores. The overall rate of disciplinary action was low (65/34,865, 0.19%) and the mean time to discipline was 810 days (standard deviation [SD] = 440). None of the physicians with fitness to practice concerns identified as students went on to receive disciplinary action after they qualified as physicians. The multivariate survival analysis demonstrated that a score increase of 1 SD (approximately 7.6 percentage points) on the EPM reduced the hazard of disciplinary action by approximately 50% (HR = 0.51; 95% confidence interval [CI]: 0.38, 0.69; P < .001). There was not a statistically significant association between the SJT score and the hazard of disciplinary action (HR = 0.84; 95% CI: 0.62, 1.13; P = .24). An increase in EPM score was significantly associated with a reduced hazard of disciplinary action, whereas performance on the SJT was not. Early identification of increased risk of disciplinary action may provide an opportunity for remediation and avoidance of patient harm.

Rule‐based versus feedback‐based coaching for situational judgment tests

AbstractThe use of situational judgment tests (SJTs) in high‐stakes selection contexts has led to the emergence of commercial coaching programs aimed at increasing individuals’ performance. The present study uses a pre–post experimental design to investigate coaching effects on an SJT’s mean score and relationships with external variables. General strategies with which to guide how respondents answer SJT questions were developed for rule‐based coaching. Item‐level feedback was provided for feedback‐based coaching. Results suggest that (a) both rule‐based and feedback‐based coaching are similarly effective at increasing mean SJT scores, (b) coaching does not detract from the SJT’s ability to correlate with other variables, and (c) knowledge of ineffective behaviors are more coachable than knowledge of effective behaviors.

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio-economic status on situational judgement test performance.

BackgroundSituational judgement tests (SJTs) measure non‐cognitive attributes and have recently drawn attention as a selection method for initial teacher education programmes. To date, very little is known about adverse impact in teacher selection SJT performance.AimsThis study aimed to shed light on adverse effects of gender, ethnicity, and socio‐economic status (SES) on SJT scores, by exploring both main effects and interactions, and considering both overall SJT performance and separate SJT domain scores (mindset, emotion regulation, and conscientiousness).SampleA total of 2,808 prospective teachers from the United Kingdom completed the SJTs as part of the initial stage of selection into a teacher education programme.MethodsIn addition to SJT scores, the variables gender (female vs. male), ethnicity (majority group vs. minority group), and home SES background (higher SES status vs. lower SES status) were used in the analyses. Regression models and moderated regression models were employed.Results and conclusionsResults from the regression models revealed that gender effects (females scoring higher than males) were restricted to emotion regulation, while ethnicity effects (ethnic majority group members scoring higher than ethnic minority group members) emerged for SJT overall scores and all three domains. Moderated regression modelling results furthermore showed significant interactions (gender and ethnicity) for SJT overall scores and two domains. Considering the importance of reducing subgroup differences in selection test scores to ensure equal access to teacher education, this study’s findings are a critical contribution. The partially differentiated results for overall vs. domain‐specific scores point towards the promise of applying a domain‐level perspective in research on teacher selection SJTs.

Predictors of Pharmacy Student Performance on Written and Clinical Examinations in a Flipped Classroom Curriculum

Objective. To examine the effects of student demographics, prior academic performance, course engagement, and time management on pharmacy students' performance on course examinations and objective structured clinical examinations (OSCEs).Methods. Study participants were one cohort of pharmacy students enrolled in a five-year combined Bachelor and Master of Pharmacy degree program at one institution. Variables included student demographics, baseline factors (language assessment and situational judgement test scores), prior academic performance (high school admission rank), course engagement, and student time management of pre-class online activities. Data were collected from course, learning management system, and institutional databases. Data were analyzed for univariate, bivariate, and multivariate associations (four linear regression models) between explanatory factors and outcome variables.Results. Three years of data on 159 pharmacy students were obtained and entered in the dataset. Significant positive predictors of OSCE communication performance included domestic (ie, Australian) student designation, higher baseline written English proficiency, and pre-class online activity completion. Positive predictors of OSCE problem-solving included workshop attendance and low empathy as measured by a baseline situational judgment test (SJT). Positive predictors of performance on year 2 end-of-course examinations included the Australian Tertiary Academic Rank, completing pre-class online activities prior to lectures, and high integrity as measured by an SJT.Conclusion. Several explanatory factors predicted pharmacy students' examination and OSCE performance in the regression models. Future research should continue to study additional contexts, explanatory factors, and outcome variables.

The role of Situations in Situational Judgment Tests: Effects on construct saturation, predictive validity, and applicant perceptions.

Recent theorizing and empirical evidence suggesting that Situational Judgment Tests (SJTs) are more context-independent than previously thought has sparked a debate about the role of situation descriptions in SJTs. To contribute to this debate and add to our understanding of how SJTs work, this article conceptually embeds SJT performance in a situation construal model and examines the effects of situation descriptions on the construct saturation and predictive validity of SJT scores, as well as on applicant perceptions. Across two studies (N = 1,092 and 578) and different SJTs, personality and cognitive ability were equally important determinants of SJT performance regardless of whether situation descriptions were presented or omitted. The effects of removing situation descriptions on the criterion-related validity of SJT scores differed depending on the breadth of the criteria. For predicting global job performance criteria (in-role performance and organizational citizenship behavior), SJT validity was not significantly affected, whereas it decreased for predicting more specific criteria (interpersonal adaptability, efficacy for teamwork). Finally, the effects of omitting situation descriptions in SJTs on applicant perceptions were either negligible or small. Implications for SJT theory, research, and design are discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

The association between Situational Judgement Test (SJT) scores and professionalism concerns in undergraduate medical education

Introduction: Situational judgement tests (SJTs) are widely used in selecting medical students and doctors. Emerging evidence suggests SJTs are capable of testing an individual’s ability to respond to role-relevant professionalism scenarios, however, evidence is lacking for their use in identifying students with concerning professional behaviours.This study aimed to determine the association between medical student professionalism-based SJT scores and recorded professionalism concerns during training.Methods: A cross-sectional study was conducted utilising SJT scores from second-year medical students and occurrences of student professionalism concerns. Concerns were reviewed and mapped to General Medical Council standards. Multivariate logistic regression was used to determine associations between SJT scores and professionalism concerns.Results: 247 students were included in the study. For every point increase in SJT score, students were 10% less likely to have multiple professionalism concerns [OR (95% CI) 0.90 (0.83–0.97); p = .007].Students scoring below 1 and 2 standard deviations from the mean score were 4 and 11-times more likely to have multiple concerns [OR (95% CI) 4.52 (1.12–18.25); p = .034] and [OR (95% CI) 11.45 (1.72–76.15); p = .012].Conclusion: Lower SJT scores were significantly associated with an increased risk of professionalism concerns. These findings support the potential for SJT exams to identify medical students that may require closer supervision and remediation during undergraduate education.

Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test

Construct: We investigated whether a situational judgment test (SJT) designed to measure professionalism in physicians predicts residents’ performance on (a) Accreditation Council for Graduate Medical Education (ACGME) competencies and (b) a multisource professionalism assessment (MPA). Background: There is a consensus regarding the importance of assessing professionalism and interpersonal and communication skills in medical students, residents, and practicing physicians. Nonetheless, these noncognitive competencies are not well measured during medical education selection processes. One promising method for measuring these noncognitive competencies is the SJT. In a typical SJT, respondents are presented with written or video-based scenarios and asked to make choices from a set of alternative courses of action. Interpersonally oriented SJTs are commonly used for selection to medical schools in the United Kingdom and Belgium and for postgraduate selection of trainees to medical practice in Belgium, Singapore, Canada, and Australia. However, despite international evidence suggesting that SJTs are useful predictors of in-training performance, end-of-training performance, supervisory ratings of performance, and clinical skills licensing objective structured clinical examinations, the use of interpersonally oriented SJTs in residency settings in the United States has been infrequently investigated. The purpose of this study was to investigate whether residents’ performance on an SJT designed to measure professionalism-related competencies—conscientiousness, integrity, accountability, aspiring to excellence, teamwork, stress tolerance, and patient-centered care—predicts both their current and future performance as residents on two important but conceptually distinct criteria: ACGME competencies and the MPA. Approach: We developed an SJT to measure seven dimensions of professionalism. During calendar year 2017, 21 residency programs from 2 institutions administered the SJT. We conducted analyses to determine the validity of SJT and USMLE scores in predicting milestone performance in ACGME core competency domains and the MPA in June 2017 and 3 months later in September 2017 for the MPA and 1 year later, in June 2018, for ACGME domains. Results: At both periods, the SJT score predicted overall ACGME milestone performance (r = .13 and .17, respectively; p < .05) and MPA performance (r = .19 and .21, respectively; p < .05). In addition, the SJT predicted ACGME patient care, systems-based practice, practice-based learning and improvement, interpersonal and communication skills, and professionalism competencies (r = .16, .15, .15, .17, and .16, respectively; p < .05) 1 year later. The SJT score contributed incremental validity over USMLE scores in predicting overall ACGME milestone performance (ΔR = .07) 1 year later and MPA performance (ΔR = .05) 3 months later. Conclusions: SJTs show promise as a method for assessing noncognitive attributes in residency program applicants. The SJT’s incremental validity to the USMLE series in this study underscores the importance of moving beyond these standardized tests to a more holistic review of candidates that includes both cognitive and noncognitive measures.

Using video- and text-based situational judgement tests for teacher selection: a quasi-experiment exploring the relations between test format, subgroup differences, and applicant reactions

ABSTRACT The present study examines whether video-based situational judgement test (SJT) formats provide benefits over “traditional” text-based SJTs. Focusing on three SJT conditions – two video-based conditions (with and without text), and a text-based condition – we investigated mean differences in applicant reactions and SJT scores, subgroup differences (ethnicity and gender), and relations between SJT scores and applicant reactions. Using a quasi-experimental design, 290 prospective teachers (56.6% female) were randomly assigned to one of the three SJT conditions. SJT scores did not significantly differ between conditions, but both video-based formats were perceived as more engaging than the text-based format. Results from a multigroup path model indicated that there were statistically significant gender effects for the text-based condition (females outperforming males), but not for the two video-based conditions. However, ethnicity effects (members from majority groups outperforming members from minority groups) occurred in all conditions. Differentiated patterns of relations were found between applicant reactions and SJT performance, with engagement statistically significantly predicting SJT performance in the video without text condition. Implications for future research and teacher selection practice are discussed.