Judgment Test Research Articles

Development and validation of situational judgement test for assessment of behavioural competencies required for effective medical practice in Nigeria

BackgroundThe personal qualities of health workers determine the way health services are provided to clients. Some key personal qualities (also called behavioural competencies) of physicians that contribute to quality healthcare delivery include ethical responsibility, empathy, patient-centeredness, diligence, good judgment, respectful, teamwork, team leadership/conflict management, ability to take correction and tolerance. In this study, we developed and validated clinical scenarios (dilemmas) for assessing priority behavioural competencies for medical practice in Nigeria.MethodsDrawing on prioritized competencies generated via a scoping review and nominal group technique (NGT) exercises in a previous study in the study series, Faculty members from the University of Nigeria Teaching Hospital were consulted to develop or adapt clinical scenarios that could be used to assess these competencies in a physician. The clinical scenarios and options were framed as situational judgement tests (SJTs) and these tests were administered to a random sample of 192 undergraduate and 111 postgraduate medical doctors in a tertiary hospital in Enugu State. Using Kane’s validity argument framework, we assessed scoring and generalization inferences of situational judgment tests (SJT) based on the developed scenarios.ResultScoring inference – difficulty and discrimination index – shows that most of the SJT items are good test items and can differentiate between high and low performers. The corrected point biserial correlations show positive correlation for most of the items. Generalization inference shows the items represent the domains of interest and are internally consistent. However, few items that show poor difficulty and discrimination index were subjected to re-evaluation and possible elimination.ConclusionThis study has produced a set of valid clinical scenarios that can be used to evaluate specific behavioural competencies among trainee medical doctors. It demonstrates that SJTs can be used to assess behavioural competencies for medical practice. However, further research is needed to establish the applicability of SJT beyond the immediate context, such as the medical school, in which it is developed.

Professionalism assessment of students in clinical education through Situational Judgment Test (SJT)

BackgroundThe study aims to assess the situational judgment capability of students in various professions, including medicine, surgical nursing, anesthesia nursing, and emergency medical technology, using a validated and adapted Situational Judgment Test (SJT).MethodsThe cross-sectional study was conducted at Qom University of Medical Sciences in 2023–2024. The study consisted of two steps: (1) adaptation and validity assessment of the SJT in various health professions, and (2) evaluation of students’ situational judgment capability using the adapted SJT. Participants included 207 students from surgical nursing, anesthesia nursing, medicine, and emergency medical technology departments, selected through stratified random sampling. The SJT included 10 constructed-response scenarios developed by Alkhuzaee and colleagues in 2022. The validation and adaptation process utilized consensus methods, and the SJT was modified to a selected response format based on a knowledge-based approach. Scores ranged from 0 to 30. Cohen’s method and norm-referencing standard deviation were used to calculate a cut-off score. Data were analyzed using descriptive statistics (frequency, mean, standard deviation) and inferential statistics (independent t-test, ANOVA, and ANCOVA).ResultsThe face and content validity of the adapted SJT with a selected-response format was confirmed by expert agreement. The reliability of the SJT was approved, with a Cronbach’s Alpha of 0.81. The mean score of students’ situational judgment in professionalism was 11.90 (SD 4.54). Students’ professions significantly impacted their situational judgment scores (F (3.201) = 3.67, p = 0.01, Partial Eta Squared = 0.052).ConclusionThe adaption and validation of SJT with a selected response format were confirmed by experts who evaluated the situational judgment capabilities of students in different clinical professions. The reliability of the SJT was acceptable—Cohen’s method as a standard-setting approach assisted in identifying students needing support and training. The moderate level of situational judgment capacity confirmed the development of the educational program for the improvement of non-cognitive and non-technical skills such as judgment and decision-making in the field of professionalism.

Large language models can outperform humans in social situational judgments

Large language models (LLM) have been a catalyst for the public interest in artificial intelligence (AI). These technologies perform some knowledge-based tasks better and faster than human beings. However, whether AIs can correctly assess social situations and devise socially appropriate behavior, is still unclear. We conducted an established Situational Judgment Test (SJT) with five different chatbots and compared their results with responses of human participants (N = 276). Claude, Copilot and you.com’s smart assistant performed significantly better than humans in proposing suitable behaviors in social situations. Moreover, their effectiveness rating of different behavior options aligned well with expert ratings. These results indicate that LLMs are capable of producing adept social judgments. While this constitutes an important requirement for the use as virtual social assistants, challenges and risks are still associated with their wide-spread use in social contexts.

A head‐to‐head comparison of situational judgment tests and assessment centers for measuring and predicting the same performance dimensions

AbstractTo date, a limited set of studies have compared the criterion‐related validity of low‐fidelity (SJT) versus high‐fidelity (AC) simulations for predicting job performance. Unfortunately, these studies validated these simulations through the overall assessment rating (OAR) instead of on the basis of specific dimensions. Given SJTs and ACs were compared that measured different dimensions, our understanding of the relative and comparative validity of these assessment approaches in measuring the same set of dimensions is still limited. Therefore, this study aims to conduct a head‐to‐head comparison of the criterion‐related validity of the AC and the SJT (and their incremental validity) while keeping the performance dimensions under investigation constant. Data were collected from 406 applicants for supervisory and management positions in a large Iranian steel industry company. In this process, a general mental ability test, a personality inventory, an SJT, and an AC were used as predictors, and supervisory ratings of job performance dimensions (Thinking, Feeling, and Power) served as criteria. The AC had relatively high validity for all three dimensions, whereas the SJT had a similar validity only for the Thinking dimension. So, the SJT was significantly weaker in assessing the Feeling and Power dimensions. These results were confirmed by incremental validity analyses. Overall, this study shows that understanding the relationships between predictor and criterion dimensions plays a critical role in developing valid selection systems.

Associations between an open-response situational judgment test and performance on OSCEs and fieldwork: implications for admissions decisions and matriculant diversity in an occupational therapy program

BackgroundCasper, an online open-response situational judgement test that assesses social intelligence and professionalism [1], is used in admissions to health professions programs.MethodThis study (1) explored the incremental validity of Casper over grade point average (GPA) for predicting student performance on objective structured clinical examinations (OSCEs) and fieldwork placements within an occupational therapy program, (2) examined optimal weighting of Casper in GPA in admissions decisions using non-linear optimization and regression tree analysis to find the weights associated with the highest average competency scores, and (3) investigated whether Casper could be used to impact the diversity of a cohort selected for admission to the program.ResultsMultiple regression analysis results indicate that Casper improves the prediction of OSCE and fieldwork performance over and above GPA (change in Adj. R2 = 3.2%). Non-linear optimization and regression tree analysis indicate the optimal weights of GPA and Casper for predicting performance across fieldwork placements are 0.16 and 0.84, respectively. Furthermore, the findings suggest that students with a slightly lower GPA (e.g., 3.5–3.6) could be successful in the program as assessed by fieldwork, which is considered to be the strongest indicator of success as an entry-level clinician. In terms of diversity, no statistically significant differences were found between those actually admitted and those who would have been admitted using Casper.ConclusionThese results constitute preliminary validity evidence supporting the integration of Casper into applicant selection in an occupational therapy graduate program.

Incremental validity of the test of practical judgment (TOP-J) in the prediction of diagnosis in preclinical dementia

ABSTRACT The Test of Practical Judgment (TOP-J) has not been thoroughly investigated in terms of its incremental validity. In the current study, we explored whether the TOP-J adds unique and meaningful information to the neuropsychological assessment beyond other executive functioning tests that are often used as proxies for practical judgment. Ninety-seven older adults who were classified as cognitively unimpaired, with subjective cognitive decline, or with mild cognitive impairment completed a comprehensive neuropsychological evaluation. Incremental validity was assessed through hierarchical ordinal regression analysis by modeling the TOP-J (Forms A and B, 15-item and 9-item versions), in addition to widely used tests of executive function, with participant classification/diagnosis as the outcome. The addition of the TOP-J (both 15-item versions) added incremental validity beyond traditional executive functioning measures to predict diagnosis. Including the TOP-J within neuropsychological evaluations of older adults may enhance differentiation of preclinical dementia diagnoses and provide clinically valuable information to the exam.

Far transfer of retrieval-practice benefits: rule-based learning as the underlying mechanism.

Benefits of self-testing for learning have been consistently shown for simple materials such as word lists learned by rote memorization. Considerably less evidence for such benefits exists for complex, more educationally relevant materials and its application to new situations. The present study explores the mechanisms underlying this transfer. To this end, a typical retrieval-practice-effect paradigm was applied to foster the learning of an artificial language. Participants either repeatedly studied grammatically correct exemplar sentences of the artificial language or engaged in a cloze test as the interim testafter learning. To assess far transfer, participants in both groups of restudy and retrieval practice engaged in a grammaticality judgment test after a delay of 5min and 1week. In addition, participants in both groups completed a final memory test (i.e., a cloze test identical to the initial test) 1week after learning. In addition to a long-term memory benefit of retrieval practice,results revealed alsoa retrieval-practicebenefit in the far-transfer test after the 1-week delay. The findings further support the view that far transfer is supported by learning the underlying grammatical rules as opposed to memorizing the material. Thus, retrieval practice is also effective for fostering learning of complex materials and, even more importantly, for promoting transfer of learning-a crucial goal in modern educational practices.

Construct validity and applicant reactions of a gamified personality assessment

This study introduces the HEXACO-RUSH, a gamified version of the HEXACO Situational Judgment Test developed by Oostrom and her colleagues (2019). HEXACO-RUSH is a new fantasy-adventure, narrative game, where the player makes a series of decisions in a similar format like a situational judgment test. We first present evidence on the construct validity (N = 240) of the game in relation to an existing traditional personality assessment (HEXACO-60), and in a second study (N = 160), we explore perceptions towards the game using a repeated measures, within-samples research design, again compared to HEXACO-60, with a sample of participants who assumed the role of a job applicant. Our results provided preliminary support for a moderate alignment between the new game and the HEXACO model, with average correlations of .43 across the six dimensions. It also exhibited positive participants' reactions, compared to HEXACO-60, although these were moderated by previous video-gaming experience (but not age). Overall, this study provides some initial evidence for an alternative assessment of the six personality factors with improved participants’ reactions and higher resistance to faking, but further research is required to confirm its test re-test reliability and criterion-related validity, with larger employee or applicant samples.

Risk and benefits of expanded donor screening: A viewpoint from Germany.

This review describes the risks and benefits of expanding screening for transmissible pathogens in deceased organ donors. The focus is on the experience and procedure in Germany to make a decision on how to proceed with a possible donor. Three issues are of interest in how screening policies impact the process with the aim of mitigating unexpected transmission risks: (1) Should we add universal or targeted nucleic acid testing to serological tests for common blood-borne viruses (BBVs; HIV, HBV, and HCV)? (2) Which tests should be added for screening in a geographically restricted region beyond testing for these BBVs? (3) Being faced with changes (e.g., climate and population) in the own geographically restricted region, what strategies are needed before implementing new tests, and which considerations apply for proper indication to do this? Testing may only be effective when during donor characterization the appropriate conclusions are drawn from the existing findings and screening tests are initiated. This statement overlaps the need to implement universal screening for a pathogen or targeted screening based on the risk that the donor has acquired the transmissible pathogen or is not as possible to identify by current methods of clinical judgment and/or specific tests.

Ujian Pertimbangan Situasi (SJT): pengenalan kepada pengukuran psikologi [Situation Judgment Test (SJT): an introduction to psychological measurement

The Situational judgment Test (SJT) is a psychological measurement instrument that promotes ideas and knowledge to enable decisions to be made by stakeholders and measures decisions based on the responses of test candidates. SJT is gaining popularity all over the world and is often used in various fields especially for Personnel Selection, promotion and professional development. However, limited practice and discussion has been identified mainly in terms of concepts and theories related to SJT. Therefore, this article discusses the definition of SJT, the history of SJT, and the components of SJT items. The ideas and discussions in this article can not only increase the theoretical value, but also the practical value in measuring the quality of test candidates, especially involving psychological measurements.

Predicting undergraduate OSCE performance using traditional and construct-driven situational judgment tests at admission.

Situational Judgement Tests (SJTs) are popular to screen for social skills during undergraduate medical admission as they have been shown to predict relevant study outcomes. Two different types of SJTs can be distinguished: Traditional SJTs, which measure general effective behavior, and construct-driven SJTs which are designed to measure specific constructs. To date, there has been no comparison of the predictive validity of these two types of SJTs in medical admission. With the present research, we examine whether the HAM-SJT, a traditional SJT, and the CD-SJT, a construct-driven SJT with an agentic and a communal scale, administered during undergraduate medical admission can predict OSCE (i.e., objective structured clinical examination) performance in a low-stakes (nLS = 159) and a high-stakes (nHS = 160) sample of medical students. Results showed a moderate positive relation between the communal scale of the CD-SJT and performance in OSCE stations with trained patients in the high-stakes sample (r =.20, p =.009). This SJT had also an incremental value in predicting the OSCE performance above and beyond GPA (i.e., grade point average), a science test (i.e., HAM-Nat), and gender (ß = 0.18, 95% CI [0.03; 0.33], p =.020). That is, individuals who chose more communal behavioral responses in the SJT were rated more favorably in interactions with trained patients in the OSCE. A comparable correlation coefficient was observed for the HAM-SJT when controlling for range restriction due to admission (rraw = 0.14 vs. rcontrolled = 0.20). Our research provides a first indication for the predictive validity of construct-driven SJTs in high-stakes undergraduate medical admission.

Insights into undergraduate medical student selection tools: a systematic review and meta-analysis

Evaluating medical school selection tools is vital for evidence-based student selection. With previous reviews revealing knowledge gaps, this meta-analysis offers insights into the effectiveness of these selection tools. A systematic review and meta-analysis were conducted applying the following criteria: peer-reviewed articles available in English, published from 2010 and which include empirical data linking performance in selection tools with assessment and dropout outcomes of undergraduate entry medical programs. Systematic reviews, meta-analyses, general opinion pieces, or commentaries were excluded. Effect sizes (ESs) of the predictability of academic and clinical performance within and by the end of the medicine program were extracted, and the pooled ESs were presented. Sixty-seven out of 2,212 articles were included, which yielded 236 ESs. Previous academic achievement predicted medical program academic performance (Cohen’s d=0.697 in early program; 0.619 in end of program) and clinical exams (0.545 in end of program). Overall aptitude tests predicted academic achievement in both the early and last years (0.550 & 0.371, respectively). Within aptitude tests, verbal reasoning and quantitative reasoning best predicted academic achievement in the early program (0.704 & 0.643, respectively). Neither panel interviews, multiple mini-interviews, nor situational judgement tests (SJT) yielded statistically significant pooled ES. Current evidence suggests that learning outcomes are predicted by previous academic achievement and aptitude tests. The predictive value of SJT and topics such as selection algorithms, features of interview (e.g., content of the questions) and the way the interviewers’ reports are used, warrant further research.

La scelta dell’ausiliare nei tempi composti: Variazione nell'interlingua di apprendenti ungheresi e inglesi di italiano L2/LS

The aim of this study is to identify some factors that determine variability in the interlanguage of L2/FL learners of Italian concerning auxiliary selection (essere ‘to be’ vs avere ‘to have’) in compound past tenses. Data were collected by means of a grammaticality judgement test and a binary choice sentence completion test. The participants were L1 English and Hungarian university students of various levels of proficiency (A2, B1, B2 and C1). The results were interpreted both in correlation with certain learning variables such as L1, proficiency level, and the category the main verbs belong to according to Sorace’s Auxiliary Selection Hierarchy. Based on the results, some hypotheses are put forward on teaching material design and on the possibility of harmonising the didactic intervention and the acquisitional sequence observed.

Generative AI in the service of situational judgment tests (SJT) for self-regulation learning during problem solving (SRL-PS)

Those involved in education see great importance in promoting self-regulated learning (SRL) and problem-solving (PS) among learners. Teachers are a major factor in promoting SRL-PS in the classroom, but very little is known about whether and how they promote it in their classroom, therefore a tool is needed to assess teachers in this domain. Furthermore, the situational judgment test (SJT) is a potential tool to diagnose teachers’ knowledge. However, such a tool is currently lacking in the SRL-PS domain. Human interaction with generative artificial intelligence (GenAI) enables overcoming difficulties and mutually complementing each other. The purpose of this paper is to present an initial attempt to develop an SJT tool for assessing teachers’ SRL-PS knowledge using the assistance of ChatGPT. This humane-machine interaction led to the formulation of 15 difficulties categories and 20 scenarios that form the basis of the SJT tool for SRL-PS. It was found that scenarios created by the researchers can be complemented by those generated by ChatGPT. In some instances, the scenarios from both sources are quite similar, while in others, those formulated by ChatGPT either expand upon or present an alternative perspective on the difficulty. Moreover, ChatGPT has proposed new scenarios in certain cases. A Significant product of the study is a map that enables the analysis of the pool of scenarios and the identification of over- or underrepresented difficulties categories.

Situational Judgement Tests among Palestinian community members and Red Crescent volunteers to inform humanitarian action: a cross-sectional study

BackgroundInforming humanitarian action directly from community members is recognized as critical. However, collecting community insights is also a challenge in practice. This paper reports data collected among community members and Red Crescent volunteers in the occupied Palestinian territory. The aim was to test a data collection tool, situational judgment tests (SJTs), to collect insights in the community around three themes.MethodsThe SJTs covered violence prevention, road safety, and environmental pollution (waste), and were constituted of hypothetical scenarios to which respondents indicated how they would react. For each theme, the answers’ pattern provides insights for humanitarian action regarding which beliefs to address. A cross-sectional survey was conducted in January and February 2023 with 656 community members, and 239 Red Crescent volunteers.ResultsData showed that violence is the theme for which the need is the highest among community members. Some responses varied according to the public (age, governorate, or disability level), suggesting actions could be tailored accordingly.ConclusionsDespite many difficulties during data collection, this study show that the tool allowed to collect community insights, a crucial task to ensure adequate response to the challenges faced by community members and Red Crescent volunteers in occupied Palestine.

Evaluation of simulation-based intervention for implicit bias mitigation: A response to systemic racism

BackgroundTraining health care professionals to mitigate implicit biases can reduce health inequities caused by racism. Implicit bias mitigation strategies (IBMS) have been shown to reduce biased behaviors. This study aimed to determine the impact of IBMS practice simulation on learners’ behaviors and family experience in one hospital unit. MethodsThis study used a quasi-experimental design in a large pediatric hospital. In total, 195 participants completed a simulation-based education intervention. Learners practiced IBMS with Simulated Participants. Participant perceptions were measured at four time points, and translational outcomes were collected using survey results and other outcomes. ResultsSituational judgment test results improved postintervention and at follow-up. Participants reported using IBMS skills at time points three and four. Internal and External Motivation Scale and family satisfaction scores remained unchanged. Family experience measures improved, showing decreases in security dispatch calls for perceived family escalation and in deny entries and behavioral contracts stratified by race. ConclusionsSimulation can shift perspectives and behaviors. More research is needed on the impact of simulation-based interventions on IBMS skill development and translational outcomes.

Validity and Reliability of the Situational Judgment Test Instrument for Assessing the 'Pancasila' Student Profile

Character measurement has become increasingly relevant and needed in recent years due to its significant impact on individual success in various aspects of life. In the Indonesian context, the Pancasila Student Profile (PSP) is one set of character values being promoted in the curriculum. However, methods of character measurement such as self-assessment, peer assessment, and observation still have many limitations. The Situational Judgement Test (SJT) is a behavioral measurement tool with the advantage of reducing measurement bias. Therefore, this study aims to evaluate the psychometric properties, i.e. validity and reliability, of the SJT-based PSP measurement instrument. The sample size was 351 tenth-grade students. The research findings include an Aiken's V index of 0.684, Goodness of Fit statistics with χ²/df = 122.299/48, CFI = 0.938, TLI = 0.915, RMSEA = 0.066, and SRMR = 0.046, all within the good fit category. The empirical data of PSP align with the GRM model, with item fit MNSQ method showing 59 items fitting the GRM model and item fit MNSQ method. The PSP instrument has a Construct Reliability (CR) of 0.967 and an average variance explained (AVE) of 0.830. These findings contribute theoretically and practically to the field of measurement, indicating that the SJT-based instrument is valid and reliable for measuring PSP.

DiffErential attainment and Factors AssoCiated with Training applications and Outcomes (DE FACTO) study: Trauma & Orthopaedic surgery in the UK.

The aims of this study were to describe the demographic, socioeconomic, and educational factors associated with core surgical trainees (CSTs) who apply to and receive offers for higher surgical training (ST3) posts in Trauma & Orthopaedics (T&O). Data collected by the UK Medical Education Database (UKMED) between 1 January 2014 and 31 December 2019 were used in this retrospective longitudinal cohort study comprising 1,960 CSTs eligible for ST3. The primary outcome measures were whether CSTs applied for a T&O ST3 post and if they were subsequently offered a post. A directed acyclic graph was used for detecting confounders and adjusting logistic regression models to calculate odds ratios (ORs), which assessed the association between the primary outcomes and relevant exposures of interest, including: age, sex, ethnicity, parental socioeconomic status (SES), domiciliary status, category of medical school, Situational Judgement Test (SJT) scores at medical school, and success in postgraduate examinations. This study followed STROBE guidelines. Compared to the overall cohort of CSTs, females were significantly less likely to apply to T&O (OR 0.37, 95% CI 0.30 to 0.46; n = 155/720 female vs n = 535/1,240 male; p < 0.001). CSTs who were not UK-domiciled prior to university were nearly twice as likely to apply to T&O (OR 1.99, 95% CI 1.39 to 2.85; n = 50/205 vs not UK-domiciled vs n = 585/1,580 UK-domiciled; p < 0.001). Age, ethnicity, SES, and medical school category were not associated with applying to T&O. Applicants who identified as 'black and minority ethnic' (BME) were significantly less likely to be offered a T&O ST3 post (OR 0.70, 95% CI 0.51 to 0.97; n = 165/265 BME vs n = 265/385 white; p = 0.034). Differences in age, sex, SES, medical school category, and SJT scores were not significantly associated with being offered a T&O ST3 post. There is an evident disparity in sex between T&O applicants and an ethnic disparity between those who receive offers on their first attempt. Further high-quality, prospective research in the post-COVID-19 pandemic period is needed to improve equality, diversity, and inclusion in T&O training.

Exploring plurilingualism, ethnicity, and intercultural effectiveness: Insights from university students

AbstractThe study reported in this article examined the dynamics of plurilingualism, ethnicity, and effectiveness in intercultural encounters among a sample of 221 participants from Chinese universities via an online questionnaire and a situational judgement test comprised of three scenarios involving plurilingual and intercultural encounters. It employed latent profile analysis to identify distinct profiles based on participants' plurilingualism, including how they used their languages, their exposure to linguistic diversity in oral interactions, and their experiences with plurilingualism in society, school, and university. The study also considered participants' ethnic backgrounds and self‐reported intercultural effectiveness. The findings revealed two latent profiles characterised by notable differences in ethnic background, language proficiency, language use, and experiences of plurilingualism in society, school, and university. Additionally, the most consistent predictors of participants' behaviour in the three scenarios were their fluid language use and exposure to linguistic diversity during oral interactions.

Predicting Surgical Resident Performance With Situational Judgment Tests.

Situational judgment tests (SJTs) have been proposed as an efficient, effective, and equitable approach to residency program applicant selection. This study examined how SJTs can predict milestone performance during early residency. General surgery residency program applicants during 3 selection cycles (2018-2019, 2019-2020, 2020-2021) completed SJTs. Accreditation Council for Graduate Medical Education milestone performance data from selected applicants were collected in March and April 2019, 2020, and 2021 and from residents in March 2020, August 2020, March 2021, September 2021, and March 2022. Descriptive statistics and correlations were computed and analysis of variance tests performed to examine differences among 4 SJT performance groups: green, top 10% to 25%; yellow, next 25% to 50%; red, bottom 50%; and unknown, did not complete the SJT. Data were collected for 70 residents from 7 surgery residency programs. Differences were found for patient care ( F3,189 = 3.19, P = .03), medical knowledge ( F3,176 = 3.22, P = .02), practice-based learning and improvement ( F3,189 = 3.18, P = .04), professionalism ( F3,189 = 3.82, P = .01), interpersonal and communication skills ( F3,190 = 3.35, P = .02), and overall milestone score ( F3,189 = 3.44, P = .02). The green group performed better on patient care, medical knowledge, practice-based learning and improvement, professionalism, and overall milestone score. The yellow group performed better than the red group on professionalism and overall milestone score, better than the green group on interpersonal and communication skills, and better than the unknown group on all but practice-based learning and improvement. The red group outperformed the unknown group on all but professionalism and outperformed the green group on medical knowledge. SJTs demonstrate promise for assessing important noncognitive attributes in residency applicants and align with national efforts to review candidates more holistically and minimize potential biases.