High-stakes Settings Research Articles

Debiasing Judgements Using a Distributed Cognition Approach: A Scoping Review of Technological Strategies.

To review and synthesise research on technological debiasing strategies across domains, present a novel distributed cognition-based classification system, and discuss theoretical implications for the field. Distributed cognition theory is valuable for understanding and mitigating cognitive biases in high-stakes settings where sensemaking and problem-solving are contingent upon information representations and flows in the decision environment. Shifting the focus of debiasing from individuals to systems, technological debiasing strategies involve designing system components to minimise the negative impacts of cognitive bias on performance. To integrate these strategies into real-world practices effectively, it is imperative to clarify the current state of evidence and types of strategies utilised. We conducted systematic searches across six databases. Following screening and data charting, identified strategies were classified into (i) group composition and structure, (ii) information design and (iii) procedural debiasing, based on distributed cognition principles, and cognitive biases, classified into eight categories. Eighty articles met the inclusion criteria, addressing 100 debiasing investigations and 91 cognitive biases. A majority (80%) of the identified debiasing strategies were reportedly effective, whereas fourteen were ineffective and six were partially effective. Information design strategies were studied most, followed by procedural debiasing, and group structure and composition. Gaps and directions for future work are discussed. Through the lens of distributed cognition theory, technological debiasing represents a reconceptualisation of cognitive bias mitigation, showing promise for real-world application. The study results and debiasing classification presented can inform the design of high-stakes work systems to support cognition and minimise judgement errors.

Assessing the Speed–Accuracy Tradeoff in Psychological Testing Using Experimental Manipulations

The speed–accuracy tradeoff (SAT), where increased response speed often leads to decreased accuracy, is well established in experimental psychology. However, its implications for psychological assessments, especially in high-stakes settings, remain less understood. This study presents an experimental approach to investigate the SAT within a high-stakes spatial ability assessment. By manipulating instructions in a within-subjects design to induce speed variations in a large sample ( N = 1,305) of applicants for an air traffic controller training program, we demonstrate the feasibility of manipulating working speed. Our findings confirm the presence of the SAT for most participants, suggesting that traditional ability scores may not fully reflect performance in high-stakes assessments. Importantly, we observed individual differences in the SAT, challenging the assumption of uniform SAT functions across test takers. These results highlight the complexity of interpreting high-stakes assessment outcomes and the influence of test conditions on performance dynamics. This study offers a valuable addition to the methodological toolkit for assessing the intraindividual relationship between speed and accuracy in psychological testing (including SAT research), providing a controlled approach while acknowledging the need to address potential confounders. Future research may apply this method across various cognitive domains, populations, and testing contexts to deepen our understanding of the SAT’s broader implications for psychological measurement.

Early Impressions and Adoption of the AtriAmp for Managing Arrhythmias Following Congenital Heart Surgery.

AtriAmp is a new medical device that displays a continuous real-time atrial electrogram on telemetry using temporary atrial pacing leads. Our objective was to evaluate early adoption of this device into patient care within our pediatric intensive care unit (PICU). This is a qualitative study using inductive analysis of semi-structured interviews to identify dominant themes. The study was conducted in a single-center, tertiary, academic 21-bed mixed PICU. The subjects were PICU multidisciplinary team members (Pediatric Cardiac Intensivists, PICU Nurse Practitioners, PICU nurses and Pediatric Cardiologists) who were early adopters of the AtriAmp (n = 14). Three prominent themes emerged: (1) Accelerated time from arrhythmia event to diagnosis and treatment; (2) Increased confidence in the accuracy of providers' arrhythmia diagnosis; and (3) Improvement in the ability to educate providers about post-operative arrhythmias. Providers also noted some learning curves, but none compromised medical care or clinical workflow. Insights from early adopters of AtriAmp signal the need for simplicity and fidelity in new PICU technologies. Our research suggests that such technologies can be pivotal to the support and growth of multi-disciplinary teams, even among those who do not participate in early implementation. Further research is needed to understand when and why novel technology adoption becomes widespread in high-stakes settings.

The Causes and Consequences of Drifting Expectations

Awaiting news of uncertain outcomes is distressing because the news might be disappointing. To prevent such disappointments, people often “brace for the worst,” pessimistically lowering expectations before news arrives to decrease the possibility of surprising disappointment (a negative prediction error, or PE). Computational decision-making research commonly assumes that expectations do not drift within trials, yet it is unclear whether expectations pessimistically drift in real-world, high-stakes settings, what factors influence expectation drift, and whether it effectively buffers emotional responses to goal-relevant outcomes. Moreover, individuals learn from PEs to accurately anticipate future outcomes, but it is unknown whether expectation drift also impedes PE-based learning. In a sample of students awaiting exam grades (N = 625), we found that expectations often drift and tend to drift pessimistically. We demonstrate that bracing is preferentially modulated by uncertainty; it transiently buffers the initial emotional impact of negative PEs but impairs PE-based learning, counterintuitively sustaining uncertainty into the future.

ProtoExplorer: Interpretable forensic analysis of deepfake videos using prototype exploration and refinement

In high-stakes settings, Machine Learning models that can provide predictions that are interpretable for humans are crucial. This is even more true with the advent of complex deep learning based models with a huge number of tunable parameters. Recently, prototype-based methods have emerged as a promising approach to make deep learning interpretable. We particularly focus on the analysis of deepfake videos in a forensics context. Although prototype-based methods have been introduced for the detection of deepfake videos, their use in real-world scenarios still presents major challenges, in that prototypes tend to be overly similar and interpretability varies between prototypes. This paper proposes a Visual Analytics process model for prototype learning, and, based on this, presents ProtoExplorer, a Visual Analytics system for the exploration and refinement of prototype-based deepfake detection models. ProtoExplorer offers tools for visualizing and temporally filtering prototype-based predictions when working with video data. It disentangles the complexity of working with spatio-temporal prototypes, facilitating their visualization. It further enables the refinement of models by interactively deleting and replacing prototypes with the aim to achieve more interpretable and less biased predictions while preserving detection accuracy. The system was designed with forensic experts and evaluated in a number of rounds based on both open-ended think aloud evaluation and interviews. These sessions have confirmed the strength of our prototype-based exploration of deepfake videos while they provided the feedback needed to continuously improve the system.

Strategies for case-based training with virtual patients: An experimental study of the impact of integrating mental model articulation and self-reflection

Resilient system performance in high-stakes settings, which includes the ability to monitor, respond, anticipate, and learn, can be enhanced for trainees through simulation of realistic scenarios enhanced by augmented reality. Active learning strategies can enhance simulation-based training, particularly the mental model articulation principle where students are prompted to anticipate what will happen next and the reflection principle where students self-assess their performance compared to a gold standard expert model. In this paper, we compared simulation-based training for trauma care with and without active learning strategies during pauses in the simulated action for progressively deteriorating patients. The training was conducted online and real-time without a facilitator, with 42 medical students viewing training materials and then immediately taking an online quiz for three types of trauma cases: hemorrhage, airway obstruction, and tension pneumothorax. Participants were randomly assigned to either the experimental or control condition in a between-subjects design. We compared performance in the control and experimental conditions based on: A) the proportion of cues correctly recognized, B) the proportion of accurate diagnoses, C) the proportion of appropriate treatment interventions, and D) verbal briefing quality on a 1–5 scale. We found that the training intervention increased recognition of subtle cues critical for accurate diagnosis and appropriate treatment interventions; the training did not improve the accuracy of diagnoses or the quality of the verbal briefing. We conclude that incorporating active learning strategies in simulation-based training improved foundational capabilities in detecting subtle cues and intervening to rescue deteriorating patients that can increase the readiness for trainees to contribute to resilient system performance in the high-stakes setting of emergency care in hospitals.

Winter weather on exam dates and matriculation for a prestigious university in Japan

Whereas there are growing numbers of studies on how heat affects cognitive performance at high-stakes settings, the effect of cold weather has been less studied. To fill this gap, I analyze the effects of winter weather on exam dates on performance at the centralized exam for university admission in Japan. Since students take the exam in their own prefectures, they are exposed to different weather conditions on the exam dates. I find that low temperature and snow cumulated on the ground reduce matriculation shares for the most prestigious university in Japan, the University of Tokyo. This result suggests that the winter weather on exam dates worsens exam performance.

Elevating humanism in high-stakes automation: experts-in-the-loop and resort-to-force decision making

ABSTRACT Artificial intelligence (AI) technologies pervade myriad decision systems, mobilising data at a scale, speed, and scope that far exceed human capacities. Although it may be tempting to displace humans with these automated decision systems, doing so in high-stakes settings would be a mistake. Anchored by the example of states’ resort to force, I argue that human expertise should be elevated—not relegated—within high-stakes decision contexts that incorporate AI tools. This argument builds from an empirical reality in which defence institutions increasingly rely on and invest in AI capabilities, an active debate about how (and if) humans should figure into automated decision loops, and a socio-technical landscape marked by both promise and peril. The argument proceeds through a primary claim about the amplified relevance of expert humans in light of AI, underpinned by the assumed risks of omitting human experts, together motivating a tripartite call to action. The position presented herein speaks directly to the military domain, but also generalises to a broader worldbuilding project that preserves humanism amidst suffusive AI.

AI in the ED: Assessing the efficacy of GPT models vs. physicians in medical score calculation

Background and aimsArtificial Intelligence (AI) models like GPT-3.5 and GPT-4 have shown promise across various domains but remain underexplored in healthcare. Emergency Departments (ED) rely on established scoring systems, such as NIHSS and HEART score, to guide clinical decision-making. This study aims to evaluate the proficiency of GPT-3.5 and GPT-4 against experienced ED physicians in calculating five commonly used medical scores. MethodsThis retrospective study analyzed data from 150 patients who visited the ED over one week. Both AI models and two human physicians were tasked with calculating scores for NIH Stroke Scale, Canadian Syncope Risk Score, Alvarado Score for Acute Appendicitis, Canadian CT Head Rule, and HEART Score. Cohen's Kappa statistic and AUC values were used to assess inter-rater agreement and predictive performance, respectively. ResultsThe highest level of agreement was observed between the human physicians (Kappa = 0.681), while GPT-4 also showed moderate to substantial agreement with them (Kappa values of 0.473 and 0.576). GPT-3.5 had the lowest agreement with human scorers. These results highlight the superior predictive performance of human expertise over the currently available automated systems for this specific medical outcome. Human physicians achieved a higher ROC-AUC on 3 of the 5 scores, but none of the differences were statistically significant. ConclusionsWhile AI models demonstrated some level of concordance with human expertise, they fell short in emulating the complex clinical judgments that physicians make. The study suggests that current AI models may serve as supplementary tools but are not ready to replace human expertise in high-stakes settings like the ED. Further research is needed to explore the capabilities and limitations of AI in emergency medicine.

Conscientiousness Can Predict Academic Performance – Even in High-Stake Setting

Abstract: We investigated, to what extent Conscientiousness can predict academic performance in a real application high-stakes setting. N = 267 applicants for a place at a university completed a Conscientiousness questionnaire during the selection process and 6 weeks after they commenced their studies. Students’ academic grades were used as criterion variables. The results suggest that the high-stakes setting increases the level of Conscientiousness reported, that not all applicants change their answers to the same extent and that the high-stakes setting decreases the criterion-related validity, but Conscientiousness remains a useful predictor for academic performance in high-stakes settings.

Using artificial intelligence to assess personal qualities in college admissions.

Personal qualities like prosocial purpose and leadership predict important life outcomes, including college success. Unfortunately, the holistic assessment of personal qualities in college admissions is opaque and resource intensive. Can artificial intelligence (AI) advance the goals of holistic admissions? While cost-effective, AI has been criticized as a "black box" that may inadvertently penalize already disadvantaged subgroups when used in high-stakes settings. Here, we consider an AI approach to assessing personal qualities that aims to overcome these limitations. Research assistants and admissions officers first identified the presence/absence of seven personal qualities in n = 3131 applicant essays describing extracurricular and work experiences. Next, we fine-tuned pretrained language models with these ratings, which successfully reproduced human codes across demographic subgroups. Last, in a national sample (N = 309,594), computer-generated scores collectively demonstrated incremental validity for predicting 6-year college graduation. We discuss challenges and opportunities of AI for assessing personal qualities.

Cognitive Systems Engineering at 40, Part II: Applying Theory to Practice

This expert panel is the second of a two-panel series marking the 40th anniversary of “Cognitive Systems Engineering: New Wine in New Bottles” by Hollnagel and Woods (1983) and, arguably, the beginning of Cognitive Systems Engineering (CSE). These experts were there at (or near) the beginning. They wrestled with the daunting issues of how to apply new CSE findings, many of which they themselves discovered, into workable solutions that actually improved systems performance in high-uncertainty, high-stakes settings. Their work has been implemented in a number of fields, including aviation, healthcare, power generation, and defense. They will each answer the question, “What ideas and perspectives are important about Cognitive Systems Engineering, and why?”

Automated Emotion Recognition in the Workplace: How Proposed Technologies Reveal Potential Futures of Work

Emotion recognition technologies, while critiqued for bias, validity, and privacy invasion, continue to be developed and applied in a range of domains including in high-stakes settings like the workplace. We set out to examine emotion recognition technologies proposed for use in the workplace, describing the input data and training, outputs, and actions that these systems take or prompt. We use these design features to reflect on these technologies' implications using the ethical speculation lens. We analyzed patent applications that developed emotion recognition technologies to be used in the workplace (N=86). We found that these technologies scope data collection broadly; claim to reveal not only targets' emotional expressions, but also their internal states; and take or prompt a wide range of actions, many of which impact workers' employment and livelihoods. Technologies described in patent applications frequently violated existing guidelines for ethical automated emotion recognition technology. We demonstrate the utility of using patent applications for ethical speculation. In doing so, we suggest that 1) increasing the visibility of claimed emotional states has the potential to create additional emotional labor for workers (a burden that is disproportionately distributed to low-power and marginalized workers) and contribute to a larger pattern of blurring boundaries between expectations of the workplace and a worker's autonomy, and more broadly to the data colonialism regime; 2) Emotion recognition technology's failures can be invisible, may inappropriately influence high-stakes workplace decisions and can exacerbate inequity. We discuss the implications of making emotions and emotional data visible in the workplace and submit for consideration implications for designers of emotion recognition, employers who use them, and policymakers.

Computer-based training and repeated test performance: Increasing assessment fairness instead of retest effects

ABSTRACT When subjects are repeatedly tested in cognitive assessments, systematic score gains occur. Such retest effects become even greater when test preparation is provided between assessments. In the context of personnel selection, retest gains are often increased by commercial test training, which threatens the fairness of psychological testing because not all candidates can afford such offers. In the present study, computer-based training was freely offered to all candidates as part of the personnel selection procedure. We examined the relationship between repeated cognitive ability measurements in high-stakes settings and the amount of computer-based training before each measurement. Analyses of 212 candidates showed that cognitive ability scores and the amount of prior training were only related on the first assessment but not on the second. There were still retest effects, but the magnitude of score gains was negatively correlated with the amount of initial training and was unrelated to training between assessments. Only the change in training amount was positively correlated with retest effects. We conclude that providing all candidates with preparatory training already before the first assessment substantially increases assessment fairness in the personnel selection process.

The Relationship between Preparation, Impression Management, and Interview Performance in High-Stakes Personnel Selection: A Field Study of Airline Pilot Applicants

ABSTRACT Objective The present study investigates how airline pilot applicants prepare for a personnel selection procedure and how this relates to their deceptive and honest impression management (IM) in the job interview and their interview performance. Background It is thus far unclear how preparation relates to IM in real-world and high-stakes settings. This is of particular importance for the aviation industry, because numerous commercial providers offer preparation courses for selection procedures potentially impacting selection decisions. Method We conducted a field study of high-stakes job interviews embedded in a selection procedure for airline pilots. Data from 100 interviewees was acquired. Results The variety of preparation strategies an applicant used was positively correlated with two deceptive IM (faking) strategies in the job interview. Moreover, low effort preparation via social media and guidebooks was positively correlated with overall faking. The variety of preparation and preparation via social media were also positively correlated with honest IM. However, neither the aforementioned preparation nor IM showed significant correlations with the interview performance. Conclusion The effects of preparation depend on the specific preparation methods used. Certain methods of preparation are positively associated with applicants’ use of IM strategies in the job interview.

Are Large Admissions Test Coaching Effects Widespread? A Longitudinal Analysis of Admissions Test Scores

ABSTRACT We examine longitudinal data from 120,384 students who took a version of the PSAT/SAT in the 9th, 10th, 11th, and 12th grades. We investigate score changes over time and show that socioeconomic status (SES) is related to the degree of score improvement. We note that the 9th and 10th grade PSAT are low-stakes tests, while the operational SAT is a high-stakes test. We posit that investments in coaching would be uncommon for early PSAT administrations, and would be concentrated on efforts to prepare for the operational SAT. We compare score improvements between 9th and 10th grade with improvements between 10th and 12th grade, examining results separately by level of SES. We find similar levels of score improvement in low-stakes and high-stakes settings, with 3.4% of high-SES and 1.1% of low-SES students showing larger-than-expected score improvements, which is inconsistent with claims that high-SES students have routine access to highly effective coaching.

Evaluating the Explanation Inference of a High-Stakes French Listening Test: An Argument-Based Perspective

This article draws on argument-based validation to gather and evaluate construct-related evidence (i.e., the explanation inference) of a high-stakes test. The data stemmed from the listening component of a French test used for immigration to Canada through the province of Quebec. An expert panel with varied backgrounds in applied linguistics reviewed and associated the items of two operational test forms to four listening comprehension sub-skills identified in selected sources of second language listening theory. Based on the expert panel recommendations, two confirmatory factor models were fit to examinees’ response data. The models fit the data well, providing backing for the explanation inference but suggesting construct under-representation for one of the test forms examined. The argument-based approach to validation yielded principled guidelines to evaluate construct coverage of the test across forms, providing insightful guidance on how to organize construct evidence from an argumentation perspective. Implications are discussed as they relate to the operationalization of argument-based validation in high-stakes settings.

Mitigating the impact of biased artificial intelligence in emergency decision-making

BackgroundPrior research has shown that artificial intelligence (AI) systems often encode biases against minority subgroups. However, little work has focused on ways to mitigate the harm discriminatory algorithms can cause in high-stakes settings such as medicine.MethodsIn this study, we experimentally evaluated the impact biased AI recommendations have on emergency decisions, where participants respond to mental health crises by calling for either medical or police assistance. We recruited 438 clinicians and 516 non-experts to participate in our web-based experiment. We evaluated participant decision-making with and without advice from biased and unbiased AI systems. We also varied the style of the AI advice, framing it either as prescriptive recommendations or descriptive flags.ResultsParticipant decisions are unbiased without AI advice. However, both clinicians and non-experts are influenced by prescriptive recommendations from a biased algorithm, choosing police help more often in emergencies involving African-American or Muslim men. Crucially, using descriptive flags rather than prescriptive recommendations allows respondents to retain their original, unbiased decision-making.ConclusionsOur work demonstrates the practical danger of using biased models in health contexts, and suggests that appropriately framing decision support can mitigate the effects of AI bias. These findings must be carefully considered in the many real-world clinical scenarios where inaccurate or biased models may be used to inform important decisions.

Pretrial release judgments and decision fatigue

Abstract Field studies in many domains have found evidence of decision fatigue, a phenomenon describing how decision quality can be impaired by the act of making previous decisions. Debate remains, however, over posited psychological mechanisms underlying decision fatigue, and the size of effects in high-stakes settings. We examine an extensive set of pretrial arraignments in a large, urban court system to investigate how judicial release and bail decisions are influenced by the time an arraignment occurs. We find that release rates decline modestly in the hours before lunch and before dinner, and these declines persist after statistically adjusting for an extensive set of observed covariates. However, we find no evidence that arraignment time affects pretrial release rates in the remainder of each decision-making session. Moreover, we find that release rates remain unchanged after a meal break even though judges have the opportunity to replenish their mental and physical resources by resting and eating. In a complementary analysis, we find that the rate at which judges concur with prosecutorial bail requests does not appear to be influenced by either arraignment time or a meal break. Taken together, our results imply that to the extent that decision fatigue plays a role in pretrial release judgments, effects are small and inconsistent with previous explanations implicating psychological depletion processes.

Perspectives on simulation-based assessment of operative skill in surgical training

Introduction The perspectives of the wider surgical community toward simulation-based assessment (SBA) in training is a gap in the literature. This study aims to explore the factors associated with the acceptable use of SBA in surgical training, through the perceptions and experiences of a broad range of stakeholder representatives, building on findings from a review of the published literature. Materials and methods Ten semi-structured interviews were conducted, using a sequential transformative qualitative methods approach, with representatives from identified key stakeholder groups; executive management, risk management, a practicing surgeon, an anaesthesiologist, a theatre-nursing representative, a representative from simulation industry, a patient, a medical student, a junior surgical trainee, and a senior surgical trainee. Interview transcripts underwent reflexive thematic analysis using an inductive and constructivist framework (NVIVO software, NVIVO 12, QSR International). Results Four themes emerged: the ‘need’ for SBA, the concept of a ‘minimum standard’, the ‘optimum design’ of an SBA framework, and ‘fairness’. SBA is a potential solution to challenges in the current training environment. It emerged that it should not replace trainer judgement, but could ensure that trainees meet a minimum operative competency standard. SBA should be used to identify underperforming trainees early in training to provide targeted remediation. The application of SBA in high-stakes settings such as trainee selection, autonomy granting, and end-of training certification has perceived benefits over current assessment methods. Conclusions This study builds on findings from prior research to explore factors regarding the acceptable use of simulation as an assessment method in surgical training, including perspectives from a broad range of stakeholder representatives. Findings can inform the development of simulation-based assessment curricula in surgical training.