High-stakes Settings Research Articles

Role Play Activities in Small Programs: What, Why, Where, and How?

Preservice teachers need many opportunities to practice teaching skills prior to using the skills in high-stakes settings like real, live classrooms. Role play is an accessible and flexible option for embedding skill practice into individual courses. They may focus on specific high-leverage practices (HLPs) and enable preservice teachers to engage in activities to use skills learned across courses. This article provides an analysis of why to use role play, suggestions on when to use role play, and steps and resources for creating and using your own role play activities. We conclude with lessons learned by our Small Special Education Program.

772 Perspectives on Simulation as an Assessment Tool in Surgical Training: A Qualitative Study of Key Stakeholders

Abstract Aim The perspectives of the wider surgical community toward simulation-based assessment (SBA) in training is a gap in the literature. This study aims to generate a conceptual framework regarding the acceptable use of SBA in surgical training, through the perceptions and experiences of a broad range of stakeholders. Method A sequential transformative qualitative methods approach employing constructivist grounded theory was used. Ten semi-structured interviews were conducted representatives from identified key stakeholder groups; executive management, risk management, a practicing surgeon, an anaesthesiologist, a theatre-nursing representative, a representative from simulation, a patient, a medical student, a junior surgical trainee and a senior surgical trainee. interview transcripts underwent content analysis using NVIVO software (NVIVO 12, QSR International). Results Four themes emerged: the ‘need’ for SBA, the concept of a ‘minimum standard’, the ‘optimum design’ of an SBA framework, and ‘fairness’. SBA is a potential solution to challenges in the current training environment. It emerged that it should not replace trainer judgement but could ensure that trainees meet a minimum operative competency standard. SBA should be used to identify underperforming trainees early in training, to provide remediation. The application of SBA in high stakes settings such as trainee selection, autonomy granting, and end-of training certification has perceived benefits over current assessment methods. Conclusions This study generates a conceptual framework regarding the acceptable use of simulation as an assessment tool in surgical training, including perspectives from a broad range of stakeholders. Findings can inform the development of simulation-based assessment curricula in surgical training.

Risks and responsibilities in the workplace

This study analyses experiences with and in relation to issues of risk, responsibility and evacuation in workplace settings as discussed in four focus groups in Sweden and Swedish-language Finland. The discussions, and in particular the participants’ positionings in their ‘small stories’, are analysed from two perspectives: that of narratives, and that of interpretive repertoires. The main question under investigation is how employees at different workplaces discursively construe their sense of responsibility in the face of risk and evacuation situations. The findings show that these issues are handled very differently in different workplaces, but at all in accordance with the participants’ implicit responsibility positionings and what we see as the interpretive repertoires (organisational, instinctual, skills-based and informational) they draw on. Whereas previous studies have considered workplaces with high-stakes settings, the present study is set in workplaces where risk is not thematised on a regular basis, making the results applicable not only to workplace scenarios, but to society at large.

Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment

AbstractWe have dramatically advanced our ability to create rich, complex, and effective assessments across a range of uses through technology advancement. Artificial Intelligence (AI) enabled assessments represent one such area of advancement—one that has captured our collective interest and imagination. Scientists and practitioners within the domains of organizational and workforce assessment have increasingly used AI in assessment, and its use is now becoming more common in education. While these types of solutions offer their users the promise of efficiency, effectiveness, and a “wow factor,” users need to maintain high standards for validity and fairness in high stakes settings. Due to the complexity of some AI methods and tools, this requirement for adherence to standards may challenge our traditional approaches to building validity and fairness arguments. In this edition, we review what these challenges may look like as validity arguments meet AI in educational assessment domains. We specifically explore how AI impacts Evidence‐Centered Design (ECD) and development from assessment concept and coding to scoring and reporting. We also present information on ways to ensure that bias is not built into these systems. Lastly, we discuss future horizons, many that are almost here, for maximizing what AI offers while minimizing negative effects on test takers and programs.

Can Faking Be Measured With Dedicated Validity Scales? Within-Subject Trifactor Mixture Modeling Applied to BIDR Responses.

A sample of 516 participants responded to the Balanced Inventory of Desirable Responding (BIDR) under answer honest and instructed faking conditions in a within-subjects design. We analyze these data with a novel application of trifactor modeling that models the two substantive factors measured by the BIDR-self-deceptive enhancement (SDE) and impression management (IM), condition-related common factors, and item-specific factors. The model permits examination of invariance and change within subjects across conditions. Participants were able to significantly increase their SDE and IM in the instructed faking condition relative to the honest response condition. Mixture modeling confirmed the existence of a theoretical two-class solution comprised of approximately two thirds of "compliers" and one third of "noncompliers." Factor scores had good determinacy and correlations with observed scores were near unity for continuous scoring, supporting observed score interpretations of BIDR scales in high-stakes settings. Correlations were somewhat lower for the dichotomous scoring protocol. Overall, results show that the BIDR scales function similarly as measures of socially desirable functioning in low- and high-stakes conditions. We discuss conditions under which we expect these results will and will not generalize to other validity scales.

A Checklist to Address Implicit Bias in Healthcare Settings During the COVID-19 Pandemic: The PLACE Strategy.

A Checklist to Address Implicit Bias in Healthcare Settings During the COVID-19 Pandemic: The PLACE Strategy.

Assessing distinguishable social skills in medical admission: does construct-driven development solve validity issues of situational judgment tests?

BackgroundSocial skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJTs) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs.MethodThe final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior.ResultsThe proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures.ConclusionsResults suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended.

Joint Consideration of Validity Indicators Embedded in Conners’ Adult ADHD Rating Scales (CAARS)

A decade of research has both illustrated the need for accurate clinical assessment of adult ADHD and brought forward a series of validity indicators assisting this diagnostic process. Several of these indicators have been embedded into Conners’ Adult ADHD Rating Scales (CAARS). As their different theoretical underpinnings offer the opportunity of possible synergy effects, the present study sought to examine whether the item- or index-wise combination of multiple validity indicators benefits classification accuracy. A sample of controls (n = 856) and adults with ADHD (n = 72) answered the CAARS, including the ADHD Credibility Index (ACI) honestly, while a group of instructed simulators (n = 135) completed the instrument as though they had ADHD. First, original CAARS items, which are part of the CAARS Infrequency Index (CII), and items drawn from the ACI were combined into a new CII-ACI-Compound Index. Secondly, existing validity indicators, including suspect T-score elevations and the CII, were considered in combination. Both approaches were evaluated in terms of sensitivity and specificity. The combination of four CII and five ACI items into the CII-ACI-Compound Index yielded a sensitivity between 41 and 51% and an estimated specificity above 87%. Suspect T-score elevations on all three DSM scales emerged as another potentially useful validity indicator with a sensitivity of 45 to 46% and a specificity > 90%. Deeming examinees non-credible whenever two or more validity indicators showed suspect results ensured low false-positive rates (< 10%), but reduced sensitivity significantly. Classifying respondents as non-credible as soon as any given indicator fell into the suspect range resulted in frequent false positives (> 11% of misclassified adults with ADHD). Depending on whether high specificity or high sensitivity is prioritized, such combined considerations offer valuable additions to individual validity indicators. High sensitivity provided by “either/or” combinations could prove useful in screening settings, whereas high stakes settings could benefit from “and” combinations.

Digital Microfluidic Hemagglutination Assays for Blood Typing, Donor Compatibility Testing, and Hematocrit Analysis.

Blood typing, donor compatibility testing, and hematocrit analysis are common tests that are important in many clinical applications, including those found in high-stakes settings such as the trauma center. These tests are typically performed in centralized laboratories with sample batching; the minutes that are lost in this mode can lead to adverse outcomes, especially for critical-care patients. As a step toward providing rapid results at the bedside, we developed a point-of-care hemagglutination system relying on digital microfluidics (DMF) and a unique, automated readout tool, droplet agglutination assessment using digital microfluidics (DAAD). ABO and Rhesus blood grouping, donor crossmatching, and hematocrit assays were developed on a portable DMF platform that allowed for automated sample processing. The result of each assay could be determined by eye or automatically with the DAAD imaging tool. DMF-DAAD was applied to 109 samples collected from different sources (including commercial samples, pinpricks from volunteers, and a hospital blood bank), with perfect fidelity to gold-standard results. Some of these tests were carried out by a nonexpert in a hospital trauma center. Proof-of-concept results were also collected from smaller sample sets for donor compatibility testing and hematocrit analysis. DMF-DAAD shows promise for delivering rapid, reliable results in a format well suited for a trauma center and other settings where every minute counts.

Is the Intellectual Functioning Component of AAIDD's 12th Manual Satisficing?

Is the Intellectual Functioning Component of AAIDD's 12th Manual Satisficing?

Group Agency and Artificial Intelligence

The aim of this exploratory paper is to review an under-appreciated parallel between group agency and artificial intelligence. As both phenomena involve non-human goal-directed agents that can make a difference to the social world, they raise some similar moral and regulatory challenges, which require us to rethink some of our anthropocentric moral assumptions. Are humans always responsible for those entities’ actions, or could the entities bear responsibility themselves? Could the entities engage in normative reasoning? Could they even have rights and a moral status? I will tentatively defend the (increasingly widely held) view that, under certain conditions, artificial intelligent systems, like corporate entities, might qualify as responsible moral agents and as holders of limited rights and legal personhood. I will further suggest that regulators should permit the use of autonomous artificial systems in high-stakes settings only if they are engineered to function as moral (not just intentional) agents and/or there is some liability-transfer arrangement in place. I will finally raise the possibility that if artificial systems ever became phenomenally conscious, there might be a case for extending a stronger moral status to them, but argue that, as of now, this remains very hypothetical.

Enacting the Rubric: Teacher Improvements in Windows of High-Stakes Observation

Abstract Teacher evaluation systems that use in-class observations, particularly in high-stakes settings, are frequently understood as accountability systems intended as nonintrusive measures of teacher quality. Presumably, the evaluation system motivates teachers to improve their practice—an accountability mechanism—and provides actionable feedback for improvement—an information mechanism. No evidence exists, however, establishing the causal link between an evaluation program and daily teacher practices. Importantly, it is unknown how teachers may modify their practice in the time leading up to an unannounced in-class observation, or how they integrate feedback into their practice post-evaluation, a question that fundamentally changes the design and philosophy of teacher evaluation programs. We disentangle these two effects with a unique empirical strategy that exploits random variation in the timing of in-class observations in the Washington, DC, teacher evaluation program IMPACT. Our key finding is that teachers work to improve during periods in which they are more likely to be observed, and they improve with subsequent evaluations. We interpret this as evidence that both mechanisms are at work, and as a result, policy makers should seriously consider both when designing teacher evaluation systems.

An inexpensive retrospective standard setting method based on item facilities

BackgroundStandard setting is one of the most challenging aspects of assessment in high-stakes healthcare settings. The Angoff methodology is widely used, but poses a number of challenges, including conceptualisation of the just-passing candidate, and the time-cost of implementing the method. Cohen methodologies are inexpensive and rapid but rely on the performance of an individual candidate. A new method of standard setting, based on the entire cohort and every item, would be valuable.MethodsWe identified Borderline candidates by reviewing their performance across all assessments in an academic year. We plotted the item scores of the Borderline candidates in comparison with Facility for the whole cohort and fitted curves to the resulting distribution.ResultsIt is observed that for any given Item, an equation of the formy ≈ C. eFxwhere y is the Facility of Borderline candidates on that Item, x is the observed Item Facility of the whole cohort, and C and F are constants, predicts the probable Facility for Borderline candidates over the test, in other words, the cut score for Borderline candidates. We describe ways of estimating C and F in any given circumstance, and suggest typical values arising from this particular study: that C = 12.3 and F = 0.021.ConclusionsC and F are relatively stable, and that the equationy = 12.3. e0.021xcan rapidly be applied to the item Facility for every item. The average value represents the cut score for the assessment as a whole. This represents a novel retrospective method based on test takers.Compared to the Cohen method which draws on one score and one candidate, this method draws on all items and candidates in a test. We propose that it can be used to standard set a whole test, or a particular item where the predicted Angoff score is very different from the observed Facility.

Conjunctive standards in OSCEs: The why and the how of number of stations passed criteria

Introduction Many institutions require candidates to achieve a minimum number of OSCE stations passed (MNSP) in addition to the aggregate pass mark. The stated rationale is usually that this conjunctive standard prevents excessive degrees of compensation across an assessment. However, there is a lack of consideration and discussion of this practice in the medical education literature. Methods We consider the motivations for the adoption of the MNSP from the assessment designer perspective, outlining potential concerns about the complexity of what the OSCE is trying to achieve, particularly around the blueprinting process and the limitations of scoring instruments. We also introduce four potential methods for setting an examinee-centred MNSP standard, and highlight briefly the theoretical advantages and disadvantages of these approaches. Discussion and conclusion There are psychometric arguments for and against the limiting of compensation in OSCEs, but it is clear that many stakeholders value the application of an MNSP standard. This paper adds to the limited literature on this important topic and notes that current MNSP practices are often problematic in high stakes settings. More empirical work is needed to develop understanding of the impact on pass/fail decision-making of the proposed standard setting methods developed in this paper.

Do Social Desirability Scales Measure Dishonesty?

Abstract. Social Desirability (SD) scales are sometimes treated, by researchers, as measures of dishonesty and, by practitioners, as indicators of faking on self-report assessments in high-stakes settings, such as personnel selection. Applying SD scales to measure dishonesty or faking, however, remains a point of contention among the scientific community. This two-part study investigated if SD scales, with a True/ False response format, are valid for these purposes. Initially, 46 participants completed an SD scale and 12 personality items while under instruction to “think aloud”, that is, to verbalize all the thoughts they had. These spoken thoughts were recorded and transcribed. Next, 175 judges rated the participants’ honesty in relation to each SD item, based on the participants’ transcribed spoken thoughts and their selected response to the item. The results showed that responses keyed as “socially desirable responding” were judged as significantly less honest than those not keyed as such. However, the effect size was very small, and the socially desirable responses were still being judged as somewhat honest overall. Further, participants’ SD scale sum scores were not related to the judges’ ratings of participant honesty on the personality items. Thus, overall, SD scales appear to be a poor measure of dishonesty.

Clinical information extraction for preterm birth risk prediction.

This paper contributes to the pursuit of leveraging unstructured medical notes to structured clinical decision making. In particular, we present a pipeline for clinical information extraction from medical notes related to preterm birth, and discuss the main challenges as well as its potential for clinical practice. A large collection of medical notes, created by staff during hospitalizations of patients who were at risk of delivering preterm, was gathered and analyzed. Based on an annotated collection of notes, we trained and evaluated information extraction components to discover clinical entities such as symptoms, events, anatomical sites and procedures, as well as attributes linked to these clinical entities. In a retrospective study, we show that these are highly informative for clinical decision support models that are trained to predict whether delivery is likely to occur within specific time windows, in combination with structured information from electronic health records.

Burnout or Compassion Fatigue? A Comparative Concept Analysis for Nurses Caring for Patients in High-Stakes Environments

Burnout and compassion fatigue are two distinct concepts experienced by nurses caring for patients in high-stakes environments. Nurses often do not recognize which concept they are experiencing due to the similarities and interchangeable use of these terms in literature. Nurses in high-stakes settings need to have these concepts further explored as they impact their physical and psychological health. This comparative concept analysis examines these terms using Walker and Avant methodology. Defining attributes, antecedents, consequences, empirical referents, and constructed cases are discussed. This analysis adds to the nursing knowledge needed to support nurses in achieving optimal occupational health and well-being.

Examining personality for the selection and classification of soldiers: Validity and differential validity across jobs

ABSTRACTA number of past studies have demonstrated that personality traits are modest predictors of workplace attitudes and behaviors and can provide incremental validity over cognitive ability. However, less is known about the utility of personality for job classification. In addition, concerns about the effects of faking on personality measures still remain. In this study, we examined the validity of a forced choice personality measure administered under operational conditions to explore the use of personality traits in high-stakes settings. In addition, we also examined the potential use of personality for classification into military occupational specialties (MOS). We explored these issues in a large sample of Soldiers from five different MOS to examine the prediction of performance during initial military training (IMT). Results indicated that composites of personality traits were valid predictors of performance and attrition and that these composites may be useful for classifying individuals into different military occupations. The implications of these results for Soldier selection and classification are discussed.

Stereotype threat effects in settings with features likely versus unlikely in operational test settings: A meta-analysis.

The stereotype threat literature primarily comprises lab studies, many of which involve features that would not be present in high-stakes testing settings. We meta-analyze the effect of stereotype threat on cognitive ability tests, focusing on both laboratory and operational studies with features likely to be present in high stakes settings. First, we examine the features of cognitive ability test metric, stereotype threat cue activation strength, and type of nonthreat control group, and conduct a focal analysis removing conditions that would not be present in high stakes settings. We also take into account a previously unrecognized methodological error in how data are analyzed in studies that control for scores on a prior cognitive ability test, which resulted in a biased estimate of stereotype threat. The focal sample, restricting the database to samples utilizing operational testing-relevant conditions, displayed a threat effect of d = -.14 (k = 45, N = 3,532, SDδ = .31). Second, we present a comprehensive meta-analysis of stereotype threat. Third, we examine a small subset of studies in operational test settings and studies utilizing motivational incentives, which yielded d-values ranging from .00 to -.14. Fourth, the meta-analytic database is subjected to tests of publication bias, finding nontrivial evidence for publication bias. Overall, results indicate that the size of the stereotype threat effect that can be experienced on tests of cognitive ability in operational scenarios such as college admissions tests and employment testing may range from negligible to small. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Investigating faking effects on the construct validity through the Monte Carlo simulation study

Personality tests have been widely used for managerial and organizational decision-making in high-stakes settings. Despite its popularity, the use of personality tests in high-stakes settings has been criticized due to faking. The existing literature reached somewhat inconsistent conclusions concerning the faking effect on the construct validity and the emergence and meaning of general factor of personality (GFP). However, it has not proposed clear moderators of the inconsistent research evidence. To fill the research gap, this research investigated the effect of various faking conditions (e.g., prevalence of fakers, percentage of faked items, and magnitude of faking effect) on the construct validity of the Big Five personality measure through a Monte Carlo simulation. The results provide important evidence of how construct validity can be affected by faking under motivated testing conditions. We discuss practical implications of the results and offer suggestions for future research.