Stakes Contexts Research Articles

"Thoughts & Prayers" or " ❤️ & 🙏 ": How the Release of New Reactions on CaringBridge Reshapes Supportive Communication in Health Crises

Following Facebook's introduction of the "Like" in 2009, CaringBridge (a nonprofit health journaling platform) implemented a "Heart" symbol as a single-click reaction affordance in 2012. In 2016, Facebook expanded its Like into a set of emotion-based reactions. In 2021, CaringBridge likewise added three new reactions: "Prayer", "Happy", and "Sad." Through user surveys (N=808) and interviews (N=13), we evaluated this product launch. Unlike Likes on mainstream social media, CaringBridge's single-click Heart was consistently interpreted as a simple, meaningful expression of acknowledgement and support. Although most users accepted the new reactions, the product launch transformed user perceptions of the feature and ignited major disagreement regarding the meanings and functions of reactions in the high stakes context of health crises. Some users found the new reactions to be useful, convenient, and reducing of caregiver burden; others felt they cause emotional harms by stripping communication of meaningful expression and authentic care. Overall, these results surface tensions for small social media platforms that need to survive amidst giants, as well as highlighting crucial trade-offs between the cognitive effort, meaningfulness, and efficiency of different forms of Computer-Mediated Communication (CMC). Our work provides three contributions to support researchers and designers in navigating these tensions: (1) empirical knowledge of how users perceived the reactions launch on CaringBridge; (2) design implications for improving health-focused CMC; and (3) concrete questions to guide future research into reactions and health-focused CMC.

Designing human resource management systems in the age of AI

The increasing adoption of artificial intelligence (AI) is reshaping the practices of human resource management (HRM). We propose a typology of HR–AI collaboration systems across the dimensions of task characteristics (routine vs. non-routine; low vs. high cognitive complexity) and social acceptability of such systems among organizational members. We discuss how organizations should design HR–AI collaboration systems in light of issues of AI explainability, high stakes contexts, and threat to employees’ professional identities. We point out important design considerations that may affect employees' perceptions of organizational fairness and emphasize HR professionals' role in the design process. We conclude by discussing how our Point of View article contributes to literatures on organization design and human–AI collaboration and suggesting potential avenues for future research.

High Performers = Better Leaders? Evidence From 55 Years of Professional Soccer on the Validity of Performance-based Promotion to Leader Positions

Promoting high-performing employees to leadership positions is a pervasive practice and has high face validity. However, little is known about the actual link between employee and subsequent leader performance as prior results are inconsistent. Given the prevalence of this performance-based promotion strategy, we conducted a study to address this inconsistency. To account for prior diverging results, we (a) competitively tested predictions from different theoretical perspectives (i.e., the performance requirements perspective, the follower-centric perspective, and the Theory of Expert Leadership), (b) considered possible changes in the predictive validity of this strategy over time, and (c) included job complexity as potential moderator of the link between employee and subsequent leader performance. In a high stakes context (i.e., the first German soccer league), we tested the predictive validity of employee performance for leader performance. Our results suggest a low validity of performance-based promotion, as we could not find evidence for a link between employee performance and leader performance—neither initially following the promotion nor over time, which is most in line with the performance requirements perspective. We, thus, caution against the (sole) application of performance-based promotion principles.

Values in Emotion Artificial Intelligence Hiring Services: Technosolutions to Organizational Problems

Despite debates about emotion artificial intelligence's (EAI) validity, legality, and social consequences, EAI is increasingly present in the high stakes context of hiring, with potential to shape the future of work and the workforce. The values laden in technology play a significant role in its societal impact.We conducted qualitative content analysis on the public-facing websites (N=229) of EAI hiring services. We identify the organizational problems that EAI hiring services claim to solve and reveal the values emerging in desired EAI uses as promoted by EAI hiring services to solve organizational problems. Our findings show that EAI hiring services market their technologies as technosolutions to three purported organizational hiring problems: 1) hiring (in)accuracy, 2) hiring (mis)fit, and 3) hiring (in)authenticity. We unpack these problems to expose how these desired uses of EAI are legitimized by the corporate ideals of data-driven decision making, continuous improvement, precision, loyalty, and stability. We identify the unfair and deceptive mechanisms by which EAI hiring services claim to solve the purported organizational hiring problems, suggesting that they unfairly exclude and exploit job candidates through EAI's creation, extraction, and affective commodification of a candidate's affective value through pseudoscientific approaches. Lastly, we interrogate EAI hiring service claims to reveal the core values that underpin their stated desired use: techno-omnipresence, techno-omnipotence, and techno-omniscience. We show how EAI hiring services position desired use of their technology as a moral imperative for hiring organizations with supreme capabilities to solve organizational hiring problems, then discuss implications for fairness, ethics, and policy in EAI-enabled hiring within the US policy landscape.

Longitudinal assessment of competency development at The Ohio State University using the competency-based veterinary education (CBVE) model.

With the development of the American Association of Veterinary Medical Colleges' Competency-Based Veterinary Education (CBVE) model, veterinary schools are reorganizing curricula and assessment guidelines, especially within the clinical rotation training elements. Specifically, programs are utilizing both competencies and entrustable professional activities (EPAs) as opportunities for gathering information about student development within and across clinical rotations. However, what evidence exists that use of the central tenets of the CBVE model (competency framework, milestones and EPAs) improves our assessment practices and captures reliable and valid data to track competency development of students as they progress through their clinical year? Here, we report on validity evidence to support the use of scores from in-training evaluation report forms (ITERs) and workplace-based assessments of EPAs to evaluate competency progression within and across domains described in the CBVE, during the final year clinical training period of The Ohio State University's College of Veterinary Medicine (OSU-CVM) program. The ITER, used at the conclusion of each rotation, was modified to include the CBVE competencies that were assessed by identifying the stage of student development on a series of descriptive milestones (from pre-novice to competent). Workplace based assessments containing entrustment scales were used to assess EPAs from the CBVE model within each clinical rotation. Competency progression and entrustment scores were evaluated on each of the 31 rotations offered and high-stakes decisions regarding student performance were determined by a collective review of all the ITERs and EPAs recorded for each learner across each semester and the entire year. Results from the class of 2021, collected on approximately 190 students from 31 rotations, are reported with more than 55 299 total competency assessments combined with milestone placement and 2799 complete EPAs. Approximately 10% of the class was identified for remediation and received additional coaching support. Data collected longitudinally through the ITER on milestones provides initial validity evidence to support using the scores in higher stakes contexts such as identifying students for remediation and for determining whether students have met the necessary requirements to successfully complete the program. Data collected on entrustment scores did not, however, support such decision making. Implications are discussed.

Handle with care: Jury deliberation and demeanour-based assessments of witness credibility

It is unclear how effectively jurors perform their task of assessing witness credibility. Drawing on evidence from a mock jury study involving 863 mock jurors deliberating across 64 juries, and building on existing research, this paper explores juries’ reliance on demeanour. While jurors make use of factors which the research literature suggests are often appropriate credibility markers, for example external consistency of accounts, there is cause for concern over the nuance with which jurors apply those assessments in high stakes contexts. The manner in which jurors look to manner of delivery as evidence of credibility is also problematic. The paper makes the case for a more circumspect approach towards jurors’ use of demeanour assessments. At a minimum, this requires that judicial directions no longer advocate their reliability, but remind jurors of the complexities associated with such assessments and the need to treat any conclusions grounded on presentational cues with caution.

Identifying positive impression management using the MMPI-2 and the MMPI-2-RF: A meta-analysis

Objective:Individuals may be motivated to conceal or minimize psychological symptoms and engage in positive impression management (PIM) in order to achieve desired outcomes across high stakes contexts (e.g. fitness for duty evaluations, pre-sentencing assessments, medical procedure evaluations, civil commitment). Despite the importance of this topic, the most recent meta-analysis of the MMPI-2, a widely used instrument for detecting PIM, was conducted more than two decades ago. Method:Using a Robust Variance Estimation method, this meta-analysis synthesized the results of 27 studies that examined the MMPI-2 (k = 22) and MMPI-2-RF (k = 5) validity scales’ ability to discriminate individuals who engage in PIM from genuine responders, with a particular focus on the L, K, and S scales. Results:The MMPI-2 L scale produced the largest effect size (g = 1.30), whereas the MMPI-2-RF L-r scale effect size was moderate (g = 1.16). Moderate effect sizes were also found for the K (g = 1.01) and K-r (g = 1.21) scales, and for the MMPI-2 S scale (g = 1.23). Conclusions: Effect sizes did not significantly vary between the two versions of the MMPI. Findings suggest that both versions of the MMPI have demonstrated utility in identifying PIM, but clinicians should interpret T scores conservatively to account for the modest elevations associated with defensiveness. Findings are discussed in the context of the recently released MMPI-3.

Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ?

The onset of the COVID-19 pandemic brought with it the closure of many bricks-and-mortar testing centres and a concomitant move to the remote (online) proctoring of many credentialing exams that were scheduled to take place throughout 2020. In effect, this meant changing from a standardized to a non-standardized administration of tests given that candidates could take tests at home using personal desktop or laptop devices. This in turn begs the question: do outcomes across tests taken in testing centres and via live remote proctoring differ? Due to the relative novelty of remote proctoring, research addressing this question is currently thin on the ground, thus justifying the research described in this paper. The study uses data from eleven professional licensure examinations taken by 14,097 candidates across four US States to compare outcomes for tests proctored either in test centres or remotely in real-time using live remote proctoring (LRP) software. Candidate outcomes were compared using average percent correct and passing rates. Test psychometric properties were compared using KR-20 reliability, decision consistency (Subkoviak's c), item difficulty, item discrimination and time taken to complete tests. At the individual test level, while some statistically significant differences were observed in outcomes across the two proctoring modes, most effect sizes were small. Overall, with the exception of some observable differences in metrics pertaining to item discrimination and timing, no detectable pattern was observed in favour of either mode. The study is significant in that it provides early evidence supporting the use of LRP in high stakes contexts.

Modeling the Feedback Process in Teams: A Field Study of Teamwork

Research on teamwork has considered the effects of different types of feedback and information exchanged as teams coordinate their actions and has described team processes in terms of phases of task performance. Interpersonal communication processes represent the behavioral mechanism by which teams create shared mental models, but it is not clear how teams use and exchange feedback as part of these processes as they perform together. Using a 5-year-period, grounded theory methodology, we investigated how action teams exchange feedback to achieve teamwork. We examined the feedback process of three different elite French military teams. Our findings offer new evidence that action teams working in extreme/high stakes contexts use different types of feedback from multiple sources and that certain feedback sources are more relevant and certain feedback practices are more prevalent at certain stages. Specifically, throughout briefing, team task performance, formal debriefing, and informal debriefing stages, the teams we studied used information from multiple sources as they engaged in technique, reflexive, and socialization feedback practices. Teamwork was evident as the teams coordinated their action during recurring performance cycles. We use our findings to model the feedback process for teamwork and discuss implications for research and practice.

High stakes assessment policy implementation in the time of COVID-19: the case of calculated grades in Ireland

This paper provides a perspective on the manner in which Irish post-primary teachers interpreted and implemented a set of guidelines created by the Department of Education and Skills (DES) in Ireland when faced with the cancellation of the traditional high stakes Leaving Certificate (LC) examination due to COVID-19. Subject teachers were asked to engage with a system of calculated grades whereby they would estimate a percentage mark and a class rank for each of their students before meeting with school colleagues to agree a final set of data to be submitted for national standardisation. This was a remarkable event in Irish education as teachers had never before been directly involved in assessing their own students for certification purposes. Data from a survey conducted with teachers (n = 713) show that a wide variety of evidence was used to support their judgements and that the DES guidelines were not always implemented as intended. Challenges highlighted in the paper include decision making around grade boundaries, the lack of evidence for newer subjects, negotiating with school colleagues, and anticipating the impact of national standardisation. The study findings will be of interest to future initiatives involving the professional judgement of teachers in high stakes contexts.

In the eye of the beholder: Considering culture in assessing the social desirability of personality.

Multinational organizations are increasingly looking to deploy assessments on a global basis. However, the social desirability of different personality characteristics may vary as a function of culture, yet limited research has explored this idea. Based on the GLOBE cultural dimensions and the theory of purposeful behavior, we examined potential connections between cultural practice dimensions and the desirability of personality aspects with a large personality item bank, utilizing raters across 34 countries. Findings indicated few connections of societal level cultural practices and social desirability perceptions for particular aspects. An exception to this was the finding that higher social desirability ratings were given for cooperativeness, sensitivity, and assertiveness in Confucian Asian, South Asian, and Sub-Saharan contexts compared with Germanic Europe, Eastern Europe, Latin America, and Latin Europe contexts. Limitations of the study as well as implications for development and use of personality assessments globally in high stakes contexts are discussed. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The PCL–R and capital sentencing: A commentary on “Death is different” DeMatteo et al. (2020a).

DeMatteo et al. (2020a) published a Statement in this journal declaring that the Psychopathy Checklist-Revised (PCL-R) “cannot and should not” be used in U.S. capital-sentencing cases to assess risk for serious institutional violence. Their stated concerns were the PCL-R’s “imperfect interrater reliability,” its “variability in predictive validity,” and its prejudicial effects on the defendant. In a Cautionary Note, we (Olver et al., 2020) raised questions about the Statement’s evaluation of the PCL-R’s psychometric properties, presented new data, including a meta-meta-analysis, and argued that the evidence did not support the Statement’s declaration that the PCL-R “cannot” be used in high stakes contexts. In their reply, titled “Death is Different,” DeMatteo et al. (2020b) concurred with several points in our Cautionary Note, disputed others, asserted that we had misunderstood or mischaracterized their Statement, and dismissed our new data and comments as irrelevant to the Statement’s purpose. This perspective on our commentary is inimical to balanced academic discourse. In this article, we contend that DeMatteo et al. (2020b) underestimated the reliability and predictive validity of PCL-R ratings, overestimated the centrality of the PCL-R in sentencing decisions, and underplayed the importance of other factors. Most of their arguments depended on sources other than capital cases, including mock trials, Sexually Violent Predator (SVP) hearings, and studies that included the prediction of general violence. We conclude that the rationale for the bold “cannot and should not” decree is open to debate and in need of research in real-life venues.

Assertion, Stakes and Expected Blameworthiness: An Insensitive Invariantist Solution to the Bank Cases

Contextualists and Subject Sensitive Invariantists often cite the knowledge norm of assertion as part of their argument. They claim that the knowledge norms in conjunction with our intuitions about when a subject is properly asserting in low or high stakes contexts provides strong evidence that what counts as knowledge depends on practical factors. In this paper, I present new data to suggest they are mistaken in the way they think about cases involving high and low stakes and I show how insensitive invariantists can explain the data. I exploit recent work done on the distinction between flouting a norm and being blamed for that violation to formulate a rigorous theory of rational expected blameworthiness that allows insensitive invariantists to explain the data cited.

Introduction to the Special Section on High Stakes Online Learning Design Cases

What do medical professionals, local government officials, and preservice teachers have in common? These learners often have regulated initial and continuing education requirements that require special considerations. For some, the content of the learning experience must include specific regulatory and legal information. For others, the content may be particularly sensitive, requiring a delicacy with respect to content presentation not often encountered in other learning designs.
 Learning designed for these audiences represent high stakes contexts that involve high risk if executed poorly and/or hold inherently high value for the learners. Misleading or misrepresented information can result in legal ramifications for and to these designers and their learners. Complicating the high stakes design constraint, these designers often find themselves designing their learning for online delivery.

Using the early childhood environment rating scale-Revised in high stakes contexts: Does evidence warrant the practice?

Increasingly, states establish different thresholds on the Early Childhood Environment Rating Scale–Revised (ECERS–R), and use these thresholds to inform high-stakes decisions. However, the validity of the ECERS-R for these purposes is not well established. The objective of this study is to identify thresholds on the ECERS-R that are associated with preschool-aged children’s social and cognitive development. Applying non-parametric modeling to the nationally-representative Early Childhood Longitudinal Study Birth Cohort (ECLS-B) dataset, we found that once classrooms achieved a score of 3.4 on the overall ECERS-R composite score, there was a leveling-off effect, such that no additional improvements to children’s social, cognitive, or language outcomes were observed. Additional analyses found that ECERS-R subscales that focused on teaching and caregiving processes, as opposed to the physical environment, did not show leveling-off effects. The findings suggest that the usefulness of the ECERS-R for discerning associations with children’s outcome may be limited to certain score ranges or subscales.

Predicting success in medical school: a longitudinal study of common Australian student selection tools.

BackgroundMedical student selection and assessment share an underlying high stakes context with the need for valid and reliable tools. This study examined the predictive validity of three tools commonly used in Australia: previous academic performance (Grade Point Average (GPA)), cognitive aptitude (a national admissions test), and non-academic qualities of prospective medical students (interview).MethodsA four year retrospective cohort study was conducted at Flinders University Australia involving 382 graduate entry medical students first enrolled between 2006 and 2009. The main outcomes were academic and clinical performance measures and an indicator of unimpeded progress across the four years of the course.ResultsA combination of the selection criteria explained between 7.1 and 29.1 % of variance in performance depending on the outcome measure. Weighted GPA consistently predicted performance across all years of the course. The national admissions test was associated with performance in Years 1 and 2 (pre-clinical) and the interview with performance in Years 3 and 4 (clinical). Those students with higher GPAs were more likely to have unimpeded progress across the entire course (OR = 2.29, 95 % CI 1.57, 3.33).ConclusionsThe continued use of multiple selection criteria to graduate entry medical courses is supported, with GPA remaining the single most consistent predictor of performance across all years of the course. The national admissions test is more valuable in the pre-clinical years, and the interview in the clinical years. Future selections research should develop the fledgling research base regarding the predictive validity of the Graduate Australian Medical School Admissions Test (GAMSAT), the algorithms for how individual tools are combined in selection, and further explore the usefulness of the unimpeded progress index.

We Are What We Repeatedly Do—But in What Context? The Role of Situational Factors in Assessing Personality via Nonverbal Behavior

AbstractWhile the vast majority of assessments designed to measure noncognitive skills rely heavily on self‐report, concerns surrounding faking and response bias have led to an increased demand for the development of new and innovative methods by which to measure such traits, particularly in high stakes contexts. The focus of this report is to address one such method—the inference of noncognitive skills, namely personality, through nonverbal behavior—and the role contextual factors may play in influencing the validity of conclusions drawn from its application. Specifically, this report discusses the various ways in which characteristics of the task (e.g., level of cooperation, physical effort) may facilitate or hinder the assessment of personality when personality is being inferred through observable behavior. This discussion is dovetailed by a review of research from the nonverbal assessment and interpersonal task literatures, as well as a synthesis of these literatures to highlight the degree to which situational factors (as manifested through different tasks) influence the assessment of personality via nonverbal behavior. Additionally, the potential value of using noninvasive tools, such as the sociometer, to collect nonverbal behavioral data during these tasks is discussed. The report concludes with a brief commentary on the feasibility of using nonverbal methods to assess noncognitive skills, with a specific focus on the extent to which structuring the context to elicit certain behaviors may influence the validity and robustness of nonverbal responses as a measurement source.

The Consistencies and Vagaries of the Washington State Inventory of Evidence-Based Practice: The Definition of "Evidence-Based" in a Policy Context.

As states increasingly establish the importance of evidence-based practice through policy and funding mandates, the definition of evidence-based practice can have a significant impact on investment decisions. Not meeting established criteria can mean a loss of funding for established programs and the implementation disruption of programs without a strong research base. Whether the definition of "evidence-based" is influenced by these high stakes contexts is an interesting question that can inform the larger field about the value and utility of evidence-based practice lists/inventories for disseminating knowledge. In this paper we review the development of the Washington State Inventory of Evidence-Based, Research-Based and Promising Practices as a case study for the process of defining evidence-based practice in a policy context. As part of this study we also present a comparison of other well-known evidence-based practice inventories and examine consistencies and differences in the process of identifying and developing program ratings.

Selfish play increases during high-stakes NBA games and is rewarded with more lucrative contracts.

High-stakes team competitions can present a social dilemma in which participants must choose between concentrating on their personal performance and assisting teammates as a means of achieving group objectives. We find that despite the seemingly strong group incentive to win the NBA title, cooperative play actually diminishes during playoff games, negatively affecting team performance. Thus team cooperation decreases in the very high stakes contexts in which it is most important to perform well together. Highlighting the mixed incentives that underlie selfish play, personal scoring is rewarded with more lucrative future contracts, whereas assisting teammates to score is associated with reduced pay due to lost opportunities for personal scoring. A combination of misaligned incentives and psychological biases in performance evaluation bring out the “I” in “team” when cooperation is most critical.

Polite incivility in defensive attack: Strategic politeness and impoliteness in cross-examination in the David Irving vs. Penguin Books Ltd and Deborah Lipstadt trial

Kurzon (Journal of Pragmatics 33: 61–85, 2001: 64) argues that the language of the courtroom is inherently polite due to the formal, high stakes context, but does this extend to all the institutional participants throughout trials? The linguistic politeness (and impoliteness) literature gives us a nuanced account of politeness activity (e. g., Culpeper, Journal of Pragmatics 25: 349–367, 1996; Harris, Discourse and Society 12: 451–472: 2001), including “expressive politeness” (Eelen, Pragmatics 9: 163–173, 1999) and “mock politeness” (Kienpointner, Journal of Politeness Research 4: 243–265, 2008: 249). We argue that any analysis of courtroom trial interaction needs to take account of the “multiple goals” (Penman, Facework and politeness: Multiple goals in courtroom discours, Multilingual Matters, 1990; Archer, Verbal aggression and impoliteness: Related or synonymous?, Mouton de Gruyter, 2008) of this “complex genre” (Heffer, The language of jury trial: A corpus-aided analysis of legallay discourse, Palgrave Macmillan, 2005) and, in doing so, we need to account for the inherently facethreatening nature of cross-examination. Drawing on Brown and Levinson's (Universals in language usage: Politeness phenomena, Cambridge University Press, 1978) notion of positive and negative face, we focus on how lawyers and witnesses use “politeness as a strategic act” and how they prioritize the “individual, creative aspects of politeness over the social aspects” (Christie, Journal of Politeness Research 3: 269–294, 2007: 263), to coerce, undermine and challenge. We use a corpus-based approach to investigate the case of the libel trial David Irving vs. Penguin Books Ltd and Deborah Lipstadt (2000). This paper examines direct and indirect impoliteness, arguing that pragmatic im/politeness strategies contribute to the style of the lawyer's defensive attack in cross-examination, one which employs both polite and impolite incivility.