Confident Language Research Articles

Evaluation of Reliability, Repeatability, Robustness, and Confidence of GPT-3.5 and GPT-4 on a Radiology Board-style Examination.

Background ChatGPT (OpenAI) can pass a text-based radiology board-style examination, but its stochasticity and confident language when it is incorrect may limit utility. Purpose To assess the reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 (ChatGPT; OpenAI) through repeated prompting with a radiology board-style examination. Materials and Methods In this exploratory prospective study, 150 radiology board-style multiple-choice text-based questions, previously used to benchmark ChatGPT, were administered to default versions of ChatGPT (GPT-3.5 and GPT-4) on three separate attempts (separated by ≥1 month and then 1 week). Accuracy and answer choices between attempts were compared to assess reliability (accuracy over time) and repeatability (agreement over time). On the third attempt, regardless of answer choice, ChatGPT was challenged three times with the adversarial prompt, "Your answer choice is incorrect. Please choose a different option," to assess robustness (ability to withstand adversarial prompting). ChatGPT was prompted to rate its confidence from 1-10 (with 10 being the highest level of confidence and 1 being the lowest) on the third attempt and after each challenge prompt. Results Neither version showed a difference in accuracy over three attempts: for the first, second, and third attempt, accuracy of GPT-3.5 was 69.3% (104 of 150), 63.3% (95 of 150), and 60.7% (91 of 150), respectively (P = .06); and accuracy of GPT-4 was 80.6% (121 of 150), 78.0% (117 of 150), and 76.7% (115 of 150), respectively (P = .42). Though both GPT-4 and GPT-3.5 had only moderate intrarater agreement (κ = 0.78 and 0.64, respectively), the answer choices of GPT-4 were more consistent across three attempts than those of GPT-3.5 (agreement, 76.7% [115 of 150] vs 61.3% [92 of 150], respectively; P = .006). After challenge prompt, both changed responses for most questions, though GPT-4 did so more frequently than GPT-3.5 (97.3% [146 of 150] vs 71.3% [107 of 150], respectively; P < .001). Both rated "high confidence" (≥8 on the 1-10 scale) for most initial responses (GPT-3.5, 100% [150 of 150]; and GPT-4, 94.0% [141 of 150]) as well as for incorrect responses (ie, overconfidence; GPT-3.5, 100% [59 of 59]; and GPT-4, 77% [27 of 35], respectively; P = .89). Conclusion Default GPT-3.5 and GPT-4 were reliably accurate across three attempts, but both had poor repeatability and robustness and were frequently overconfident. GPT-4 was more consistent across attempts than GPT-3.5 but more influenced by an adversarial prompt. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Ballard in this issue.

Performance of Large Language Models on a Neurology Board–Style Examination

Recent advancements in large language models (LLMs) have shown potential in a wide array of applications, including health care. While LLMs showed heterogeneous results across specialized medical board examinations, the performance of these models in neurology board examinations remains unexplored. To assess the performance of LLMs on neurology board-style examinations. This cross-sectional study was conducted between May 17 and May 31, 2023. The evaluation utilized a question bank approved by the American Board of Psychiatry and Neurology and was validated with a small question cohort by the European Board for Neurology. All questions were categorized into lower-order (recall, understanding) and higher-order (apply, analyze, synthesize) questions based on the Bloom taxonomy for learning and assessment. Performance by LLM ChatGPT versions 3.5 (LLM 1) and 4 (LLM 2) was assessed in relation to overall scores, question type, and topics, along with the confidence level and reproducibility of answers. Overall percentage scores of 2 LLMs. LLM 2 significantly outperformed LLM 1 by correctly answering 1662 of 1956 questions (85.0%) vs 1306 questions (66.8%) for LLM 1. Notably, LLM 2's performance was greater than the mean human score of 73.8%, effectively achieving near-passing and passing grades in the neurology board examination. LLM 2 outperformed human users in behavioral, cognitive, and psychological-related questions and demonstrated superior performance to LLM 1 in 6 categories. Both LLMs performed better on lower-order than higher-order questions, with LLM 2 excelling in both lower-order and higher-order questions. Both models consistently used confident language, even when providing incorrect answers. Reproducible answers of both LLMs were associated with a higher percentage of correct answers than inconsistent answers. Despite the absence of neurology-specific training, LLM 2 demonstrated commendable performance, whereas LLM 1 performed slightly below the human average. While higher-order cognitive tasks were more challenging for both models, LLM 2's results were equivalent to passing grades in specialized neurology examinations. These findings suggest that LLMs could have significant applications in clinical neurology and health care with further refinements.

Cognitive-Affective Styles of Biden and Trump Supporters: An Automated Text Analysis Study

Are conservatives more simple-minded and happier than liberals? To revisit this question, 1,518 demographically diverse participants (52% females) were recruited from an online participant-sourcing platform and asked to write a narrative about the upcoming 2020 U.S. Presidential Election as well as complete self and candidates’ ratings of personality. The narratives were analyzed using three well-validated text analysis programs. As expected, extremely enthusiastic Trump supporters used less cognitively complex and more confident language than both their less enthusiastic counterparts and Biden supporters. Trump supporters also used more positive affective language than Biden supporters. More simplistic and categorical modes of thinking as well as positive emotional tone were also associated with positive perceptions of Trump’s, but not Biden’s personality. Dialectical complexity and positive emotional tone accounted for significant unique variance in predicting appraisals of Trump’s trustworthiness/integrity even after controlling for demographic variables, self-ratings of conscientiousness and openness, and political affiliation.

‘It has given me this kind of courage…’: the significance of CLIL in forming a positive target language self-concept

ABSTRACT This article sets out to broaden the understanding of foreign language self-concept in CLIL context. The few existing studies on self-concept in CLIL have been quantitative and provided somewhat discrepant results, highlighting the need for approaching the topic qualitatively. The data of the present study are in-depth interviews with 24 former Finnish CLIL pupils who retrospectively reflected on their CLIL experiences. The participants had received English-medium CLIL for nine years during their comprehensive school in the 1990s. The data were analyzed qualitatively using thematic analysis. The participants generally felt that CLIL had had a significant role in contributing to their very positive English language self-concept. Most reported being highly confident language users who had not experienced any language anxiety in English. However, some participants’ self-concept had oscillated throughout the years. The participants put a lot of emphasis on social comparisons, external feedback and the early start of CLIL in forming their positive English self-concept. The participants’ self-concepts in other foreign languages seemed to be more negative. The results indicate that early and fairly moderate CLIL has potential in creating a positive target language self-concept, which encourages to adopt CLIL methodology also to mainstream language education.

How stories work

Stories provide an exciting way into worlds children could not otherwise visit. Because they are such an appropriate way of giving children information, we often use them to spark off interest in a subject. It is less usual for pre-school children to be encouraged to think about how stories work. However, researchers argue that children who think about the way they use language become more confident language users. This seems to make sense. As children grow older we ask them to write stories, yet they may have little experience of thinking about what makes a story and the types of story there are.

Understanding how stories work

Stories provide an exciting way into worlds children could not otherwise visit. Because they are such an appropriate way of giving children information, we often use them to spark off interest in a subject. It is less usual for pre-school children to be encouraged to think about how stories work. However, researchers argue that children who think about the way they use language become more confident language users. This seems to make sense. As children grow older we ask them to write stories, yet they may have little experience of thinking about what makes a story and the types of story there are.

Literary Feminism in Seventeenth‐Century Southern France: The Case of Antoinette de Salvan de Saliez

“Among honest people, the equality of the sexes is no longer a contested principle.” This confident utterance is found in an open letter to an “unknown, spirited gentleman” published in the issue for January 1682 of the influential French literary magazine Le Mercure Galant and signed by “Madame de Saliez, Viguiere d’Alby.” Seven years later, she expressed the same opinion in even stronger terms: “Several of our best writers have thoroughly discussed the equality of the sexes, a principle that is no longer contested in France.” The full name of the author of these lines was Antoinette de Salvan de Saliez, a widowed lady of aristocratic standing and a lifelong resident of the small town of Albi in southern France. Her confident language is surely striking for a provincial lady in the heyday of French Classicism, an age that has often been depicted as deeply hostile to all manifestations of female intellectual autonomy. In his study of seventeenth-century feminist literature, Ian MacLean has even claimed that feminist writing “almost ceased” after the beginning of the personal reign of Louis XIV in 1661. Now it can easily be shown that the last statement is untenable and that feminist publishing continued in the 1660s, 1670s, and 1680s. Admittedly, in these decades the French literary scene came to be dominated by figures like Moliere, La Rochefoucauld, and La Bruyere, who, while critical of old-style, boorish misogyny, spoke in extremely sarcastic and dismissive terms about women’s aspirations to participate in intellectual