General Modeling Language Research Articles

Learning to Rank in Generative Retrieval

Generative retrieval stands out as a promising new paradigm in text retrieval that aims to generate identifier strings of relevant passages as the retrieval target. This generative paradigm taps into powerful generative language models, distinct from traditional sparse or dense retrieval methods. However, only learning to generate is insufficient for generative retrieval. Generative retrieval learns to generate identifiers of relevant passages as an intermediate goal and then converts predicted identifiers into the final passage rank list. The disconnect between the learning objective of autoregressive models and the desired passage ranking target leads to a learning gap. To bridge this gap, we propose a learning-to-rank framework for generative retrieval, dubbed LTRGR. LTRGR enables generative retrieval to learn to rank passages directly, optimizing the autoregressive model toward the final passage ranking target via a rank loss. This framework only requires an additional learning-to-rank training phase to enhance current generative retrieval systems and does not add any burden to the inference stage. We conducted experiments on three public benchmarks, and the results demonstrate that LTRGR achieves state-of-the-art performance among generative retrieval methods. The code and checkpoints are released at https://github.com/liyongqi67/LTRGR.

Emotion detection in educational dialogues by transfer learning

Emotion detection in educational dialogues, particularly within student-teacher interactions, has become a crucial research area for improving the learning experience. In this paper, we employ two models, one generic Bidirectional Encoder Representations from Transformers (BERT) and the Emotion detection model Robustly Optimized BERT Approach (EmoRoBERTa), to automatically classify emotions in a corpus of student-teacher chat interactions. Then subsequently, we validate these classifications using a scheme based on oracles, employing two generative large language models (ChatGPT and Bard). Experiments on emotion detection in dialogues between students and teachers revealed that EmoRoBERTa exhibited a reasonable level of agreement with the oracles, while ChatGPT demonstrated the highest consistency with EmoRoBERTa’s predictions. Furthermore, we identified the impact of specific words on emotion classification, offering insights into the decision-making process of these models. The results not only highlight the prominent presence of emotions like approval, gratitude, curiosity, disapproval, amusement, confusion, remorse, joy, and surprise but also provide substantial support for the utilization of the proposed emotion detection model to enhance the student learning environment. Exploring the emotional aspects of educational dialogues holds the potential to enhance instruction methods, provide timely assistance to students in need, and create an improved learning atmosphere.

AI-assisted literature exploration of innovative Chinese medicine formulas.

Objective: Our study provides an innovative approach to exploring herbal formulas that contribute to the promotion of sustainability and biodiversity conservation. We employ data mining, integrating keyword extraction, association rules, and LSTM-based generative models to analyze classical Traditional Chinese Medicine (TCM) texts. We systematically decode classical Chinese medical literature, conduct statistical analyses, and link these historical texts with modern pharmacogenomic references to explore potential alternatives. Methods: We present a novel iterative keyword extraction approach for discerning diverse herbs in historical TCM texts from the Pu-Ji Fang copies. Utilizing association rules, we uncover previously unexplored herb pairs. To bridge classical TCM herbal pairs with modern genetic relationships, we conduct gene-herb searches in PubMed and statistically validate this genetic literature as supporting evidence. We have expanded on the present work by developing a generative language model for suggesting innovative TCM formulations based on textual herb combinations. Results: We collected associations with 7,664 PubMed cross-search entries for gene-herb and 934 for Shenqifuzheng Injection as a positive control. We analyzed 16,384 keyword combinations from Pu-Ji Fang's 426 volumes, employing statistical methods to probe gene-herb associations, focusing on examining differences among the target genes and Pu-Ji Fang herbs. Conclusion: Analyzing Pu-Ji Fang reveals a historical focus on flavor over medicinal aspects in TCM. We extend our work on developing a generative model from classical textual keywords to rapidly produces novel herbal compositions or TCM formulations. This integrated approach enhances our comprehension of TCM by merging ancient text analysis, modern genetic research, and generative modeling.

Explanation–Question–Response dialogue: An argumentative tool for explainable AI

Advancements and deployments of AI-based systems, especially Deep Learning-driven generative language models, have accomplished impressive results over the past few years. Nevertheless, these remarkable achievements are intertwined with a related fear that such technologies might lead to a general relinquishing of our lives’s control to AIs. This concern, which also motivates the increasing interest in the eXplainable Artificial Intelligence (XAI) research field, is mostly caused by the opacity of the output of deep learning systems and the way that it is generated, which is largely obscure to laypeople. A dialectical interaction with such systems may enhance the users’ understanding and build a more robust trust towards AI. Commonly employed as specific formalisms for modelling intra-agent communications, dialogue games prove to be useful tools to rely upon when dealing with user’s explanation needs. The literature already offers some dialectical protocols that expressly handle explanations and their delivery. This paper fully formalises the novel Explanation–Question–Response (EQR) dialogue and its properties, whose main purpose is to provide satisfactory information (i.e., justified according to argumentative semantics) whilst ensuring a simplified protocol, in comparison with other existing approaches, for humans and artificial agents.

Representation Learning for Stack Overflow Posts: How Far Are We?

The tremendous success of Stack Overflow has accumulated an extensive corpus of software engineering knowledge, thus motivating researchers to propose various solutions for analyzing its content. The performance of such solutions hinges significantly on the selection of representation models for Stack Overflow posts. As the volume of literature on Stack Overflow continues to burgeon, it highlights the need for a powerful Stack Overflow post representation model and drives researchers’ interest in developing specialized representation models that can adeptly capture the intricacies of Stack Overflow posts. The state-of-the-art (SOTA) Stack Overflow post representation models are Post2Vec and BERTOverflow, which are built upon neural networks such as convolutional neural network and transformer architecture (e.g., BERT). Despite their promising results, these representation methods have not been evaluated in the same experimental setting. To fill the research gap, we first empirically compare the performance of the representation models designed specifically for Stack Overflow posts (Post2Vec and BERTOverflow) in a wide range of related tasks (i.e., tag recommendation, relatedness prediction, and API recommendation). The results show that Post2Vec cannot further improve the SOTA techniques of the considered downstream tasks, and BERTOverflow shows surprisingly poor performance. To find more suitable representation models for the posts, we further explore a diverse set of transformer-based models, including (1) general domain language models (RoBERTa, Longformer, and GPT2) and (2) language models built with software engineering related textual artifacts (CodeBERT, GraphCodeBERT, seBERT, CodeT5, PLBart, and CodeGen). This exploration shows that models like CodeBERT and RoBERTa are suitable for representing Stack Overflow posts. However, it also illustrates the “No Silver Bullet” concept, as none of the models consistently wins against all the others. Inspired by the findings, we propose SOBERT, which employs a simple yet effective strategy to improve the representation models of Stack Overflow posts by continuing the pre-training phase with the textual artifact from Stack Overflow. The overall experimental results demonstrate that SOBERT can consistently outperform the considered models and increase the SOTA performance significantly for all the downstream tasks.

Assisting the infection preventionist: Use of artificial intelligence for health care–associated infection surveillance

Health care-associated infection (HAI) surveillance is vital for safety in health care settings. It helps identify infection risk factors, enhancing patient safety and quality improvement. However, HAI surveillance is complex, demanding specialized knowledge and resources. This study investigates the use of artificial intelligence (AI), particularly generative large language models, to improve HAI surveillance. We assessed 2 AI agents, OpenAI's chatGPT plus (GPT-4) and a Mixtral 8×7b-based local model, for their ability to identify Central Line-Associated Bloodstream Infection (CLABSI) and Catheter-Associated Urinary Tract Infection (CAUTI) from 6 National Health Care Safety Network training scenarios. The complexity of these scenarios was analyzed, and responses were matched against expert opinions. Both AI models accurately identified CLABSI and CAUTI in all scenarios when given clear prompts. Challenges appeared with ambiguous prompts including Arabic numeral dates, abbreviations, and special characters, causing occasional inaccuracies in repeated tests. The study demonstrates AI's potential in accurately identifying HAIs like CLABSI and CAUTI. Clear, specific prompts are crucial for reliable AI responses, highlighting the need for human oversight in AI-assisted HAI surveillance. AI shows promise in enhancing HAI surveillance, potentially streamlining tasks, and freeing health care staff for patient-focused activities. Effective AI use requires user education and ongoing AI model refinement.

The Latest Progress in Human-Computer Dialogue System

The Human-Computer dialogue system allows a computer to make conversations with humans through natural languages, and some designs can accomplish tasks given by humans. The development of the Human-Computer dialogue system can significantly expand the range of users for computers. The profound ELIZA Program designed scripts containing keywords and their corresponding rule-based sentence transformation to respond to the user. As the computation power increases, Human-Computer dialogue systems are handling more varied and complicated scenarios by introducing deeper and more complex artificial neural networks. The designer of a task-oriented system can choose between the pipeline method and the end-to-end method. The pipeline method is basically a pipeline that lets the user input go through three parts handling Natural Language Understanding, Dialogue Management, and Natural Language Generation, to generate an appropriate response back to the user. The end-to-end method, however, uses joint models to allow the parts to interact with others and get more efficient in handling the information. With the development of ChatGPT, more general language models and more varied methodologies are becoming more prevalent in dialogue systems.

Performance of Two Artificial Intelligence Generative Language Models on the Orthopaedic In-Training Examination.

Artificial intelligence (AI) generative large language models are powerful and increasingly accessible tools with potential applications in health care education and training. The annual Orthopaedic In-Training Examination (OITE) is widely used to assess resident academic progress and preparation for the American Board of Orthopaedic Surgery Part 1 Examination. Open AI's ChatGPT and Google's Bard generative language models were administered the 2022 OITE. Question stems that contained images were input without and then with a text-based description of the imaging findings. ChatGPT answered 69.1% of questions correctly. When provided with text describing accompanying media, this increased to 77.8% correct. In contrast, Bard answered 49.8% of questions correctly. This increased to 58% correct when text describing imaging in question stems was provided (P<.0001). ChatGPT was most accurate in questions within the shoulder category, with 90.9% correct. Bard performed best in the sports category, with 65.4% correct. ChatGPT performed above the published mean of Accreditation Council for Graduate Medical Education orthopedic resident test-takers (66%). There is significant variability in the accuracy of publicly available AI models on the OITE. AI generative language software may play numerous potential roles in the future in orthopedic education, including simulating patient presentations and clinical scenarios, customizing individual learning plans, and driving evidence-based case discussion. Further research and collaboration within the orthopedic community is required to safely adopt these tools and minimize risks associated with their use. [Orthopedics. 2024;47(3):e146-e150.].

Study on the Impact of Utilizing ChatGPT and Other AI Tools for Feedback in EAP Writing Classrooms on the Discursive Writing Performance of English Major Students

This study aims to delve into the impact of utilizing ChatGPT and other AI tools on the discourse writing performance of English major students, as well as to explore their potential value and future prospects in the field of English education. With the continuous development and application of artificial intelligence technology, AI tools have gradually demonstrated their unique advantages and potential in language learning and writing instruction. However, for English major students, how to effectively utilize these technological tools to enhance their discourse writing abilities remains a subject of considerable interest. In the field of language learning, AI technology has made significant strides in vocabulary acquisition, grammar correction, and discourse generation. ChatGPT, as a natural language generation model, possesses strong language comprehension and generation capabilities and is widely used in dialogue generation, text generation, and other areas. In English education, ChatGPT can provide personalized writing guidance and feedback to students, helping them clarify their thoughts and improve their expression skills. Additionally, other AI tools such as speech recognition technology and text-to-speech technology offer more possibilities for students' English learning and writing. However, despite the numerous advantages demonstrated by AI tools in discourse writing instruction for English major students, their application also faces challenges and limitations. For instance, AI tools still have limitations in understanding academic language and specialized terminology, and they cannot fully replace human professional judgment and language application abilities. Furthermore, the widespread adoption and use of AI tools require addressing technical challenges and adaptation issues in teaching practice to ensure their effective application in education. In conclusion, this study will comprehensively explore the actual effects and potential value of AI tools in discourse writing instruction for English major students through a combination of literature review and empirical research methods, providing new insights and guidance for teaching practice and future research endeavors.

Framework-based qualitative analysis of free responses of Large Language Models: Algorithmic fidelity.

Today, with the advent of Large-scale generative Language Models (LLMs) it is now possible to simulate free responses to interview questions such as those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial "silicon participants" generated by LLMs may be productively studied using qualitative analysis methods in such a way as to generate insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a validity concept capturing the degree to which LLM-generated outputs mirror human sub-populations' beliefs and attitudes. By definition, high algorithmic fidelity suggests that latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with "silicon participants" matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of a hyper-accuracy distortion. We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect in silico research on it to generalize to real human populations. However, rapid advances in artificial intelligence raise the possibility that algorithmic fidelity may improve in the future. Thus we stress the need to establish epistemic norms now around how to assess the validity of LLM-based qualitative research, especially concerning the need to ensure the representation of heterogeneous lived experiences.

Evaluation of a Generative Language Model Tool for Writing Examination Questions

ObjectiveTo describe an evaluation of a generative language model tool to write examination questions for a new elective course focused on the interpretation of common clinical laboratory results being developed as an elective for students in a Bachelor of Science in Pharmaceutical Sciences program. MethodsA total of 100 multiple-choice questions were generated using a publicly available large language model for a course dealing with common laboratory values. Two independent evaluators with extensive training and experience in writing multiple-choice questions evaluated each question for appropriate formatting, clarity, correctness, relevancy, and difficulty. For each question, a final dichotomous judgment was assigned by each reviewer, usable as written or not usable written. ResultsThe major finding of this study was that a generative language model (ChatGPT 3.5) could generate multiple-choice questions for assessing common laboratory value information but only about half the questions (50% and 57% for the 2 evaluators) were deemed usable without modification. General agreement between evaluator comments was common (62% of comments) with more than 1 correct answer being the most common reason for commenting on the lack of usability (N = 27). ConclusionThe generally positive findings of this study suggest that the use of a generative language model tool for developing examination questions is deserving of further investigation.

CodeKGC: Code Language Model for Generative Knowledge Graph Construction

Current generative knowledge graph construction approaches usually fail to capture structural knowledge by simply flattening natural language into serialized texts or a specification language. However, large generative language model trained on structured data such as code has demonstrated impressive capability in understanding natural language for structural prediction and reasoning tasks. Intuitively, we address the task of generative knowledge graph construction with code language model: given a code-format natural language input, the target is to generate triples which can be represented as code completion tasks. Specifically, we develop schema-aware prompts that effectively utilize the semantic structure within the knowledge graph. As code inherently possesses structure, such as class and function definitions, it serves as a useful model for prior semantic structural knowledge. Furthermore, we employ a rationale-enhanced generation method to boost the performance. Rationales provide intermediate steps, thereby improving knowledge extraction abilities. Experimental results indicate that the proposed approach can obtain better performance on benchmark datasets compared with baselines. 1

Standardized nomenclature for litigational legal prompting in generative language models

With the increasing availability of commercial Artificial Intelligence, General Language Models (GLMs) have been widely explored in various domains, including law. However, to ensure accurate and standardized legal results, it is crucial to establish a consistent framework for prompting GLMs. This paper presents one of the first instances of such nomenclature, providing a robust framework of “variables” and “clauses” that enhances legal-focused results. The proposed framework was applied in diverse legal scenarios, demonstrating its potential from both client and attorney perspectives. By introducing standardized variables and clauses, legal professionals can effectively communicate with GLMs. This not only improves the accuracy of the generated outcomes but also facilitates collaboration between AI systems and legal experts. With a common framework in place, legal practitioners can leverage AI technology confidently, knowing that the results produced align with established legal principles. Furthermore, the framework serves as a foundation for future research in the field of legal prompting with GLMs, and several avenues for future research are recommended in this paper. This standardization of nomenclature is expected to contribute to the wider adoption and benefit of GLMs in the legal field, leading to more accurate and reliable outcomes.

Exploring the use of ChatGPT in predicting anterior circulation stroke functional outcomes after mechanical thrombectomy: a pilot study

BackgroundAccurate prediction of functional outcomes is crucial in stroke management, but this remains challenging.ObjectiveTo evaluate the performance of the generative language model ChatGPT in predicting the functional outcome of patients...

Incorporating evidence into mental health Q&A: a novel method to use generative language models for validated clinical content extraction

ABSTRACT Generative language models have changed the way we interact with computers using natural language. With the release of increasingly advanced GPT models, systems are able to correctly respond to questions in various domains. However, they still have important limitations, such as hallucinations, lack of substance in answers, inability to justify responses, or showing high confidence with fabricated content. In digital mental health, every decision must be traceable and based on scientific evidence and these shortcomings are hindering the integration of LLMs into clinical practice. In this paper, we provide a novel automated method to develop evidence-based question answering systems. Powerful state-of-the-art generalist language models are used and forced to employ only contents in validated clinical guidelines, tracking the source of the evidence for each generated response. This way, the system is able to protect users from hallucinatory responses. As a proof of concept, we present the results obtained building question-answering systems circumscribed to the clinical practice guidelines of the Spanish National Health System about the management of depression and attention deficit hyperactivity disorder. The coherence, veracity, and evidence supporting the responses have been evaluated by human experts obtaining high reliability, clarity, completeness, and traceability of evidence results.

Large-scale text analysis using generative language models: A case study in discovering public value expressions in AI patents

Abstract We put forward a novel approach using a generative language model (GPT-4) to produce labels and rationales for large-scale text analysis. The approach is used to discover public value expressions in patents. Using text (5.4 million sentences) for 154,934 US AI patent documents from the United States Patent and Trademark Office (USPTO), we design a semi-automated, human-supervised framework for identifying and labeling public value expressions in these sentences. A GPT-4 prompt is developed that includes definitions, guidelines, examples, and rationales for text classification. We evaluate the labels and rationales produced by GPT-4 using BLEU scores and topic modeling, finding that they are accurate, diverse, and faithful. GPT-4 achieved an advanced recognition of public value expressions from our framework, which it also uses to discover unseen public value expressions. The GPT-produced labels are used to train BERT-based classifiers and predict sentences on the entire database, achieving high F1 scores for the 3-class (0.85) and 2-class classification (0.91) tasks. We discuss the implications of our approach for conducting large-scale text analyses with complex and abstract concepts. With careful framework design and interactive human oversight, we suggest that generative language models can offer significant assistance in producing labels and rationales.

Generating Role-Playing Game Quests With GPT Language Models

Quests represent an integral part of role-playing games (RPGs). While evocative, narrative-rich quests are still mostly hand-authored, player demands towards more and richer game content, as well as business requirements for continuous player engagement necessitate alternative, procedural quest generation methods. While existing methods produce mostly uninteresting, mechanical quest descriptions, recent advances in AI have brought forth generative language models with promising computational storytelling capabilities. We leverage two of the most successful Transformer models, GPT-2 and GPT-3, to procedurally generate RPG video game quest descriptions. We gathered, processed and openly published a data set of 978 quests and their descriptions from six RPGs. We fine-tuned GPT-2 on this data set with a range of optimizations informed by several mini studies. We validated the resulting Quest-GPT-2 model via an online user study involving 349 RPG players. Our results indicate that one in five quest descriptions would be deemed acceptable by a human critic, yet the variation in quality across individual quests is large. We provide recommendations on current applications of Quest-GPT-2. This is complemented by case-studies on GPT-3 to highlight the future potential of state-of-the-art natural language models for quest generation.

Harnessing the Power of Generative AI for Clinical Summaries: Perspectives From Emergency Physicians

ImportanceThe workload of clinical documentation contributes to healthcare costs and professional burnout. The advent of generative AI language models presents a promising solution. The perspective of clinicians may contribute to effective and responsible implementation of such tools. ObjectiveTo evaluate 3 uses for generative AI for clinical documentation in pediatric emergency medicine (PEM), measuring time savings, effort reduction, physician attitudes, and identifying potential risks and barriers. DesignA mixed-methods study. SettingSingle Pediatric Emergency Department. Participants10 PEM attending physicians. InterventionParticipants were asked to write a supervisory note for four clinical scenarios, with varying level of complexity, twice without any assistance and twice with the assistance of ChatGPT Version 4.0. Participants evaluated two additional ChatGPT-generated clinical summaries: a structured handoff and a visit summary for a family, written at an 8th grade reading level. Finally, a semi-structured interview was performed to assess physicians’ perspective on the use of ChatGPT in PEM. Main Outcomes and MeasuresBetween subjects’ comparison of the effort and time taken to complete the supervisory note with and without ChatGPT assistance. Effort was measured using a self-reported Likert scale of 0-10. Physicians’ scoring of and attitude toward the ChatGPT-generated summaries was measured using a 0-10 Likert scale and open-ended questions. Summaries were scored for completeness, accuracy, efficiency, readability, and overall satisfaction. A thematic analysis was performed to analyze the content of the open-ended questions and to identify key themes. ResultsChatGPT yielded a 40% reduction in time and a 33% decrease in effort for supervisory notes in intricate cases, with no discernible impact on simpler ones. ChatGPT-generated summaries for structured handoffs and family-letters were highly rated, ranging from 7.0 to 9.0 out of 10, and most participants favored their inclusion in clinical practice. However, there were several critical reservations, out of which a set of general recommendations for applying ChatGPT to clinical summaries was formulated. ConclusionsPEM attendings in our study perceived that ChatGPT can deliver high-quality summaries while saving time and effort in many scenarios, but not all.

Application of generative language models to orthopaedic practice

ObjectiveTo explore whether large language models (LLMs) Generated Pre-trained Transformer (GPT)-3 and ChatGPT can write clinical letters and predict management plans for common orthopaedic scenarios.DesignFifteen scenarios were generated and ChatGPT...

Observing Schrödinger’s cat with artificial intelligence: emergent classicality from information bottleneck

We train a generative language model on the randomized local measurement data collected from Schrödinger’s cat quantum state. We demonstrate that the classical reality emerges in the language model due to the information bottleneck: although our training data contains the full quantum information about Schrödinger’s cat, a weak language model can only learn to capture the classical reality of the cat from the data. We identify the quantum–classical boundary in terms of both the size of the quantum system and the information processing power of the classical intelligent agent, which indicates that a stronger agent can realize more quantum nature in the environmental noise surrounding the quantum system. Our approach opens up a new avenue for using the big data generated on noisy intermediate-scale quantum devices to train generative models for representation learning of quantum operators, which might be a step toward our ultimate goal of creating an artificial intelligence quantum physicist.