General Modeling Language Research Articles

Disability Ethics and Education in the Age of Artificial Intelligence: Identifying Ability Bias in ChatGPT and Gemini

ObjectiveTo identify and quantify ability bias in generative artificial intelligence large language model chatbots, specifically OpenAI's ChatGPT and Google's Gemini. DesignObservational study of language usage in generative artificial intelligence models. SettingInvestigation-only browser profile restricted to ChatGPT and Gemini. ParticipantsEach chatbot generated 60 descriptions of people prompted without specified functional status, 30 descriptions of people with a disability, 30 descriptions of patients with a disability, and 30 descriptions of athletes with a disability (N=300). InterventionsNot applicable. Main Outcome MeasuresGenerated descriptions produced by the models were parsed into words that were linguistically analyzed into favorable qualities or limiting qualities. ResultsBoth large language models significantly underestimated disability in a population of people, and linguistic analysis showed that descriptions of people, patients, and athletes with a disability were generated as having significantly fewer favorable qualities and significantly more limitations than people without a disability in both ChatGPT and Gemini. ConclusionsGenerative artificial intelligence chatbots demonstrate quantifiable ability bias and often exclude people with disabilities in their responses. Ethical use of these generative large language model chatbots in medical systems should recognize this limitation, and further consideration should be taken in developing equitable artificial intelligence technologies.

Email subjects generation with large language models: GPT-3.5, PaLM 2, and BERT

In order to enhance marketing efforts and improve the performance of marketing campaigns, the effectiveness of language generation models needs to be evaluated. This study examines the performance of large language models (LLMs), namely GPT-3.5, PaLM 2, and bidirectional encoder representations from transformers (BERT), in generating email subjects for advertising campaigns. By comparing their results, the authors evaluate the efficacy of these models in enhancing marketing efforts. The objective is to explore how LLMs contribute to creating compelling email subject lines and improving opening rates and campaign performance, which gives us an insight into the impact of these models in digital marketing. In this paper, the authors first go over the different types of language models and the differences between them, before giving an overview of the most popular ones that will be used in the study, such as GPT-3.5, PaLM 2, and BERT. This study assesses the relevance, engagement, and uniqueness of GPT-3.5, PaLM 2, and BERT by training and fine-tuning them on marketing texts. The findings provide insights into the major positive impact of artificial intelligence (AI) on digital marketing, enabling informed decision-making for AI-driven email marketing strategies.

Automatic Generation Algorithm of Movie and TV Scripts Based on ChatGPT

A movie or TV script is a meticulously crafted document that serves as the blueprint for a visual storytelling project. It outlines the dialogue, actions, and scene descriptions that will unfold on screen, guiding actors, directors, and crew members in bringing the story to life. Each script is divided into scenes and acts, with clear instructions on character entrances and exits, camera angles, and pacing. This paper introduces a novel framework for enhancing the quality of movie and TV scripts through the integration of the Combinational Multi-Stage Genetic Optimization (CMSGO) model with ChatGPT, a state-of-the-art language generation model. The CMSGO model utilizes iterative optimization techniques to systematically refine and enhance script elements such as coherence, dialogue flow, character development, and overall narrative structure. The proposed CMSGO model comprises the Combinational model with the genetic optimization function. The function CMSGO model examines the fitness function with the Multi-Stage Optimization process. The proposed CMSGO model uses the estimation of features in the Multi-stage optimization model with the computation of features related to the scripts. Through 20 generations of optimization, the CMSGO model demonstrates its effectiveness in improving script quality, as evidenced by a steady increase in average script quality scores. Additionally, the multi-stage optimization approach targets specific aspects of script quality, allowing for targeted adjustments to parameters related to character motivations, plot coherence, and tone. Viewer opinions further validate the efficacy of the generated scripts, with positive evaluations across various aspects such as audience engagement, coherence, emotional impact, and originality. The proposed framework offers a robust and data-driven approach to scriptwriting, enabling the creation of high-quality movie and TV scripts that captivate and resonate with audiences, thus enriching the overall viewing experience.

Improving biomedical entity linking for complex entity mentions with LLM-based text simplification.

Large amounts of important medical information are captured in free-text documents in biomedical research and within healthcare systems, which can be made accessible through natural language processing (NLP). A key component in most biomedical NLP pipelines is entity linking, i.e. grounding textual mentions of named entities to a reference of medical concepts, usually derived from a terminology system, such as the Systematized Nomenclature of Medicine Clinical Terms. However, complex entity mentions, spanning multiple tokens, are notoriously hard to normalize due to the difficulty of finding appropriate candidate concepts. In this work, we propose an approach to preprocess such mentions for candidate generation, building upon recent advances in text simplification with generative large language models. We evaluate the feasibility of our method in the context of the entity linking track of the BioCreative VIII SympTEMIST shared task. We find that instructing the latest Generative Pre-trained Transformer model with a few-shot prompt for text simplification results in mention spans that are easier to normalize. Thus, we can improve recall during candidate generation by 2.9percentage points compared to our baseline system, which achieved the best score in the original shared task evaluation. Furthermore, we show that this improvement in recall can be fully translated into top-1 accuracy through careful initialization of a subsequent reranking model. Our best system achieves an accuracy of 63.6% on the SympTEMIST test set. The proposed approach has been integrated into the open-source xMEN toolkit, which is available online via https://github.com/hpi-dhc/xmen.

BartSmiles: Generative Masked Language Models for Molecular Representations.

We discover a robust self-supervised strategy tailored toward molecular representations for generative masked language models through a series of tailored, in-depth ablations. Using this pretraining strategy, we train BARTSmiles, a BART-like model with an order of magnitude more compute than previous self-supervised molecular representations. In-depth evaluations show that BARTSmiles consistently outperforms other self-supervised representations across classification, regression, and generation tasks, setting a new state-of-the-art on eight tasks. We then show that when applied to the molecular domain, the BART objective learns representations that implicitly encode our downstream tasks of interest. For example, by selecting seven neurons from a frozen BARTSmiles, we can obtain a model having performance within two percentage points of the full fine-tuned model on task Clintox. Lastly, we show that standard attribution interpretability methods, when applied to BARTSmiles, highlight certain substructures that chemists use to explain specific properties of molecules. The code and pretrained model are publicly available.

Programmable Protein Stabilization with Language Model-Derived Peptide Guides.

Dysregulated protein degradation via the ubiquitin-proteasomal pathway can induce numerous disease phenotypes, including cancer, neurodegeneration, and diabetes. Stabilizing improperly ubiquitinated proteins via target-specific deubiquitination is thus a critical therapeutic goal. Building off the major advances in targeted protein degradation (TPD) using bifunctional small-molecule degraders, targeted protein stabilization (TPS) modalities have been described recently. However, these rely on a limited set of chemical linkers and warheads, which are difficult to generate de novo for new targets and do not exist for classically "undruggable" targets. To address the limited reach of small molecule-based degraders, we previously engineered ubiquibodies (uAbs) by fusing computationally-designed "guide" peptides to E3 ubiquitin ligase domains for modular, CRISPR-analogous TPD. Here, we expand the TPS target space by engineering "deubiquibodies" (duAbs) via fusion of computationally-designed guides to the catalytic domain of the potent OTUB1 deubiquitinase. In human cells, duAbs effectively stabilize exogenous and endogenous proteins in a DUB-dependent manner. To demonstrate duAb modularity, we swap in new target-binding peptides designed via our generative language models to stabilize diverse target proteins, including key tumor suppressor proteins such as p53 and WEE1, as well as heavily-disordered fusion oncoproteins, such as PAX3::FOXO1. In total, our duAb system represents a simple, programmable, CRISPR-analogous strategy for TPS.

On Representation Learning-based Methods for Effective, Efficient, and Scalable Code Retrieval

Code retrieval consists of finding relevant code snippets regarding a programmer’s query — an increasingly important task due to software ubiquity. Although significant progress has been made, there is still a need for refinement and improvement in current code retrieval solutions. Besides effective, we argue that solutions for the code retrieval task should be efficient and scalable to ever-increasing large code repositories. This paper introduces xCoFormer, the first representation learning-based model for the code retrieval task in the literature, which meets all these requirements. Our contributions include: (i) an interactive tag-training method that efficiently places a query closer to its relevant codes in the embedding space and (ii) use of a specialized loss function (N-pair) better suited to the code retrieval task. To evaluate our proposal, we conducted experiments with different versions of xCoFormer against several state-of-the-art deep learning models in datasets with different properties. In our experiments, xCoFormer produced the best cost-effectiveness tradeoff — it is as effective as our closest competitor while being thousands of times faster at search time. xCoFormer’s attention mechanism also (partially) meets the desirable requirement of explainability by exposing the “reasoning” of the model’s predictions. Finally, when used as a “proxy” method, fine-tuned with domain-specific code language models, such as Unixcoder and CodeBERT, our proposal achieves gains of more than 33% in effectiveness when compared to the use of general purpose language models such as BERT and RoBERTa.

Unsupervised evolution of protein and antibody complexes with a structure-informed language model.

Large language models trained on sequence information alone can learn high-level principles of protein design. However, beyond sequence, the three-dimensional structures of proteins determine their specific function, activity, and evolvability. Here, we show that a general protein language model augmented with protein structure backbone coordinates can guide evolution for diverse proteins without the need to model individual functional tasks. We also demonstrate that ESM-IF1, which was only trained on single-chain structures, can be extended to engineer protein complexes. Using this approach, we screened about 30 variants of two therapeutic clinical antibodies used to treat severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. We achieved up to 25-fold improvement in neutralization and 37-fold improvement in affinity against antibody-escaped viral variants of concern BQ.1.1 and XBB.1.5, respectively. These findings highlight the advantage of integrating structural information to identify efficient protein evolution trajectories without requiring any task-specific training data.

Reform and Development of English Teaching in Colleges and Universities with the Integration of ChatGPT Technology

As an artificial intelligence-powered language generation model based on natural language processing, ChatGPT, like other artificial intelligence technologies, possesses super real-time dialogue capabilities and is considered an innovative and intelligent auxiliary teaching tool in the field of language learning. This article first analyzes the current state of development of artificial intelligence technologies such as ChatGPT. Secondly, it examines the challenges encountered in the reform and development of English teaching in colleges and universities in China. Lastly, it summarizes the applications and recommendations for using ChatGPT technology in college-level English teaching.

From text to treatment: the crucial role of validation for generative large language models in health care

From text to treatment: the crucial role of validation for generative large language models in health care

Assessment of Generative AI Large Language Model Knowledge in Children's Nutrition Education

Assessment of Generative AI Large Language Model Knowledge in Children's Nutrition Education

Human Versus Machine Intelligence: Assessing Natural Language Generation Models Through Complex Systems Theory.

The introduction of Transformer architectures - with the self-attention mechanism - in automatic Natural Language Generation (NLG) is a breakthrough in solving general task-oriented problems, such as the simple production of long text excerpts that resemble ones written by humans. While the performance of GPT-X architectures is there for all to see, many efforts are underway to penetrate the secrets of these black-boxes in terms of intelligent information processing whose output statistical distributions resemble that of natural language. In this work, through the complexity science framework, a comparative study of the stochastic processes underlying the texts produced by the English version of GPT-2 with respect to texts produced by human beings, notably novels in English and programming codes, is offered. The investigation, of a methodological nature, consists first of all of an analysis phase in which the Multifractal Detrended Fluctuation Analysis and the Recurrence Quantification Analysis - together with Zipf's law and approximate entropy - are adopted to characterize long-term correlations, regularities and recurrences in human and machine-produced texts. Results show several peculiarities and trends in terms of long-range correlations and recurrences in the last case. The synthesis phase, on the other hand, uses the complexity measures to build synthetic text descriptors - hence a suitable text embedding - which serve to constitute the features for feeding a machine learning system designed to operate feature selection through an evolutionary technique. Using multivariate analysis, it is then shown the grouping tendency of the three analyzed text types, allowing to place GTP-2 texts in between natural language texts and computer codes. Similarly, the classification task demonstrates that, given the high accuracy obtained in the automatic discrimination of text classes, the proposed set of complexity measures is highly informative. These interesting results allow us to add another piece to the theoretical understanding of the surprising results obtained by NLG systems based on deep learning and let us to improve the design of new informetrics or text mining systems for text classification, fake news detection, or even plagiarism detection.

Research Project Topic Recommender System Using Generative Language Model

Education has become a driver of a person's continuous innovation to improve their quality. Currently, the use of artificial intelligence determines progress in education. In this research, artificial intelligence technology was applied to develop a web-based recommendation system to help students at the Faculty of Computer Science, Klabat University, choose appropriate research topics for their final assignments. To provide personalized and contextually relevant suggestions, the recommendation system leverages deep learning and generative language models, specifically GPT-3. The Rapid Application Development process model is employed to develop the system. Its key components include semantic search, rapid engineering, and an advanced vector database for effective data management and retrieval. The functions provided by the system include user account registration, login, input of major subject grades and research preferences, and personalized recommendation results. Some additional features such as profile management, previous recommendation history, and password reset options are also provided. All these functions have been tested using the black box method.

Self-supervised BGP-graph reasoning enhanced complex KBQA via SPARQL generation

Knowledge base question answering aims to answer complex questions from large-scale knowledge bases. Although existing generative language models that translate questions into SPARQL queries have achieved promising results, there are still generation errors due to redundancies or errors in the knowledge fed to the generative models and difficulties in representing the implicit logic of knowledge as the specific syntax of SPARQL. To address above issues, we propose TrackerQA, a novel self-supervised reasoning framework based on basic graph patterns (BGP) to determine precise paths and enhance SPARQL generation. First, we develop a contrastive learning semantic matching model to reduce the large knowledge searching space. Then, we built a BGP parser that parses the recalled knowledge and constraints into BGP graphs, which can deconstruct complex knowledge into BGP triples and naturally obtain supervision from gold SPARQL. Next, we design a self-supervised BGP graph neural network that encodes knowledge through graph transformation layers with directed message-passing control and employs a question-aware attention mechanism to predict the exact BGP paths. Finally, a SPARQL generator integrates the paths into a pre-trained language model to improve the performance of SPARQL generation. Experiments on the KQA Pro dataset show that our model achieves state-of-the-art answering accuracy scores of 95.32%, being the closest to the human level at 97.5%, and reasons out KB paths with F1 scores of 0.98 for nodes and 0.99 for edges.

Virtual Assistant by using Large Language Model

Abstract— This paper presents the development of a virtual assistant named “Pragyan,” utilizing a Large Language Model (LLM) for natural language understanding and response generation. The virtual assistant is designed to interact with users through text and speech inputs, providing informative and conversational responses. Pragyan is built with a frontend user interface, a backend server, and a database for efficient communication and data management. The performance evaluation and result analysis demonstrate the effectiveness and user-friendliness of the virtual assistant Keywords: Virtual Assistant, Large Language Model, Natural Language Processing, Frontend, Backend, Database, Performance Evaluation, Result Analysis.

Assessing citation integrity in biomedical publications: corpus annotation and NLP models.

Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model-which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification-yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy. We make the corpus and the best-performing NLP model publicly available at https://github.com/ScienceNLP-Lab/Citation-Integrity/.

Evaluating large language models’ ability to generate interpretive arguments

In natural language understanding, a crucial goal is correctly interpreting open-textured phrases. In practice, disagreements over the meanings of open-textured phrases are often resolved through the generation and evaluation of interpretive arguments, arguments designed to support or attack a specific interpretation of an expression within a document. In this paper, we discuss some of our work towards the goal of automatically generating and evaluating interpretive arguments. We have curated a set of rules from the code of ethics of various professional organizations and a set of associated scenarios that are ambiguous with respect to some open-textured phrase within the rule. We collected and evaluated arguments from both human annotators and state-of-the-art generative language models in order to determine the relative quality and persuasiveness of both sets of arguments. Finally, we performed a Turing test-inspired study in order to assess whether human annotators can tell the difference between human arguments and machine-generated arguments. The results show that machine-generated arguments, when prompted a certain way, can be consistently rated as more convincing than human-generated arguments, and to the untrained eye, the machine-generated arguments can convincingly sound human-like.

Utilizing large language models for EFL essay grading: An examination of reliability and validity in rubric‐based assessments

AbstractThis study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student essays of varying quality. The grading scale comprised five domains: grammar, content, organization, style & expression and mechanics. The results revealed that fine‐tuned ChatGPT model demonstrated a very high level of reliability with an intraclass correlation (ICC) score of 0.972, Default ChatGPT model exhibited an ICC score of 0.947 and Bard showed a substantial level of reliability with an ICC score of 0.919. Additionally, a significant overlap was observed in certain domains when comparing the grades assigned by LLMs and human raters. In conclusion, the findings suggest that while LLMs demonstrated a notable consistency and potential for grading competency, further fine‐tuning and adjustment are needed for a more nuanced understanding of non‐objective essay criteria. The study not only offers insights into the potential use of LLMs in grading student essays but also highlights the need for continued development and research. Practitioner notesWhat is already known about this topic Large language models (LLMs), such as OpenAI's ChatGPT and Google's Bard, are known for their ability to generate text that mimics human‐like conversation and writing. LLMs can perform various tasks, including essay grading. Intraclass correlation (ICC) is a statistical measure used to assess the reliability of ratings given by different raters (in this case, EFL instructors and LLMs). What this paper adds The study makes a unique contribution by directly comparing the grading performance of expert EFL instructors with two LLMs—ChatGPT and Bard—using an analytical grading scale. It provides robust empirical evidence showing high reliability of LLMs in grading essays, supported by high ICC scores. It specifically highlights that the overall efficacy of LLMs extends to certain domains of essay grading. Implications for practice and/or policyThe findings open up potential new avenues for utilizing LLMs in academic settings, particularly for grading student essays, thereby possibly alleviating workload of educators.The paper's insistence on the need for further fine‐tuning of LLMs underlines the continual interplay between technological advancement and its practical applications.The results lay down a footprint for future research in advancing the use of AI in essay grading.

The effects of supplemental instruction derived from peer leaders on student outcomes in undergraduate human anatomy.

Supplemental instruction (SI) confers student success, as represented by grades, knowledge retention, and student engagement. However, studies often report professional, not undergraduate, program findings. To measure these effects, students studying human anatomy at a university in Ontario, Canada, attended structured (peer-assisted) or unstructured (nonpeer-assisted) SI sessions and completed a pre-/post-survey. Fifty-eight learners (39 systems (SYS) and 19 musculoskeletal (MSK) anatomy) completed both surveys and had responses analyzed. Both cohorts, maintained initial perceptions across pre-/post-analyses (MSK p = 0.1376 and SYS p = 0.3521). Resource usage was similar across both cohorts with discrepancies in skeletal model and textbook use. No MSK learner ranked any labresourcesas "not at all useful." MSK learners felt more prepared to write a graded assessment (p = 0.0269), whereas SYS learners did not (p = 0.0680). Stratification of learners in MSK and SYS revealed learners spending between 30 and 60 min in SI sessions during the study period had the highest grades compared to students who spent less than 30 (p = 0.0286) or more than 60 (p = 0.0286) min attending SI sessions, respectively. Most learners in MSK (89.4%) and SYS (66%) concluded that they preferred structured over unstructured SI. Sentiment/thematic analysis using a generative AI-driven large language model revealed learners held positive perceptions of SI, emphasizing structured learning, resources, personalized learning, and support offered as the most prevalent themes surrounding SI. Ultimately, this study provides evidence that supports SI for improving student outcomes related to perceived preparedness for completing assessments and preferred teaching/learning styles in undergraduate human anatomy.

The Pitfalls of Defining Hallucination

Abstract Despite impressive advances in Natural Language Generation (NLG) and Large Language Models (LLMs), researchers are still unclear about important aspects of NLG evaluation. To substantiate this claim, I examine current classifications of hallucination and omission in data-text NLG, and I propose a logic-based synthesis of these classfications. I conclude by highlighting some remaining limitations of all current thinking about hallucination and by discussing implications for LLMs.