Human-written Text Research Articles

Evaluating the Efficacy of Perplexity Scores in Distinguishing AI-Generated and Human-Written Abstracts.

We aimed to evaluate the efficacy of perplexity scores in distinguishing between human-written and AI-generated radiology abstracts and to assess the relative performance of available AI detection tools in detecting AI-generated content. Academic articles were curated from PubMed using the keywords "neuroimaging" and "angiography." Filters included English-language, open-access articles with abstracts without subheadings, published before 2021, and within Chatbot processing word limits. The first 50 qualifying articles were selected, and their full texts were used to create AI-generated abstracts. Perplexity scores, which estimate sentence predictability, were calculated for both AI-generated and human-written abstracts. The performance of three AI tools in discriminating human-written from AI-generated abstracts was assessed. The selected 50 articles consist of 22 review articles (44%), 12 case or technical reports (24%), 15 research articles (30%), and one editorial (2%). The perplexity scores for human-written abstracts (median; 35.9 IQR; 25.11-51.8) were higher than those for AI-generated abstracts (median; 21.2 IQR; 16.87-28.38), (p=0.057) with an AUC=0.7794. One AI tool performed less than chance in identifying human-written from AI-generated abstracts with an accuracy of 36% (p>0.05) while another tool yielded an accuracy of 95% with an AUC=0.8688. This study underscores the potential of perplexity scores in detecting AI-generated and potentially fraudulent abstracts. However, more research is needed to further explore these findings and their implications for the use of AI in academic writing. Future studies could also investigate other metrics or methods for distinguishing between human-written and AI-generated texts.

Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges

Accurate attribution of authorship is crucial for maintaining the integrity of digital content, improving forensic investigations, and mitigating the risks of misinformation and plagiarism. Addressing the imperative need for proper authorship attribution is essential to uphold the credibility and accountability of authentic authorship. The rapid advancements of Large Language Models (LLMs) have blurred the lines between human and machine authorship, posing significant challenges for traditional methods. We present a comprehensive literature review that examines the latest research on authorship attribution in the era of LLMs. This survey systematically explores the landscape of this field by categorizing four representative problems: (1) Human-written Text Attribution; (2) LLM-generated Text Detection; (3) LLM-generated Text Attribution; and (4) Human-LLM Co-authored Text Attribution. We also discuss the challenges related to ensuring the generalization and explainability of authorship attribution methods. Generalization requires the ability to generalize across various domains, while explainability emphasizes providing transparent and understandable insights into the decisions made by these models. By evaluating the strengths and limitations of existing methods and benchmarks, we identify key open problems and future research directions in this field. This literature review serves a roadmap for researchers and practitioners interested in understanding the state of the art in this rapidly evolving field. Additional resources and a curated list of papers are available and regularly updated at https://llm-authorship.github.io/.

Artificial Intelligence vs. Human: Decoding Text Authenticity with Transformers

This paper presents a comprehensive study on detecting AI-generated text using transformer models. Our research extends the existing RODICA dataset to create the Enhanced RODICA for Human-Authored and AI-Generated Text (ERH) dataset. We enriched RODICA by incorporating machine-generated texts from various large language models (LLMs), ensuring a diverse and representative corpus. Methodologically, we fine-tuned several transformer architectures, including BERT, RoBERTa, and DistilBERT, on this dataset to distinguish between human-written and AI-generated text. Our experiments examined both monolingual and multilingual settings, evaluating the model’s performance across diverse datasets such as M4, AICrowd, Indonesian Hoax News Detection, TURNBACKHOAX, and ERH. The results demonstrate that RoBERTa-large achieved superior accuracy and F-scores of around 83%, particularly in monolingual contexts, while DistilBERT-multilingual-cased excelled in multilingual scenarios, achieving accuracy and F-scores of around 72%. This study contributes a refined dataset and provides insights into model performance, highlighting the transformative potential of transformer models in detecting AI-generated content.

How Text Transformations Affect AI Detection

This study addresses the critical issue of AI writing detection, which currently plays a key role in deterring technology misuse and proposes a foundation for the controllable and conscious use of AI. The ability to differentiate between human-written and AI-generated text is crucial for the practical application of any policies or guidelines. Current detection tools are unable to interpret their decisions in a way that is understandable to humans or provide any human-readable evidence or proof for their decisions. We assume that there should be a traceable footprint in LLM-generated texts that is invisible to the human eye but can be detected by AI detection tools-referred to as the AI footprint. Understanding its nature will help bring more light into the guiding principles lying at the core of AI detection technology and help build more trust in the technology in general. The main goal of this paper is to examine the AI footprint in text data generated by large language models (LLMs). To achieve this, we propose a new method for text transformation that should measurably decrease the AI footprint in the text data, impacting AI writing scores. We applied a set of stage-by-stage text transformations focused on decreasing meaningfulness by masking or removing words. Using a set of AI detectors, we measured the AI writing score as a proxy metric for assessing the impact of the proposed method. The results demonstrate a significant correlation between the severity of changes and the resulting impact on AI writing scores, highlighting the need for developing more reliable AI writing identification methods that are immune to attempts to hide the AI footprint through subtle changes

A perspective on gender bias in generated text data

Text generation by artificial intelligence became available to a broader public, latterly. This technology is based on machine learning and language models that need to be trained with input data. Many studies have focused on the distinction of human-written text. vs. generated texts but recent studies show that the underlying language models might be prone to reproduce gender bias in their output and, consequently, reinforcing gender roles and imbalances. In this paper, we give a perspective on this topic, considering both the generated text data itself and the machine learning models used for language generation. We present a case study of gender bias in generated text data and review recent literature addressing language models. Our results indicate that researching gender bias in the context of text generation faces significant challenges and that future work needs to overcome a lack of definitions as well as a lack of transparency.

Spot the bot: the inverse problems of NLP.

This article concerns the problem of distinguishing human-written and bot-generated texts. In contrast to the classical problem formulation, in which the focus falls on one type of bot only, we consider the problem of distinguishing texts written by any person from those generated by any bot; this involves analysing the large-scale, coarse-grained structure of the language semantic space. To construct the training and test datasets, we propose to separate not the texts of bots, but bots themselves, so the test sample contains the texts of those bots (and people) that were not in the training sample. We aim to find efficient and versatile features, rather than a complex classification model architecture that only deals with a particular type of bots. In the study we derive features for human-written and bot generated texts, using clustering (Wishart and K-Means, as well as fuzzy variations) and nonlinear dynamic techniques (entropy-complexity measures). We then deliberately use the simplest of classifiers (support vector machine, decision tree, random forest) and the derived characteristics to identify whether the text is human-written or not. The large-scale simulation shows good classification results (a classification quality of over 96%), although varying for languages of different language families.

Assessing AI and Human Writing Skills: A Comparative Analysis of Pre-Service Teachers’ Perceptions

Artificial intelligence (AI), specifically conversational AI, is increasingly utilized to enhance writing skills, necessitating future teachers’ awareness of its capabilities for implementation. The present mixed-methods study explores the ability of AI to write a summary and an argumentative text and pre-service teachers’ perceptions of it. A total of 154 pre-service teachers evaluated two texts generated by ChatGPT and two written by another student previously anonymized. The results show that they rated the AI-generated texts significantly higher overall and regarding linguistic parameters like text structure. Additionally, they generally consider that the human-written text conveys more emotions, while the AI text would deserve a better grade in an exam. However, many participants struggled to correctly identify the author of each text, which highlights the need to prepare them for the possibilities of current AI to take advantage of it.

Automated classification of online reviews of otolaryngologists.

The study aimed to extract online comments of otolaryngologists in the 20 most populated cities in the United States from healthgrades.com, develop and validate a natural language processing (NLP) logistic regression algorithm for automated text classification of reviews into 10 categories, and compare 1- and 5-star reviews in directly-physician-related and non-physician-related categories. 1977 1-star and 12,682 5-star reviews were collected. The primary investigator manually categorized a training dataset of 324 1-star and 909 5-star reviews, while a validation subset of 100 5-star and 50 1-star reviews underwent dual manual categorization. Using scikit-learn, an NLP algorithm was trained and validated on the subsets, with F1 scores evaluating text classification accuracy against manual categorization. The algorithm was then applied to the entire dataset with comparison of review categorization according to 1- and 5-star reviews. F1 scores for NLP validation ranged between 0.71 and 0.97. Significant associations emerged between 1-star reviews and treatment plan, accessibility, wait time, office scheduling, billing, and facilities. 5-star reviews were associated with surgery/procedure, bedside manner, and staff/mid-levels. The study successfully validated an NLP text classification system for categorizing online physician reviews. Positive reviews were found to have an association with directly-physician related context. 1-star reviews were related to treatment plan, accessibility, wait time, office scheduling, billing, and facilities. This method of text classification effectively discerned the nuances of human-written text, providing valuable insights into online healthcare feedback that is scalable.Level of evidence: Level 3.

Testing the capacity of Bard and ChatGPT for writing essays on ethical dilemmas: A cross-sectional study

Recent advancements in artificial intelligence (AI) have led to concerns about its potential misuse in education. As large language models (LLMs) such as ChatGPT and Bard can generate human-like text, researchers and educators noted the potential redundancy of tasking students with writing academic essays. We aimed to explore if the two LLMs could generate unstructured essays on medical students’ personal experiences of challenges and ethical dilemmas that are indistinguishable from human-written texts. We collected 47 original student-written essays from which we extracted keywords to develop prompts for the LLMs. We then used these prompts to generate an equivalent number of essays using ChatGPT and Bard. We analysed the essays using the Language Inquiry and Word Count (LIWC) 22 software, extracting the main LIWC summary measures and variables related to social and psychological processes. We also conducted sub-analyses for sixteen student essays that were presumably written entirely or in part by AI, according to two AI detectors. We found that AI-written essays used more language related to affect, authenticity, and analytical thinking compared to student-written essays after we removed AI-co-written student essays from the analysis. We observed that, despite the differences in language characteristics compared to student-written essays, both LLMs are highly effective in generating essays on students’ personal experiences and opinions regarding real-life ethical dilemmas.

Almost Nobody Is Using ChatGPT to Write Academic Science Papers (Yet)

We assessed 19,000 scientific introductions to measure the level of undisclosed use of ChatGPT in scientific papers published in 2023 and early 2024. We applied a “stylistics” approach that has previously been shown to be effective at differentiating AI-generated text from human-written text in a variety of venues. Ten different MDPI journals were selected for this study, and the rate of use of undisclosed AI writing in these journals was fairly consistent across the journals. We estimate that ChatGPT was used for writing or significant editing in about 1 to 3% of the introductions tested. This analysis is the first systematic study of detecting undisclosed ChatGPT in published manuscripts in cases where obvious indicators, such as phrases like “regenerate response”, are not present. The work demonstrates that generative AI is not polluting mainstream journals to any appreciable extent and that the overwhelming majority of scientists remain hesitant to embrace this tool for late-stage writing and editing.

A comparative study of thematic choices and thematic progression patterns in human-written and AI-generated texts

By adopting systemic functional grammar as the theoretical framework, the present research investigates the choices of the textual, the interpersonal, the topical Themes, and thematic progression patterns in human-written and AI-generated argumentative texts to examine how the human author and the robot organize the clause and text. The findings suggest that human-written and AI-generated texts differ significantly in using the subtypes of textual, interpersonal, and marked topical Themes. In terms of textual Themes, the machine tends to use more concession signals to repeat information that has been stated earlier, adopt fewer condition signals since the robot may be less likely to imagine possible or impossible situations, and repetitively rely on clause-initial conjunctive adjuncts of addition to extend the clause to the previous text. As for interpersonal Themes, the robot seldom adopts modal adjuncts or modal verbal operators as interpersonal Themes, which may suggest its lack of awareness of interaction with the reader and its avoidance of expressing its viewpoints in a typical or congruent way. Regarding topical Themes, the less use of marked themes in AI-generated texts demonstrates that AI may less carefully plan the development of the text to foreground the setting or construct coherent text. Considering thematic progression patterns, the frequent use of the constant pattern in AI-generated texts prevents the text from development and makes the text redundant and simplistic like a list of ideas.

Evaluating Generative AI's Ability to Identify Cancer Subtypes in Publicly Available Structured Genetic Datasets.

Genetic data play a crucial role in diagnosing and treating various diseases, reflecting a growing imperative to integrate these data into clinical care. However, significant barriers such as the structure of electronic health records (EHRs), insurance costs for genetic testing, and the interpretability of genetic results impede this integration. This paper explores solutions to these challenges by combining recent technological advances with informatics and data science, focusing on the diagnostic potential of artificial intelligence (AI) in cancer research. AI has historically been applied in medical research with limited success, but recent developments have led to the emergence of large language models (LLMs). These transformer-based generative AI models, trained on vast datasets, offer significant potential for genetic and genomic analyses. However, their effectiveness is constrained by their training on predominantly human-written text rather than comprehensive, structured genetic datasets. This study reevaluates the capabilities of LLMs, specifically GPT models, in performing supervised prediction tasks using structured gene expression data. By comparing GPT models with traditional machine learning approaches, we assess their effectiveness in predicting cancer subtypes, demonstrating the potential of AI models to analyze real-world genetic data for generating real-world evidence.

Effective and Imperceptible Adversarial Textual Attack Via Multi-objectivization

The field of adversarial textual attack has significantly grown over the past few years, where the commonly considered objective is to craft adversarial examples (AEs) that can successfully fool the target model. However, the imperceptibility of attacks, which is also essential for practical attackers, is often left out by previous studies. In consequence, the crafted AEs tend to have obvious structural and semantic differences from the original human-written text, making them easily perceptible. In this work, we advocate leveraging multi-objectivization to address such an issue. Specifically, we reformulate the problem of crafting AEs as a multi-objective optimization problem, where the attack imperceptibility is considered as an auxiliary objective. Then, we propose a simple yet effective evolutionary algorithm, dubbed HydraText, to solve this problem. HydraText can be effectively applied to both score-based and decision-based attack settings. Exhaustive experiments involving 44,237 instances demonstrate that HydraText consistently achieves competitive attack success rates and better attack imperceptibility than the recently proposed attack approaches. A human evaluation study also shows that the AEs crafted by HydraText are more indistinguishable from human-written text. Finally, these AEs exhibit good transferability and can bring notable robustness improvement to the target model by adversarial training.

Applying ChatGPT to writing scientific articles on the use of telemedicine: Opportunities and limitations

In the rapidly evolving world of technology, artificial intelligence (AI) has significantly integrated into various aspects of our lives, including health-care, education, finance, transportation, and entertainment. Notably, AI has also impacted the writing of textual works such as scientific papers, professional opinions, and educational texts. This study investigates the application of OpenAI’s ChatGPT language model in writing scientific articles on telemedicine, specifically in the areas of cardiology, oncology, and remote medical examination. The study uses ChatGPT versions 3.5 and 4 to create articles using three different prompts. The created articles were evaluated based on the reliability of the cited literature references, the impact factor (IF) of the journal in which the sources were published, and the relevance of the sources. The sources were divided into three categories: reliable, semi-reliable, and completely fictitious. The results demonstrate that ChatGPT can produce semantically coherent and error-free texts indistinguishable from human-written texts. However, the reliability of literary references varies significantly. ChatGPT 4, benefitting from its larger training dataset, generates a higher percentage of reliable sources compared to ChatGPT 3.5. The IF analysis indicates the prevalence of high-impact journals among reliable sources, which emphasizes the effectiveness of the model in selecting quality references. The study highlights the need for caution when using AI to write scientific articles due to the potential for biased, unverified, and inaccurate information. It is important to critically evaluate and vet AI-generated content. In addition, the study emphasizes that the correct use of AI and thoughtful drafting of prompts can improve the efficiency and quality of scientific papers. Future advancements in AI technology are expected to further minimize errors and biases.

Survey for Detecting AI-generated Content

In large language models (LLMs) field, the rapid advancements have significantly improved text generation, which has blured the distinction between AI-generated and human-written texts. These developments have sparked concerns about potential risks, such as disseminating fake information or engaging in academic cheating. As the responsible use of LLMs becomes imperative, the detection of AI-generated content has become a crucial task. Most existing surveys on AI-generated text (AIGT) Detection have analysed the detection approaches from a computational perspective, with less attention to linguistic aspects. This survey seeks to provide a fresh perspective to drive progress in the area of LLM-generated text detection. Futhermore, in order to make the assessment more explainable, we emphasize the great importence of leveraging specific parameters or metrics to linguistically evaluate the candidate text.

A publishing infrastructure for Artificial Intelligence (AI)-assisted academic authoring.

Investigate the use of advanced natural language processing models to streamline the time-consuming process of writing and revising scholarly manuscripts. For this purpose, we integrate large language models into the Manubot publishing ecosystem to suggest revisions for scholarly texts. Our AI-based revision workflow employs a prompt generator that incorporates manuscript metadata into templates, generating section-specific instructions for the language model. The model then generates revised versions of each paragraph for human authors to review. We evaluated this methodology through 5 case studies of existing manuscripts, including the revision of this manuscript. Our results indicate that these models, despite some limitations, can grasp complex academic concepts and enhance text quality. All changes to the manuscript are tracked using a version control system, ensuring transparency in distinguishing between human- and machine-generated text. Given the significant time researchers invest in crafting prose, incorporating large language models into the scholarly writing process can significantly improve the type of knowledge work performed by academics. Our approach also enables scholars to concentrate on critical aspects of their work, such as the novelty of their ideas, while automating tedious tasks like adhering to specific writing styles. Although the use of AI-assisted tools in scientific authoring is controversial, our approach, which focuses on revising human-written text and provides change-tracking transparency, can mitigate concerns regarding AI's role in scientific writing.

On the Unexpected Abilities of Large Language Models

Large Language Models (LLMs) are capable of displaying a wide range of abilities that are not directly connected with the task for which they are trained: predicting the next words of human-written texts. In this article, I review recent research investigating the cognitive abilities developed by LLMs and their relation to human cognition. I discuss the nature of the indirect process that leads to the acquisition of these cognitive abilities, their relation to other indirect processes, and the implications for the acquisition of integrated abilities. Moreover, I propose the factors that enable the development of abilities that are related only very indirectly to the proximal objective of the training task. Finally, I discuss whether the full set of capabilities that LLMs could possibly develop is predictable.

Exploring the potential of ChatGPT in facilitating consciousness-raising tasks: A promising solution for alleviating the burden on educators

When studying grammar, students must not only focus on its structure but also on its form. Form-focused activities are integral to this process, requiring students to identify and manipulate language forms. A well-established technique for facilitating language acquisition is the consciousness-raising task (CR), which aims to heighten learners' awareness of language form. By prioritizing the language input, students can cultivate a more precise comprehension of grammar structures, thereby enhancing their own language proficiency. This approach entails various strategies, such as inferring grammatical rules from examples, comparing different forms of expression, and examining differences between a learner's usage of a grammar item and that of native speakers. However, designing effective CR tasks is cognitively demanding, time-intensive, and laborious for teachers, as it entails accommodating numerous requirements, including the definition of clear objectives, task relevance, and task engagement. To address these challenges, it is worth utilizing ChatGPT which is an advanced conversational AI system with the capability to process and generate various modalities of language. It has access to extensive databases and can produce written content that is frequently indistinguishable from human-written text. Accordingly, ChatGPT can be utilized to generate CR tasks or materials that may be applied in class. The present study endeavors to offer a range of CR tasks for grammar teaching that have been generated by ChatGPT. The tasks include identification tasks, text correction tasks, word choice tasks, and others.

Identifying ChatGPT-Written Patient Education Materials Using Text Analysis and Readability.

Artificial intelligence (AI)-based text generators such as Chat Generative Pre-Trained Transformer (ChatGPT) have come into the forefront of modern medicine. Given the similarity between AI-generated and human-composed text, tools need to be developed to quickly differentiate the two. Previous work has shown that simple grammatical analysis can reliably differentiate AI-generated text from human-written text. In this study, ChatGPT was used to generate 25 articles related to obstetric topics similar to those made by the American College of Obstetrics and Gynecology (ACOG). All articles were geared towards patient education. These AI-generated articles were then analyzed for their readability and grammar using validated scoring systems and compared to real articles from ACOG. Characteristics of the 25 AI-generated articles included fewer overall characters than original articles (mean 3,066 vs. 7,426; p < 0.0001), a greater average word length (mean 5.3 vs. 4.8; p < 0.0001), and a lower Flesch-Kincaid score (mean 46 vs. 59; p < 0.0001). With this knowledge, a new scoring system was develop to score articles based on their Flesch-Kincaid readability score, number of total characters, and average word length. This novel scoring system was tested on 17 new AI-generated articles related to obstetrics and 7 articles from ACOG, and was able to differentiate between AI-generated articles and human-written articles with a sensitivity of 94.1% and specificity of 100% (Area Under the Curve [AUC] 0.99). As ChatGPT is more widely integrated into medicine, it will be important for health care stakeholders to have tools to separate originally written documents from those generated by AI. While more robust analyses may be required to determine the authenticity of articles written by complex AI technology in the future, simple grammatical analysis can accurately characterize current AI-generated texts with a high degree of sensitivity and specificity. · More tools are needed to identify AI-generated text in obstetrics, for both doctors and patients.. · Grammatical analysis is quick and easily done.. · Grammatical analysis is a feasible and accurate way to identify AI-generated text..

Towards an understanding and explanation for mixed-initiative artificial scientific text detection

Large language models (LLMs) have gained popularity in various fields for their exceptional capability of generating human-like text. Their potential misuse has raised social concerns about plagiarism in academic contexts. However, effective artificial scientific text detection is a non-trivial task due to several challenges, including (1) the lack of a clear understanding of the differences between machine-generated and human-written scientific text, (2) the poor generalization performance of existing methods caused by out-of-distribution issues, and (3) the limited support for human-machine collaboration with sufficient interpretability during the detection process. In this paper, we first identify the critical distinctions between machine-generated and human-written scientific text through a quantitative experiment. Then, we propose a mixed-initiative workflow that combines human experts’ prior knowledge with machine intelligence, along with a visual analytics system to facilitate efficient and trustworthy scientific text detection. Finally, we demonstrate the effectiveness of our approach through two case studies and a controlled user study. We also provide design implications for interactive artificial text detection tools in high-stakes decision-making scenarios.