An objective cross-sectional assessment of ChatGPT in hematology-oncology manuscript composition: Balancing promise with factual inaccuracies

Abstract

ABSTRACT Background: Artificial intelligence (AI)-based large language models (LLMs), such as Chat Generative Pre-training Transformer (ChatGPT), exhibit promise in aiding manuscript composition and literature search, encompassing various research tasks. However, their utilization remains unregulated. Objectives: The primary objective of this study was to objectively assess the ability of ChatGPT 3.5 (free version) to assist with various tasks associated with manuscript preparation and research based on pre-defined scoring criteria. Secondary objectives included an assessment of the factual accuracy of data and any false information returned by ChatGPT. Materials and Methods: This cross-sectional study was planned in the Departments of Clinical Hematology and Medical Oncology of Dayanand Medical College and Hospital, Ludhiana, Punjab, India, a tertiary care referral center. Between July 1, 2023, and July 30, 2023, seven prompts comprising queries related to manuscript design, specific data, or complex discussion in hematology/oncology subjects were used. The responses were scored based on detailed pre-defined criteria for accuracy and completeness, and scoring was independently performed by a panel of five reviewers with current expertise in the field of hematology/medical oncology. Negative marking was performed for factual inaccuracies. Cronbach’s alpha and interclass correlation coefficient were calculated to assess inter-observer agreement. Results: ChatGPT readily provided information on the structural components of the manuscript and research design, with the ability to customize responses immediately. However, the presence of factual inaccuracies, fictional citations, and false information presented confidently were notable drawbacks. Cronbach’s alpha was 0.995, and the intraclass correlation coefficient was 0.995, indicating good inter-observer agreement. The overall score was 34.2 out of 90, with a poor score on the veracity of data and references. Conclusion: The current iteration of ChatGPT rapidly provides plausible and professional-looking information on up-to-date topics but is hindered by significant factual inaccuracies. Future research focusing on improving response accuracy and addressing ethical considerations of content generated by LLMs will help us maximize their potential in scientific paper development.