Quality and correctness of AI-generated versus human-written abstracts in psychiatric research papers

Abstract

This study aimed to assess the ability of an artificial intelligence (AI)-based chatbot to generate abstracts from academic psychiatric articles. We provided 30 full-text psychiatric papers to ChatPDF (based on ChatGPT) and prompted generating a similar style structured or unstructured abstract. We further used 10 papers from Psychiatry Research as active comparators (unstructured format). We compared the quality of the ChatPDF-generated abstracts with the original human-written abstracts and examined the similarity, plagiarism, detected AI-content, and correctness of the AI-generated abstracts. Five experts evaluated the quality of the abstracts using a blinded approach. They also identified the abstracts written by the original authors and validated the conclusions produced by ChatPDF. We found that the similarity and plagiarism were relatively low (only 14.07% and 8.34%, respectively). The detected AI-content was 31.48% for generated structure-abstracts, 75.58% for unstructured-abstracts, and 66.48% for active comparators abstracts. For quality, generated structured-abstracts were rated similarly to originals, but unstructured ones received significantly lower scores. Experts rated 40% accuracy with structured abstracts, 73% with unstructured ones, and 77% for active comparators. However, 30% of AI-generated abstract conclusions were incorrect. In conclusion, the data organization capabilities of AI language models hold significant potential for applications to summarize information in clinical psychiatry. However, the use of ChatPDF to summarize psychiatric papers requires caution concerning accuracy.