Comparative analysis of paraphrasing performance of ChatGPT, GPT‐3, and T5 language models using a new ChatGPT generated dataset: ParaGPT

Abstract

AbstractParaphrase generation is a fundamental natural language processing (NLP) task that refers to the process of generating a well‐formed and coherent output sentence that exhibits both syntactic and/or lexical diversity from the input sentence, while simultaneously ensuring that the semantic similarity between the two sentences is preserved. However, the availability of high‐quality paraphrase datasets has been limited, particularly for machine‐generated sentences. In this paper, we present ParaGPT, a new paraphrase dataset of 81,000 machine‐generated sentence pairs, including 27,000 reference sentences (ChatGPT‐generated sentences), and 81,000 paraphrases obtained by using three different large language models (LLMs): ChatGPT, GPT‐3, and T5. We used ChatGPT to generate 27,000 sentences that cover a diverse array of topics and sentence structures, thus providing diverse inputs for the models. In addition, we evaluated the quality of the generated paraphrases using various automatic evaluation metrics. Furthermore, we provide insights into the strengths and drawbacks of each LLM in generating paraphrases by conducting a comparative analysis of the paraphrasing performance of the three LLMs. According to our findings, ChatGPT's performance, as per the evaluation metrics provided, was deemed impressive and commendable, owing to its higher‐than‐average scores for semantic similarity, which implies a higher degree of similarity between the generated paraphrase and the reference sentence, and its relatively lower scores for syntactic diversity, indicating a greater diversity of syntactic structures in the generated paraphrase. ParaGPT is a valuable resource for researchers working on NLP tasks like paraphrasing, text simplification, and text generation. We make the ParaGPT dataset publicly accessible to researchers, and as far as we are aware, this is the first paraphrase dataset produced based on ChatGPT.