СПЕЦІАЛІЗОВАНЕ ПОПЕРЕДНЄ НАВЧАННЯ НЕЙРОМЕРЕЖЕВИХ МОДЕЛЕЙ НА СИНТЕТИЧНИХ ДАНИХ ДЛЯ ПОКРАЩЕННЯ ГЕНЕРАЦІЇ ПЕРЕФРАЗУВАННЯ

Abstract

SPECIALIZED PRE-TRAINING OF NEURAL NETWORKS ON SYNTHETIC DATA FOR IMPROVING PARAPHRASE GENERATION Abstract. Generating paraphrases is a fundamental problem in natural language processing. In light of the significant success of transfer learning technology, the “pre-training fine-tuning” approach has become the standard. However, popular general-purpose pre-training methods typically require large datasets and computational resources, and available pre-trained models are limited by fixed architecture and size. We propose a simple and effective approach for pre-training specifically for paraphrase generation, which significantly improves model quality and matches the quality level of general-purpose models. Both existing public data and new data generated by large language models were used. The impact of this procedure on neural networks of different architectures was investigated, and it was shown to work for all of them. Keywords: artificial intelligence, machine learning, neural networks, paraphrase generation, pre-training, fine-tuning.