Leveraging Pre-Trained Language Model for Summary Generation on Short Text

Shuai Zhao,Zeng Yuan Liu,Fucheng You

doi:10.1109/access.2020.3045748

Shuai Zhao, Zeng Yuan Liu + Show 1 more

Open Access

https://doi.org/10.1109/access.2020.3045748

Copy DOI

Abstract

Bidirectional Encoder Representations from Transformers represents the latest incarnation of pre-trained language models which have been obtained a satisfactory effect in text summarization tasks. However, it has not achieved good results for the generation of Chinese short text summaries. In this work, we propose a novel short text summary generation model based on keyword templates, which uses templates found in training data to extract keywords to guide summary generation. The experimental results of the LCSTS data set show that our model performs better than the baseline model. The analysis shows that the methods used in our model can generate high-quality summaries.

Highlights

In deep learning research, when the target task training data is less, usually pre-training and fine-tuning methods can achieve outstanding results [1]
We introduce a short text summary generation model based on keyword templates, which makes the BERT model better applied to Chinese short text summarization generation tasks
Contributions made by this article: 1. We introduce a short text summary generation model based on keyword templates and improve the data preprocessing method of Chinese short text in summary generation tasks

Summary

INTRODUCTION

In deep learning research, when the target task training data is less, usually pre-training and fine-tuning methods can achieve outstanding results [1]. Experiments show that the BERT pre-training model does not perform well to generate Chinese short text summaries To solve this problem, we propose a short text summary generation model based on keyword templates. We introduce a short text summary generation model based on keyword templates, which makes the BERT model better applied to Chinese short text summarization generation tasks. S. Zhao et al.: Leveraging Pre-Trained Language Model for Summary Generation on Short Text reference summary and divide the input text; in the testing phase, we use similarity calculations to find the most similar text in the training set and extract the keywords in the reference summary of the training text for sentence division. We introduce a short text summary generation model based on keyword templates and improve the data preprocessing method of Chinese short text in summary generation tasks. Our model can be used as a stepping stone to improve the quality of the summary and make the pre-trained language model better used in the generation of short text summaries

RELATED WORK

EXPERIMENTAL DETAILS

EXPERIMENT ANALYSIS

Findings

CONCLUSION