Abstract Summarization Based on the Combination of Transformer and LSTM

Abstract

Summary is the task of compressing a piece of text into a short version with the main information of the original text. While previous architecture choices revolve around Convolutional Neural Networks (CNN) and long shortterm memory (LSTM) recurrent neural networks, recently self-attention and transformer have been used for the text generation task and achieved very good results. However transformer lacks local attention information and uses simple calculations as address vectors. In our article, we put forward a new seq-to-seq model for generating text summary called LTABS (Abstractive summarization on LSTM and Transformer). This model makes transformer be more suitable for summary generation tasks. For the encoder part of the model, we make use of the hidden layer result of LSTM as the location information. At the same time, transformer is used as the attention matrix of LSTM. This design allows the transformer to obtain location information and enhance local attention. We add copy mechanism to the new network to finish off the summary tasks OOV (out of vocabulary) problem. We apply our model to CNN and Daily Mail and Xsum datasets, the test outcome shows that the LTABS framework is superior to the most advanced model in semantic and syntactic structure, and has achieved competitive results in manual language quality assessment.