The performance of improved XLNet on text classification

Abstract

Text classification is the most common application of Natural Language Processing(NLP), and Transformer models have dominated the field in recent years. Currently, pre-training modeling of text through deep learning methods is a common way of text classification. This paper firstly proposes an improved XLNet text classification model based on the problems of long-term dependence and insufficient contextual semantic expression in previous pre-trained language models, and uses XLNet pre-trained language modeling to represent text as low-dimensional word vectors to obtain sequences. Secondly, the generated word vector sequence is passed into the LSTM network, and the two-way features of the sentence are extracted by the memory unit of the LSTM. On the basis of effectively extracting text features, the Multi-head attention model is used to calculate the multi-angle attention probability, to focus on the important words. Finally, text classification is achieved through a fully connected network. The experimental results show that the accuracy of the model is 0.9457 and the loss rate is 0.3133. Compared with other related models, it has achieved better results.