Category-learning attention mechanism for short text filtering

Abstract

In machine translation, the attention mechanism highlights relevant words according to the distances between the source and target vectors dynamically. However, its ability to optimize text classification is limited because this mechanism only calculates weights inside the same text. There is an uneven distribution of words between ham (not spam) and spam categories, and this category-level feature has not previously been utilized in the attention mechanism for filtering short texts. In addition, short text filtering is uniquely challenging due to the length, sparsity and informal writing of texts, as well as a need for rapid processing. We propose a novel category-level attention mechanism called “category-learning attention,” which highlights words intensely distributed in the same category by dynamically calculating a category differentiation matrix for each short text. The category-learning attention mechanism is extended to the category-learning scaled-dot-product attention and the category-learning multi-head attention (CL-MHA) mechanisms. The CL-MHA mechanism is then applied to a bidirectional gate recurrent unit (Bi-GRU) model for performance evaluation by using the SMS spam collection dataset hosted at the University of California, Irvine. Performance metrics including the accuracy, precision, recall, and F1 score demonstrate that the CL-MHA mechanism significantly improves the performance of Bi-GRU for short text filtering with an accuracy of 99.35%, higher than any previously reported machine learning models. In addition, experiments conducted on three datasets - a Chinese SMS spam dataset, a benchmark movie review dataset, and a benchmark customer review dataset - further validate the effectiveness of the proposed model. The proposed CL-MHA Bi-GRU model has an accuracy of 99.46% when evaluated on the Chinese SMS spam dataset.