Multi-Grained Attention Representation With ALBERT for Aspect-Level Sentiment Classification

Yuezhe Chen,Dezhi Kong,Lingyun Kong,Yang Wang

doi:10.1109/access.2021.3100299

Yuezhe Chen, Dezhi Kong + Show 2 more

Open Access

https://doi.org/10.1109/access.2021.3100299

Copy DOI

Abstract

Aspect-level sentiment classification aims to solve the problem, which is to judge the sentiment tendency of each aspect in a sentence with multiple aspects. Previous works mainly employed Long Short-Term Memory (LSTM) and Attention mechanisms to fuse information between aspects and sentences, or to improve large language models such as BERT to adapt aspect-level sentiment classification tasks. The former methods either did not integrate the interactive information of related aspects and sentences, or ignored the feature extraction of sentences. This paper proposes a novel multi-grained attention representation with ALBERT (MGAR-ALBERT). It can learn the representation that contains the relevant information of the sentence and the aspect, while integrating it into the process of sentence modeling with multi granularity, and finally get a comprehensive sentence representation. In Masked LM (MLM) task, in order to avoid the influence of aspect words being masked in the initial stage of the pre-training, the noise linear cosine decay is introduced into n-gram. We implemented a series of comparative experiments to verify the effectiveness of the method. The experimental results show that our model can achieve excellent results on Restaurant dataset with numerous number of parameters reduced, and it is not inferior to other models on Laptop dataset.

Highlights

T HE process of judging the polarity of product reviews is sentiment analysis
Chen et al.: Multi-Grained Attention Representation with ALBERT for Aspect-Level Sentiment Classification over Attention (AOA) solves the problem that the above method only considers the attention from aspect to text
Our main contributions of this paper are summarized as follows: (1) We propose Multi-head Aspect-Context Attention over Attention (Multi AC-AOA), which focuses on interactively extracting features between aspect and context

Summary

INTRODUCTION

T HE process of judging the polarity of product reviews is sentiment analysis. Document-level sentiment classification, similar to text classification, is to classify a document or a comment [1], [2]. Y. Chen et al.: Multi-Grained Attention Representation with ALBERT for Aspect-Level Sentiment Classification over Attention (AOA) solves the problem that the above method only considers the attention from aspect to text. (2) We contributed a contextual block based on ALBERT In this part, we modified Masked LM task of ALBERT, which makes it more compatible with aspect-level sentiment classification task and more effective learning sentence representation [24]. There are many models that borrow AOA to implement aspect-level sentiment analysis tasks [14], [15] This structure can calculate the attention of both the query and the document at the same time, and can benefit from the mutual information. Contains m words, there are 3 categories of predicted output

ASPECT-CONTEXT ATTENTION OVER ATTENTION

CONTEXTUAL BLOCK

Findings

CONCLUSION