Abstract
Aspect-Based Sentiment Analysis (ABSA) is one of the highly challenging tasks in natural language processing. It extracts fine-grained sentiment information in user-generated reviews, as it aims at predicting the polarities towards predefined aspect categories or relevant entities in free text. Previous deep learning approaches usually rely on large-scale pre-trained language models and the attention mechanism, which applies the complete computed attention weights and does not place any restriction on the attention assignment. We argue that the original attention mechanism is not the ideal configuration for ABSA, as for most of the time only a small portion of terms are strongly related to the sentiment polarity of an aspect or entity. In this paper, we propose a masked attention mechanism customized for ABSA, with two different approaches to generate the mask. The first method sets an attention weight threshold that is determined by the maximum of all weights, and keeps only attention scores above the threshold. The second selects the top words with the highest weights. Both remove the lower score parts that are assumed to be less relevant to the aspect of focus. By ignoring part of input that is claimed irrelevant, a large proportion of input noise is removed, keeping the downstream model more focused and reducing calculation cost. Experiments on the Multi-Aspect Multi-Sentiment (MAMS) and SemEval-2014 datasets show significant improvements over state-of-the-art pre-trained language models with full attention, which displays the value of the masked attention mechanism. Recent work shows that simple self-attention in Transformer quickly degenerates to a rank-1 matrix, and masked attention may be another cure for that trend.
Highlights
Sentiment analysis [1]–[4] is one of the prevalent tasks in natural language processing (NLP)
As sentiment information related to aspect terms is the key to solving the Aspect-based sentiment analysis (ABSA) task, it may be assumed that combining the attention mechanism with pre-trained language models should improve the performance at aspect level
That proves the representative power of pre-trained language models, which sets a competitive baseline for further research
Summary
Sentiment analysis [1]–[4] is one of the prevalent tasks in natural language processing (NLP). As sentiment information related to aspect terms is the key to solving the ABSA task, it may be assumed that combining the attention mechanism with pre-trained language models should improve the performance at aspect level. It introduces orthogonal regularization to restrict different aspects from focusing on the same parts of a sentence, ensuring more sparse attention for multiple aspects These methods extract the semantic information of the word embedding through complex network structures, and have achieved competitive results in ABSA. As an extension to the BERT model, BERT-SPC sends ‘‘[CLS] + sentence sequence + [SEP] + aspect sequence+ [SEP]’’ to the hidden layer output of the pre-trained BERT network for aspect sentiment classification It achieves good performance in ABSA, following the sentence prediction task in BERT that proves its ability to capture the relationship between aspect information and the whole sentence. 6) LOSS The cross-entropy loss is used to calculate the disagreement between the predicted label and the true label
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
