Document-level Sentiment Classification Research Articles

Hybrid sentiment analysis with textual and interactive information

In the past decade, various methods have been proposed to address document-level sentiment classification. However, the exploration of user–product interactions has not been sufficiently studied in the literature. In this work, we aim to investigate the effectiveness of exploiting user–product relations, and propose a hybrid semantic and interactive model for the classification task. The novelty of the proposed method is a ranking graph module and a latent matching module, where the former is capable of capturing high-order connectivity among the nodes, while the later is able to preserve semantics of local connectivity during the recursive graph learning processing. These two modules are seamlessly incorporated, enabling the proposed model to learn comprehensive and discriminative representations for the specific classification task. We conduct extensive experiments on three public datasets, and demonstrate the advantage of the proposed model over the state-of-the-art baselines.

Aspect Based Sentiment Analysis

Sentiment analysis, which addresses the computational treatment of opinion, sentiment, and subjectivity in text, has received considerable attention in recent years. In contrast to the traditional coarse-grained sentiment analysis tasks, such as document-level sentiment classification, we are interested in the fine-grained aspect-based sentiment analysis that aims to identify aspects that users comment on and these aspects’ polarities. Aspect-based sentiment analysis relies heavily on syntactic features. However, the reviews that this task focuses on are natural and spontaneous, thus posing a challenge to syntactic parsers. In this paper, we address this problem by proposing a framework of adding a sentiment sentence compression (Sent Comp) step before performing the aspect-based sentiment analysis. We apply a discriminative conditional random field model, with certain special features, to automatically compress sentiment sentences. Sentiment analysis is contextual mining of text which identifies and extracts subjective information in textual data. sentiment analysis proves to be an incredible asset for users to extract essential information and assists organizations with understanding the social sentiment of their brand, product or service while monitoring online conversations.

Opinion Mining-Based Term Extraction Sentiment Classification Modeling

The spread of social media has accelerated the formation and dissemination of user review data, which contain subjective opinions of users on products, in an e-commerce environment. Because these reviews significantly influence other users, opinion mining has garnered substantial attention in analyzing the positive and negative opinions of users and deriving solutions based on these analytical results. Terms that include sentimental information and used in user reviews serve as the most crucial element in sentimental classification. In this regard, it is crucial to distinguish the most influential terms in user reviews. This study proposed a document-level sentiment classification model based on the collection and application of user reviews generated in an e-commerce environment. Here, a term information extraction method was applied to the proposed model to select core terms, classify the selected terms according to parts of speech (POS), determine terms that can increase information power and influence, and adopt these terms in opinion mining research, based on SVM, SVM+, and SVM+MTL techniques. The results obtained from evaluating the proposed model indicate that it exhibited excellent sentiment analysis performance. The proposed model is expected to be effectively utilized in providing enhanced services for users and increasing competitiveness in the e-commerce environment.

Sentiment Analysis in the Era of Web 2.0: Applications, Implementation Tools and Approaches for the Novice Researcher

Nowadays, people find it easier to express opinions via social media-formally known as Web 2.0. Sentiment analysis is an essential field under natural language processing in Computer Science that deals with analyzing people's opinions on the subject matter and discovering the polarity they contain. These opinions could be processed in collective form (as a document) or segments or units as sentences or phrases. Sentiment analysis can be applied in education, research optimization, politics, business, education, health, science and so on, thus forming massive data that requires efficient tools and techniques for analysis. Furthermore, the standard tools currently used for data collection, such as online surveys, interviews, and student evaluation of teachers, limit respondents in expressing opinions to the researcher's surveys and could not generate huge data as Web 2.0 becomes bigger. Sentiment analysis techniques are classified into three (3): Machine learning algorithms, lexicon and hybrid. This study explores sentiment analysis of Web 2.0 for novice researchers to promote collaboration and suggest the best tools for sentiment data analysis and result efficiency. Studies show that machine learning approaches result in large data sets on document-level sentiment classification. In some studies, hybrid techniques that combine machine learning and lexicon-based performance are better than lexicon. Python and R programming are commonly used tools for sentiment analysis implementation, but SentimentAnalyzer and SentiWordnet are recommended for the novice.
 Keywords: Sentiment Analysis; Web 2.0; Applications; Tools; Novice

Injecting User Identity Into Pretrained Language Models for Document-Level Sentiment Classification

This paper mainly studies the combination of pre-trained language models and user identity information for document-level sentiment classification. In recent years, pre-trained language models (PLMs) such as BERT have achieved state-of-the-art results on many NLP applications, including document-level sentiment classification. On the other hand, a collection of works introduce additional information such as user identity for better text modeling. However, most of them inject user identity into traditional models, while few studies have been conducted to study the combination of pre-trained language models and user identity for even better performance. To address this issue, in this paper, we propose to unite user identity and PLMs and formulate User-enhanced Pre-trained Language Models (U-PLMs). Specifically, we demonstrate two simple yet effective attempts, i.e. embedding-based and attention-based personalization, which inject user identity into different parts of a pre-trained language model and provide personalization from different perspectives. Experiments in three datasets with two backbone PLMs show that our proposed methods outperform the best state-of-the-art baseline method with an absolute improvement of up to 3%, 2.8%, and 2.2% on accuracy. In addition, our methods encode user identity with plugin modules, which are fully compatible with most auto-encoding pre-trained language models.

LDA-Based Topic Modeling Sentiment Analysis Using Topic/Document/Sentence (TDS) Model

Customer reviews on the Internet reflect users’ sentiments about the product, service, and social events. As sentiments can be divided into positive, negative, and neutral forms, sentiment analysis processes identify the polarity of information in the source materials toward an entity. Most studies have focused on document-level sentiment classification. In this study, we apply an unsupervised machine learning approach to discover sentiment polarity not only at the document level but also at the word level. The proposed topic document sentence (TDS) model is based on joint sentiment topic (JST) and latent Dirichlet allocation (LDA) topic modeling techniques. The IMDB dataset, comprising user reviews, was used for data analysis. First, we applied the LDA model to discover topics from the reviews; then, the TDS model was implemented to identify the polarity of the sentiment from topic to document, and from document to word levels. The LDAvis tool was used for data visualization. The experimental results show that the analysis not only obtained good topic partitioning results, but also achieved high sentiment analysis accuracy in document- and word-level sentiment classifications.

Conciseness is better: Recurrent attention LSTM model for document-level sentiment analysis

Long short-term memory (LSTM) or gated recurrent units (GRUs) are usually employed to recurrently learn variable-length sentence representations with long-range dependency in document-level sentiment analysis. However, LSTM and GRUs are biased models in which the words in the tail of sentences are dominant over the words at the beginning. Although some reweighting methods, such as self-attention, are useful for reweighting each word according to its importance, when processing a document with many tokens, they tend to assign equally smaller weights to each word, leading to the keywords being covered by nonsentiment words. In this paper, a recurrent attention LSTM neural network is presented to iteratively locate an attention region covering the key sentiment words. By gradually reducing the range of attention and the number of tokens, the model can leverage the weight of the key sentiment words for final classification. Additionally, a joint loss function is implemented to highlight both keywords and appropriate attention regions. The comparative experiments are conducted on the IMDB, Yelp and Amazon document-level corpora. The results show that the proposed model outperforms several state-of-the-art methods in document-level sentiment classification.

A novel scheme of domain transfer in document-level cross-domain sentiment classification

The sentiment classification aims to learn sentiment features from the annotated corpus and automatically predict the sentiment polarity of new sentiment text. However, people have different ways of expressing feelings in different domains. Thus, there are important differences in the characteristics of sentimental distribution across different domains. At the same time, in certain specific domains, due to the high cost of corpus collection, there is no annotated corpus available for the classification of sentiment. Therefore, it is necessary to leverage or reuse existing annotated corpus for training. In this article, we proposed a new algorithm for extracting central sentiment sentences in product reviews, and improved the pre-trained language model Bidirectional Encoder Representations from Transformers (BERT) to achieve the domain transfer for cross-domain sentiment classification. We used various pre-training language models to prove the effectiveness of the newly proposed joint algorithm for text-ranking and emotional words extraction, and utilised Amazon product reviews data set to demonstrate the effectiveness of our proposed domain-transfer framework. The experimental results of 12 different cross-domain pairs showed that the new cross-domain classification method was significantly better than several popular cross-domain sentiment classification methods.

User’s Review Habits Enhanced Hierarchical Neural Network for Document-Level Sentiment Classification

Document-level sentiment classification is dedicated to predicting the sentiment polarity of document-level reviews posted by users about products and services. Many methods use neural networks have achieved very successful results on sentiment classification tasks. These methods usually focus on mining useful information from the text of the review documents. However, they ignore the importance of users’ review habits. The reviews posted by the same user when commenting on different products contain similar review habits, and reviews that contain highly similar review habits often have similar sentiment ratings. In this paper, we propose a novel sentiment classification algorithm that utilizes user’s review habits to enhance hierarchical neural networks, namely as HUSN. Firstly, we divide the reviews in the training set according to the users. All the reviews of each user are aggregated together and called the historical reviews of this user. Secondly, the target review in the test set and its multiple historical reviews in the training set are sent to the Long Short-Term Memory based hierarchical neural network to obtain the corresponding review document representations containing the user’s review habits. Finally, we calculate the similarities between the target review document representation and multiple historical review document representations. The higher the similarity, the closer the review habits of different reviews from the same user, and the closer the corresponding sentiment ratings. Experimental results show that the similarities between the review habits of different reviews from the same user can further improve the performance of document-level sentiment classification. The HUSN algorithm performs better than all baseline methods on three publicly available document-level review datasets.

DSC_NEXT: Document-Level Sentiment ClassificatioN with EXTensional-Information

Document-level sentiment classification is to assign an overall sentiment polarity to an opinion document. Some researchers have already realized that, in addition to document texts, extensional-information such as product features and user preferences can be quite useful. Many previous studies represent them as ID-type extensional-information and incorporate them into deep learning models. However, they ignore the descriptive extensional information that is also useful for document representations. This paper covers the following aspects: (1) introduces the Description of Opinion Target (DOT), a new extensional-information for document-level sentiment classification, (2) builds the Document-level Sentiment ClassificatioN with EXTensional-information (DSC_NEXT) dataset which consists of three datasets: IMDB_NEXT, Yelp_NEXT and CMRDB_NEXT and (3) validates the effectiveness of DOT by performing experiments based on current state-of-the-art (SOTA) document-level sentiment analysis methods. Implications for using extensional-information in neural network models are also considered.

Improving Document-Level Sentiment Classification Using Importance of Sentences.

Previous researchers have considered sentiment analysis as a document classification task, in which input documents are classified into predefined sentiment classes. Although there are sentences in a document that support important evidences for sentiment analysis and sentences that do not, they have treated the document as a bag of sentences. In other words, they have not considered the importance of each sentence in the document. To effectively determine polarity of a document, each sentence in the document should be dealt with different degrees of importance. To address this problem, we propose a document-level sentence classification model based on deep neural networks, in which the importance degrees of sentences in documents are automatically determined through gate mechanisms. To verify our new sentiment analysis model, we conducted experiments using the sentiment datasets in the four different domains such as movie reviews, hotel reviews, restaurant reviews, and music reviews. In the experiments, the proposed model outperformed previous state-of-the-art models that do not consider importance differences of sentences in a document. The experimental results show that the importance of sentences should be considered in a document-level sentiment classification task.

Speculative text mining for document-level sentiment classification

Many existing solutions perform document-level sentiment classification based on local document only, ignoring other texts that might contribute to better classification accuracy. In this paper, we propose a novel speculative sentiment classification model named SSC. In SSC, we speculate that users with similar rating behaviours are more likely to write documents of similar sentiments toward a product. The motivation of SSC, therefore, is to exploit those speculative similar documents for improving classification accuracy. The proposed SSC model consists of three main components, namely, user-product interaction (UPI) component, document encoding (DE) component, and speculative similar document (SSD) component. The UPI component models user-product interactions, and encodes user/product ratings behaviours into user/product embeddings. The DE component utilizes learned user/product embeddings to capture the informative word vectors for comprising more accurate document representations. The SSD component aggregates documents written by similar users toward the same product for speculative sentiment classification. Because the user similarities are calculated based on user embeddings that encode user rating behaviours, the aggregated documents are more likely to have similar sentiments. The three components are seamlessly integrated into a unified model. In the unified manner, these three components are jointly optimized, and they mutually complement each other to enhance sentiment classification. We conduct extensive experiments on three public datasets, and demonstrate the advantage of the proposed SSC model over state-of-the-art baselines.

Two-stage sentiment classification based on user-product interactive information

Document-level review sentiment classification aims to predict the sentiment category for given review documents written by users for products. Most of the existing methods focus on generating a good review document representation and classifying the review document directly. However, on the one hand, as document-level review sentiment classification usually includes many sentiment categories and the difference between these sentiment categories is not obvious, it may be difficult to obtain satisfying result by direct classification. On the other hand, this once classification process with review representation may fail to well interpret how the results are achieved. In addition, although some information such as user preference and product characteristics are incorporated when building models, the interactive information between user and product are usually ignored. In this paper, inspired by the deductive reasoning strategy of human doing multiple choice questions, we are motivated to propose a Two-Stage Sentiment Classification (TSSC) model to classify review documents in two stages: (1) Coarse classification stage, where model mainly adopts user-product interactive information to pre-judge the sentiment tendency of the review document without considering the review information; (2) Fine classification stage, where model uses text information of the review document for further classification based on the sentiment tendency obtained in coarse classification stage. Finally, the sentiment classification task is accomplished by combining both the results of coarse classification and fine classification. The experimental results demonstrate that our TSSC model significantly outperforms most of the related models (e.g., Trigram and NSC+UPA) on IMDB and Yelp datasets in terms of classification accuracy. When compared with the state-of-the-art HUAPA model, our TSSC model not only achieves slightly more accurate performance, but also has lower time complexity and stronger interpretability.

HieNN-DWE: A hierarchical neural network with dynamic word embeddings for document level sentiment classification

The recent rapid developments of neural networks have stimulated significant performance progress on document level sentiment classification. The existing researches focus on extracting long-distance features from documents, which fail to exploit the deep semantic information of documents. Besides, few of them have taken the representations of polysemous words into consideration, such that the semantic information of documents are not precisely exploited. To tackle these issues, we propose a novel hierarchical neural network model based on dynamic word embeddings (HieNN-DWE) for document level sentiment classification. Specifically, the dynamic word embeddings are produced by ELMo (Embedding from Language Models), which can deeply exploit the semantic information for polysemous words. The embeddings are then fed into HieNN-DWE which is a two-layer hierarchical neural network model. Note that the first layer utilizes bidirectional gated recurrent unit (BiGRU) as well as attention mechanism to encode sentences. The second layer uses both BiGRU and convolutional neural network (CNN) to capture global and local features from sentence representations. Thus, the final document representations are constructed by concatenating the outputs from BiGRU and CNN. Moreover, experimental results have shown that HieNN-DWE outperforms existing competitive methods on four public datasets for document level sentiment classification.

Document-level multi-topic sentiment classification of Email data with BiLSTM and data augmentation

Email data has unique characteristics, involving multiple topics, lengthy replies, formal language, high variance in length, high duplication, anomalies, and indirect relationships that distinguish it from other social media data. In order to better model Email documents and to capture complex sentiment structures in the content, we develop a framework for document-level multi-topic sentiment classification of Email data. Note that, a large volume of labeled Email data is rarely publicly available. We introduce an optional data augmentation process to increase the size of datasets with synthetically labeled data to reduce the probability of overfitting and underfitting during the training process. To generate segments with topic embeddings and topic weighting vectors as inputs for our proposed model, we apply both latent Dirichlet allocation topic modeling and semantic text segmentation to post-process Email documents. Empirical results obtained with multiple sets of experiments, including performance comparison against various state-of-the-art algorithms with and without data augmentation and diverse parameter settings, are analyzed to demonstrate the effectiveness of our proposed framework.

Aspect-Context Interactive Attention Representation for Aspect-Level Sentiment Classification

Aspect-level sentiment classification aims to determine sentiment polarities of various aspects in reviews, where each review typically contains multiple aspects, that may correspond to different polarities. Aspect-level sentiment classification, unlike document-level sentiment classification, requires different context representations for different aspects. Existing methods normally use Long Short-Term Memory (LSTM) network to model aspects and contexts separately, and they combine attention mechanisms to extract features of a specific aspect in its context. Attention mechanisms are not used for sequence modeling, so aspects are not considered when generating context sequence representations. This study proposes a novel aspect-context interactive representation structure that only relies on an attention mechanism for generating sequence-to-sequence representations in both context and aspect. It is capable of extracting features related to the specific aspect during the process of its context sequence modeling, and generating a high quality aspect representation simultaneously. We have conducted comprehensive experiments to compare with thirteen existing methods. Our experimental results show that the proposed model is able to achieve significantly better performance on Restaurant dataset, as well as very competitive results on Laptop and Twitter datasets.

An Integrated Word Embedding-Based Dual-Task Learning Method for Sentiment Analysis

Sentiment analysis aimed to automate the task of discriminating the sentiment tendency of a textual review, which expresses a simple sentiment as positive, negative, or neutral. In general, the basic sentiment analysis solution used for feature extraction is the word embedding technique, which only focuses on the contextual or global semantic information and ignores the sentiment polarity of text. Thus, the word embedding technique leads to biased analysis results, especially for some words that have the same semantic context but an opposite sentiment. In this paper, we propose an integrated sentiment embedding method to combine context and sentiment information using a dual-task learning algorithm to perform sentiment analysis. First, we propose three sentiment language models by encoding the sentiment information of texts into word embedding based on three existing semantic models, namely, continuous bag-of-words, prediction, and log-bilinear. Next, based on semantic language models and the proposed sentiment language models, we propose a dual-task learning algorithm to generate hybrid word embedding named integrated sentiment embedding, in which the joint learning method and parallel learning method are applied to jointly process tasks. Experiments on sentence-level and document-level sentiment classification tasks demonstrate that the proposed integrated sentiment embedding has better classification performances compared with basic word embedding methods.

Use of NLP Based Combined Features for Sentiment Classification

Sentiment analysis is the technique of automatic detection of the belief or the mood of an author towards a certain subject in textual form. To extract the opinion present in text, the machine needs expertise in the area of natural language processing. In this paper, machine learning based document-level sentiment classification is performed on Amazon product reviews to classify them as positive and negative. Two NLP based feature extraction techniques (Word Relation and POS based) are used in this study to determine the features that are sentiment bearing. The features are extracted as basic features (unigrams, bigrams and trigrams) and their combinations (unigrams+bigrams, unigrams+trigrams, unigrams+bigrams+trigrams). In order to identify the features that are most informative and to bring down the computational time of the classification algorithms, feature selection techniques are used. Performance of independent and combined feature sets is assessed using accuracy, precision, recall and F-measure. From the experiments conducted, it is observed that combined features outperformed independent features using Boolean Multinomial Naive Bayes (BMNB) classifier.

Combining attention-based bidirectional gated recurrent neural network and two-dimensional convolutional neural network for document-level sentiment classification

Neural networks lately have achieved a great success on sentiment classification due to their ability of feature extraction. However, it remains as an enormous challenge to model long texts in document-level sentiment classification as well as to explore semantics between sentences and dependencies between features. Moreover, most existing methods can hardly differentiate the importance of different contents when constructing a document representation. To tackle these problems, we propose a novel neural network model: AttDR-2DCNN, which mainly consists of two parts. The first part utilizes a two-layer compositional bidirectional Gated Recurrent Unit (GRU) to obtain the compositional semantics of the document, where the first layer learns the feature vector of the sentence, and the second layer learns the document matrix representation consisting of two dimensions of the time-step dimension and feature dimension, from the sentence vectors. The second part applies a two-dimensional convolution operation and two-dimensional max pooling to capture more dependencies between sentences features. We further utilize different types of attention mechanism in these two parts to distinguish the importance of words, sentences and features in the document. Experiments are conducted on four publicly available document-level review datasets and the result shows that the proposed model outperforms some existing models.

Exploring the use of syntactic dependency features for document-level sentiment classification

Exploring the use of syntactic dependency features for document-level sentiment classification