Multimodal Sentiment Analysis Research Articles

Deep Learning-Based Natural Language Processing Methods for Sentiment Analysis in Social Networks

This paper presents an in-depth study of the sentiment of social network communication through a deep learning-based natural language processing approach and designs a corresponding model to be applied in the actual social process. Specifically, the network can dynamically select the most important word in the current state according to the information available and achieve the accurate recognition of the dynamically changing important content in a sentence. Based on this, the semantic understanding of the whole sentence is achieved through a continuous cycle of the process. In addition, considering that the semantic representation of natural language is highly dependent on contextual information, the lack of contextual information will lead to the ambiguity and inaccuracy of semantic representation. In this paper, we study the sentiment analysis algorithms in social networks at two levels, unimodal and multimodal, and construct a text sentiment analysis model and a picture-text multimodal sentiment analysis model in social networks, respectively. By comparing the experiments with the existing models on several datasets, the accuracy of the two models exceeded the benchmark models by 4.45% and 5.2%, respectively, which verified the effectiveness of the two models. The feasibility of applying the optimized convolutional neural network recurrent optimization network to social network sentiment analysis is verified by practically applying the optimized convolutional neural network recurrent optimization network to single task and multitask and comparing other existing deep learning classifiers.

Effective deep learning based multimodal sentiment analysis from unstructured big data

AbstractMore recently, as images, memes and graphics interchange formats have dominated social feeds, typographic/infographic visual content has emerged as an important social media component. This multimodal text combines text and image, defining a novel visual language that must be analysed because it has the potential to modify, confirm or grade the sentiment's polarity. The problem is how to effectively use information from the visual and textual content in image‐text posts. This article presents a new deep learning‐based multimodal sentiment analysis (MSA) model using multimodal data such as images, text and multimodal text (image with embedded text). The text analytic unit, the discretization control unit, the picture analytic component and the decision‐making component are all included in this system. The discretization unit separates the text from the picture using the variant and channel augmented maximally stable extremal regions (VCA‐MSERs) technique, which are then analysed as discrete elements and fed into the appropriate image and text analytics units. The text analytics system utilizes a stacked recurrent neural network with multilevel attention and feedback module (SRNN‐MAFM) to detect the sentiment of the text. A deep convolutional neural network (CNN) structure with parallel‐dilated convolution and self‐attention module (PDC‐SAM) is developed to forecast the emotional response to visual content. Finally, the decision component employs a Boolean framework including an OR function to evaluate and classify the output into three fine‐grained sentiment classes: positive, neutral and negative. The proposed work is simulated in the python platform using the STS‐Gold, Flickr 8k and B‐T4SA datasets for sentiment analysis of text and visual and multimodal text. Simulation outcomes proved that the suggested method achieved better accuracy of 97.8%, 97.7% and 90% for text, visual and MSA individually compared to other methods.

A soft voting ensemble learning-based approach for multimodal sentiment analysis

A soft voting ensemble learning-based approach for multimodal sentiment analysis

SCANET: Improving multimodal representation and fusion with sparse‐ and cross‐attention for multimodal sentiment analysis

AbstractLearning unimodal representations and improving multimodal fusion are two cores of multimodal sentiment analysis (MSA). However, previous methods ignore the information differences between different modalities: Text modality has high‐order semantic features than other modalities. In this article, we propose a sparse‐ and cross‐attention (SCANET) framework which has asymmetric architecture to improve performance of multimodal representation and fusion. Specifically, in the unimodal representation stage, we use sparse attention to improve the representation efficiency of two modalities and reduce the low‐order redundant features of audio and visual modalities. In the multimodal fusion stage, we design an innovative asymmetric fusion module, which utilizes audio and visual modality information matrix as weights to strengthen the target text modality. We also introduce contrastive learning to effectively enhance complementary features between modalities. We apply SCANET on the CMU‐MOSI and CMU‐MOSEI datasets, and experimental results show that our proposed method achieves state‐of‐the‐art performance.

A multi-stage multimodal framework for sentiment analysis of Assamese in low resource setting

Multimodality has shown to be helpful in several natural language processing tasks. Thus, adding multiple modalities to the traditional sentiment analysis has also proven to be useful. However, multimodality in a low resource setting for sentiment analysis is yet to be explored for several resource-constrained languages. Assamese is a low-resource language spoken mainly in the state of Assam in India. This paper presents an Assamese multimodal dataset comprising of 16,000 articles from the news domain as a benchmark resource. Secondly, we present a multi-stage multimodal sentiment analysis framework that concurrently exploits textual and visual cues to determine the sentiment. The proposed architecture encodes the news content collaboratively. The text branch encodes semantic content information by considering the semantic information of the news. At the same time, the visual branch encodes the visual appearance information from the news image. Then, an intermediate fusion-based multimodal framework is proposed to exploit the internal correlation between textual and visual features for joint sentiment classification. Finally, a decision-level fusion mechanism is employed on the three models to integrate cross-modal information effectively for final sentiment prediction. Experimental results conducted on the Assamese dataset built in-house demonstrate that the contextual integration of multimodal features delivers better performance (89.3%) than the best unimodal features (85.6%).

Modality-Invariant and -Specific Representations with Crossmodal Transformer for Multimodal Sentiment Analysis

Human emotion judgments usually receive information from multiple modalities such as language, audio, as well as facial expressions and gestures. Because different modalities are represented differently, multimodal data exhibit redundancy and complementarity, so a reasonable multimodal fusion approach is essential to improve the accuracy of sentiment analysis. Inspired by the Crossmodal Transformer for multimodal data fusion in the MulT (Multimodal Transformer) model, this paper adds the Crossmodal transformer for modal enhancement of different modal data in the fusion part of the MISA (Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis) model, and proposes three MISA-CT models. Tested on two publicly available multimodal sentiment analysis datasets MOSI and MOSEI, the experimental results of the models outperformed the original MISA model.

Feature-guided Multimodal Sentiment Analysis towards Industry 4.0

• Advanced and efficient image-text multimodal fusion approach. • Clever use of matrix transformation to achieve alignment of different modal features. • Using attention mechanism to ensure the parallelism of the model and improve the training speed. • The unimodal and multimodal features are stitched together to ensure the complementarity of the modalities. • The pain point problem of difficult to obtain domain datasets is solved by constructing a generic dataset. Combining Artificial Intelligence (AI) to process rich media information has become an important part of Industry 4.0. Sentiment recognition in AI aims to analyze user emotions contained in rich media to facilitate service enhancement. Previous research on sentiment recognition has mainly focused on academia, and few have discussed algorithmic applications and innovations in industry. In this paper, we propose a general approach for multimodal sentiment recognition for images and text. The method provides a new approach for processing rich media information by fully considering the internal features of each modality itself as well as the correlations between the modalities. In the dataset constructed in this paper, the accuracy rate is improved by more than 4% compared with the method using single modality. The effectiveness and generality of the method in multimodal sentiment recognition is demonstrated by extending the experiments with a multimodal public dataset.

Ensemble transfer learning-based multimodal sentiment analysis using weighted convolutional neural networks

Huge amounts of multimodal content and comments in a mixture form of text, image, and emoji are continuously shared by users on various social networks. Most of the comments of the users in these networks have emotional aspects, which make the multimodal sentiment analysis (MSA) an important and attractive research topics in this area. In this paper, an ensemble transfer learning method is exploited to propose a hybrid MSA model based on weighted convolutional neural networks. The extended Dempster–Shafer (Yager) theory is also utilized in the proposed method of this paper to fuse the outputs of text and image classifiers to determine the final polarity at the decision level. The pre-trained VGG16 network is firstly used to extract visual features and fine-tune on the MVSA-Multiple and T4SA datasets for image sentiment classification. The Mask-RCNN model is then exploited to determine the objects in the images and convert them to text. The BERT model receives the output of this step along with the textual descriptions of the images for extracting the text features and embedding the words. The output of the BERT model is then imported into a weighted convolutional neural network ensemble (WCNNE). The texts are classified by several weak learners using the AdaBoost that is an ensemble learning technique in which, classifiers are trained sequentially. The combined use of several weak classifiers results in a strong classification. The WCNNE improves the performance and increases the accuracy of the results. As a fusing phase at the decision level, the outputs of the VGG16 and the WCNNE models will be finally merged using the extended Dempster-Shafer theory to obtain the correct sentiment label. The results of the experiments on the MVSA-Multiple and T4SA datasets show that the proposed model is better than the other compared methods and achieved an appropriate accuracy of 0.9348 on MVSA and 0.9689 on the T4SA datasets. Moreover, the proposed model reduces training time due to the use of transfer learning and the proposed AdaBoostCNN achieves better results compared to the single CNN.

RETRACTED ARTICLE: ICDN: integrating consistency and difference networks by transformer for multimodal sentiment analysis

RETRACTED ARTICLE: ICDN: integrating consistency and difference networks by transformer for multimodal sentiment analysis

Multimodal Sentiment Analysis Based on Interactive Transformer and Soft Mapping

Multimodal sentiment analysis aims to harvest people’s opinions or attitudes from multimedia data through fusion techniques. However, existing fusion methods cannot take advantage of the correlation between multimodal data but introduce interference factors. In this paper, we propose an Interactive Transformer and Soft Mapping based method for multimodal sentiment analysis. In the Interactive Transformer layer, an Interactive Multihead Guided-Attention structure composed of a pair of Multihead Attention modules is first utilized to find the mapping relationship between multimodalities. Then, the obtained results are fed into a Feedforward Neural Network. The Soft Mapping layer consisting of stacking Soft Attention module is finally used to map the results to a higher dimension to realize the fusion of multimodal information. The proposed model can fully consider the relationship between multiple modal pieces of information and provides a new solution to the problem of data interaction in multimodal sentiment analysis. Our model was evaluated on benchmark datasets CMU-MOSEI and MELD, and the accuracy is improved by 5.57% compared with the baseline standard.

MULTIMODALANALYSIS OF VIDEO WITH NEURAL NETWORKS

The cyberspace technology in age of social-media, has introduce an innovative and ravishing way of passing the information in form of online streaming videos. Interestingly, videos are able to communicate a complex message in some seconds of duration involving all auditory and visual senses. Hence, somewhere it is needed to monitor the videos, its shared context and its affect over viewers. Thereby, it becomes imperative to test the negative, positive and neutral sentiments hidden within the video which is also termed as sentiment analysis of videos. For this text, audio and visuals are known modalities in consideration to relation with utterances. This paper presents the contextual multimodal sentiment analysis using the technique of neural network using attention mechanism.

Dynamic Invariant-Specific Representation Fusion Network for Multimodal Sentiment Analysis.

Multimodal sentiment analysis (MSA) aims to infer emotions from linguistic, auditory, and visual sequences. Multimodal information representation method and fusion technology are keys to MSA. However, the problem of difficulty in fully obtaining heterogeneous data interactions in MSA usually exists. To solve these problems, a new framework, namely, dynamic invariant-specific representation fusion network (DISRFN), is put forward in this study. Firstly, in order to effectively utilize redundant information, the joint domain separation representations of all modes are obtained through the improved joint domain separation network. Then, the hierarchical graph fusion net (HGFN) is used for dynamically fusing each representation to obtain the interaction of multimodal data for guidance in the sentiment analysis. Moreover, comparative experiments are performed on popular MSA data sets MOSI and MOSEI, and the research on fusion strategy, loss function ablation, and similarity loss function analysis experiments is designed. The experimental results verify the effectiveness of the DISRFN framework and loss function.

MEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis

Multimodal sentiment analysis is a challenging task in the field of natural language processing (NLP). It uses multimodal signals (natural language, facial gestures, and acoustic behavior) in videos to generate emotional understanding. However, the importance of single modality data in the video to emotional outcomes is not static. With the extension of the time dimension, the emotional attributes of a specific natural language will be affected by non-natural language data, resulting in a vector shift in the feature space. At the same time, long-term dependencies within a specific modality and long-term dependencies between multiple modalities that are “unaligned” need to be considered. In response to the above problems, this paper proposes Multimodal Encoding-Decoding Network with Transformer. The network model encodes multimodal data through a Bidirectional Encoder Representations from Transformers (BERT) network and Transformer encoder to resolve long-term dependencies within modalities. And the network reconstructs the Transformer decoder to solve the weight problem of multimodal data in an iterative way. The network fully considers the long-term dependencies between modalities and the offset effect of non-natural language data on natural language data. Under the same experimental conditions, we validated our model on general multimodal sentiment analysis datasets. Compared with state-of-the-art models, the network achieves good progress and strong stability.

Multimodal Sentiment Analysis With Two-Phase Multi-Task Learning

Multimodal Sentiment Analysis (MSA) is a challenging research area that studies sentiment expressed from multiple heterogeneous modalities. Given those pre-trained language models such as BERT have shown state-of-the-art (SOTA) performance in multiple NLP disciplines, existing models tend to integrate these modalities into BERT and treat the MSA as a single prediction task. However, we find that simply fusing the multimodal features into BERT cannot well establish the power of a strong pre-trained model. Besides, the classification ability of each modality is also suppressed by single-task learning. In this paper, we propose a multimodal framework named Two-Phase Multi-task Sentiment Analysis (TPMSA), which applies a two-phase training strategy to make the most of the pre-trained model and a novel multi-task learning strategy to investigate the classification ability of each representation. We conduct experiments on two multimodal benchmark datasets CMU-MOSI and CMU-MOSEI, the results show that our TPMSA model significantly outperforms the current SOTA method on both datasets across all the metrics, which clearly show the effectiveness of our proposed method. Our code is available at \url{https://github.com/TPMSA/TPMSA}.

CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) is an emerging research field that aims to identify the sentiment of speakers through Audio (A), Video (V), and Text (T) modalities. The major challenge is to capture joint representations that can associate and integrate information from various modalities. Most of the existing methods are prone to acquiring joint representation through the concatenation of input features. However, these methods are short of exploiting interactions fully to ensure consistency and complementarity among modalities. To solve this problem, we design a novel multimodal sentiment analysis framework named Cross-Modal Joint Representation Transformer (CMJRT), which exploits hierarchical interactions among modalities by passing joint representations from bimodality to unimodality. Specifically, we adopt cyclic translation to obtain joint representations of bimodality, where one modality is translated to the other modality forward and backward by encoder-decoders. The translation process ensures consistency between modalities. In addition, to explore complementarity among modalities, the cross-modal transformer is used to reinforce each unimodality with common information from bimodality. Extensive experiments on CMU-MOSI and CMU-MOSEI datasets demonstrate that our proposed method outperforms existing approaches.

MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis

Multimodal sentiment analysis mainly concentrates on language, acoustic and visual information. Previous work based on BERT utilizes only text (language) representation to fine-tune BERT, while ignoring the importance of nonverbal information. Due to the fact that features extracted from a single modality may contain uncertainty, it is challenging for BERT to perform well in real-world applications. In this paper, we propose a multimodal fusion BERT that can explore the time-dependent interactions among different modalities. Additionally, prior BERT-based methods tend to train the models with only one optimizer to update the parameters. However, we argue that BERT has been pre-trained with a lot of corpora so it needs to be fine-tuned slightly. Therefore, an internal updating mechanism is introduced to avoid the overfitting of the model in the training process. We set two optimizers for multimodal fusion BERT and other components of the model with different learning rates, which enables the model to attain optimal parameters. The results of experiments on public datasets demonstrate that our model is superior to the baselines and achieves the state-of-the-art.

Multi-level Attention Map Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) is a very challenging task due to its complex and complementary interactions between multiple modalities, which can be widely applied into areas of product marketing, public opinion monitoring, and so on. However, previous works directly utilized the features extracted from multimodal data, in which the noise reduction within and among multiple modalities has been largely ignored before multimodal fusion. This paper proposes a multi-level attention map network (MAMN) to filter noise before multimodal fusion and capture the consistent and heterogeneous correlations among multi-granularity features for multimodal sentiment analysis. Architecturally, MAMN is comprised of three modules: multi-granularity feature extraction module, multi-level attention map generation module, and attention map fusion module. The first module is designed to sufficiently extract multi-granularity features from multimodal data. The second module is constructed to filter noise and enhance the representation ability for multi-granularity features before multimodal fusion. And the third module is built to extensibly mine the interactions among multi-level attention maps by the proposed extensible co-attention fusion method. Extensive experimental results on three public datasets show the proposed model is significantly superior to the state-of-the-art methods, and demonstrate its effectiveness on two tasks of document-based and aspect-based MSA tasks.

The Weighted Cross-Modal Attention Mechanism With Sentiment Prediction Auxiliary Task for Multimodal Sentiment Analysis

Human brain extracts the spatial and temporal semantic information by processing the multi-modalities, which has contextually meaningful for perceiving and understanding the emotional state of an individual. However, there are two main challenges in modeling multimodal sequences: 1) the different sampling rates from multimodal data make the cross-modal interactions very difficult; 2) how to efficiently fuse unimodal representations and effectively capture relationships among multimodal data. In this paper, we design the weighted cross-modal attention mechanism, which not only captures the temporal correlation information and the spatial dependence information of each modality, but also dynamically adjusts the weight of each modality across different time steps. And the unimodal subtasks are led for assisting the representation learning of specific modality to jointly train the multimodal tasks and unimodal subtasks to explore the complementary relationships of each modality. Our model gets a new state-of-the-art record on the CMU-MOSI dataset and brings noticeable performance improvements on all the metrics. For the CMU-MOSEI dataset, the F1 score of the binary classification, the 7-class task, and the regression task of our model are still the highest among all models and the proposed model is only lower than the multimodal split attention fusion (MSAF) model with aligned data on the accuracy of the binary classification, showing the great performance of the suggested method.

Heterogeneous Graph Convolution Based on In-Domain Self-Supervision for Multimodal Sentiment Analysis

Heterogeneous Graph Convolution Based on In-Domain Self-Supervision for Multimodal Sentiment Analysis

M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis.

Multimodal sentiment analysis aims to recognize people's attitudes from multiple communication channels such as verbal content (i.e., text), voice, and facial expressions. It has become a vibrant and important research topic in natural language processing. Much research focuses on modeling the complex intra- and inter-modal interactions between different communication channels. However, current multimodal models with strong performance are often deep-learning-based techniques and work like black boxes. It is not clear how models utilize multimodal information for sentiment predictions. Despite recent advances in techniques for enhancing the explainability of machine learning models, they often target unimodal scenarios (e.g., images, sentences), and little research has been done on explaining multimodal models. In this paper, we present an interactive visual analytics system, M2 Lens, to visualize and explain multimodal models for sentiment analysis. M2 Lens provides explanations on intra- and inter-modal interactions at the global, subset, and local levels. Specifically, it summarizes the influence of three typical interaction types (i.e., dominance, complement, and conflict) on the model predictions. Moreover, M2 Lens identifies frequent and influential multimodal features and supports the multi-faceted exploration of model behaviors from language, acoustic, and visual modalities. Through two case studies and expert interviews, we demonstrate our system can help users gain deep insights into the multimodal models for sentiment analysis.