Multimodal Sentiment Research Articles

A dual-channel multimodal sentiment analysis framework based on three-way decision

In the exploration of human–computer interaction, accurately identifying and understanding the user’s sentimental state helps enhance the user experience. Multimodal sentiment analysis aims to identify sentiments through various signals. However, inconsistencies in sentiments between text and visual–audio signals pose a challenge, as models often struggle to capture these differences adequately. To alleviate this issue, we propose a dual-channel sentiment analysis framework based on three-way decision. This framework includes a multimodal model, a visual–audio model, and a three-way decision module. Specifically, the multimodal model and visual–audio model generate predictions, and the three-way decision module selects an appropriate prediction. We improve the binary cross-entropy loss function for the visual–audio model to integrate the three-way decision module better and improve sentiment prediction accuracy. Additionally, we introduce an adversarial training task to ensure differences in predictions between the two models, thereby enhancing the influence of the visual–audio model. Experimental results on four public datasets demonstrate that our framework significantly improves the accuracy of sentiment recognition and reduces the average decision time.

Attention-optimized vision-enhanced prompt learning for few-shot multi-modal sentiment analysis

Attention-optimized vision-enhanced prompt learning for few-shot multi-modal sentiment analysis

Frame-level nonverbal feature enhancement based sentiment analysis

Multimodal Sentiment Analysis (MSA) comprehensively utilizing data from multiple modalities to obtain more accurate sentiment attribute, has important applications in other fields, such as social media analysis, user experience evaluation and medical health, etc. It is worth noting that previous studies have paid little attention to the inconsistency of the initial representation granularity between verbal (textual) and nonverbal (acoustic and visual) modalities. As a result, the imbalanced emotional information between them complicates the interaction process, and ultimately affects the model’s performance. To solve this problem, this paper proposes a Frame-level Nonverbal feature Enhancement Network (FNENet) to improve performance on MSA by reducing the gap and integrating asynchronous affective information between modalities. Specifically, Vector Quantization (VQ) is applied to nonverbal modalities to reduce the granularity differences and improve the performance of the model. Additionally, nonverbal information is integrated through the Sequence Fusion mechanism (SF) into a pre-trained language model to enhance the textual representation, which benefits the word-level semantic expression according to the asynchronous affective cues preserved in unaligned frame-level nonverbal features. Extensive experiments on three benchmark datasets demonstrate that FNENet significantly outperforms baseline methods. It indicates that our model has potential application on MSA.

Extracting method for fine-grained emotional features in videos

Multimodal Sentiment Analysis (MSA) has significant applications in social media analysis and healthcare. It utilizes features from multiple modalities e.g., video, general text, acoustics, and vision, to obtain more credible sentiment analysis results. However, previous studies have ignored substantial variations in sub-features emerging within both the acoustic and visual modalities during feature extraction. Instead, they primarily rely on a simple concatenation method to derive representations. Consequently, these approaches prevent feature extractors from effectively leveraging non-verbal (acoustic and visual) features, thereby, constraining the model’s overall performance. To solve this problem, this study proposes a method for extracting fine-grained emotional features from videos. By segregating the initial features of non-verbal modalities into distinct domains, then separately modeling and uniformly re-integrating them, our method effectively exploits the modality-specific information in these original features. Through extensive experiments, the performance of all models significantly improves using the features extracted following our method compared with original ones. This substantiates that our proposed approach more effectively utilizes features from non-verbal modalities compared with conventional approaches. This also underscores that processing non-verbal sub-features separately before integration represents a viable solution for enhancing the performance of the MSA model.

TEMM: text-enhanced multi-interactive attention and multitask learning network for multimodal sentiment analysis

TEMM: text-enhanced multi-interactive attention and multitask learning network for multimodal sentiment analysis

Disentangled variational auto-encoder for multimodal fusion performance analysis in multimodal sentiment analysis

Multimodal Sentiment Analysis (MSA) holds extensive applicability owing to its capacity to analyze and interpret users' emotions, feelings, and perspectives by integrating complementary information from multiple modalities. However, inefficient and unbalanced cross-modal information fusion substantially undermines the accuracy and reliability of MSA models. Consequently, a critical challenge in the field now lies in effectively assessing the information integration capabilities of these models to ensure balanced and equitable processing of multimodal data. In this paper, a Disentanglement-based Variable Auto-Encoder (DVAE) is proposed for systematically assessing fusion performance and investigating the factors that facilitate multimodal fusion. Specifically, a distribution constraint module is presented to decouple the fusion matrices and generate multiple low-dimensional and trustworthy disentangled latent vectors that adhere to the authentic unimodal input distribution. In addition, a combined loss term is modified to effectively balance inductive bias, signal reconstruction, and distribution constraint items to facilitate the optimization of neural network weights and parameters. Utilizing the proposed evaluation method, we can evaluate the fusion performance of multimodal models by contrasting the classification degradation ratio derived from disentangled hidden representations and joint representations. Experiments conducted with eight state-of-the-art multimodal fusion methods on the CMU-MOSEI and CMU-MOSEI benchmark datasets demonstrate that DVAE is capable of effectively evaluating the effects of multimodal fusion. Moreover, the comparative experimental results indicate that the equalizing effect among various advanced mechanisms in multimodal sentiment analysis, as well as the single-peak characteristic of the ground label distribution, both contribute significantly to multimodal data fusion.

Cognitive Hybrid Deep Learning-based Multi-modal Sentiment Analysis for Online Product Reviews

Recently the field of sentiment analysis has gained a lot of attraction in literature. The idea that a machine can dynamically spot the text’s sentiments is fascinating. In this paper, we propose a method to classify the textual sentiments in Twitter feeds. In particular, we focus on analyzing the tweets of products as either positive or negative. The proposed technique utilizes a deep learning schema to learn and predict the sentiment by extracting features directly from the text. Specifically, we use Convolutional Neural Networks with different convolutional layers. Further, we experiment with LSTMs and try an ensemble of multiple models to get the best results. We employ an n-gram-based word embeddings approach to get the machine-level word representations. Testing of the method is conducted on real-world datasets. We have discovered that the ensemble technique yields the best results after conducting experiments on a huge corpus of more than one million tweets. To be specific, we get an accuracy of 84.95%. The proposed method is also compared with several existing methods. An extensive numerical investigation has revealed the superiority of the proposed work in actual deployment scenarios.

A Novel Multi-modal Sentiment Analysis Based on Multiple Kernel Learning with Margin-Dimension Constraint

Most of the sentiment analysis studies focus on the sentimental classification of pictures in the video, ignoring the spatio-temporal information of the sequence of picture frames as well as text and audio information. The multiple kernel learning is a new hotspot in the field of nuclear machine learning, capable of handling multiple modalities. For multiple kernel learning, it is easy to ignore the basic features that are not discriminative, and cannot make full use of the base features of different modes. This paper puts forward a novel multi-modal fusion model for sentiment analysis, in which a multiple kernel learning algorithm based on convolution margin-dimension constraint is proposed for feature fusion. Moreover, the 3D convolutional neural network is used to extract the features of visual information, and the multiple kernel learning algorithm based on margin-dimension constraint is used to fuse visual, text and audio sentiment features. Experiments conducted on the MOUD and IEMOCAP sentiment databases show that the proposed model outperforms existing models in the field of multi-modal sentiment analysis research.

MCPL: Multi-model co-guided progressive learning for multimodal aspect-based sentiment analysis

Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to combine image and text information to extract aspect terms from sentences and predict aspect-based sentiment polarities. Previous joint approaches have either viewed this task as collapsed label sequence annotation or modelled each subtask as a unified index sequence generation task, overlooking the explicit relation among its subtasks. This type of method cannot effectively integrate the strengths of task-specific models and adapt to difficult joint prediction tasks. To address this problem, we propose a Multi-model Co-guided Progressive Learning (MCPL) strategy that leverages task-specific models and the correlations among downstream tasks to expand high-quality training datasets, and provides progressive supervision signals to enhance the model’s ability to adapt from simple to difficult tasks. Based on MCPL, we construct a Phase-wise Progressive Learning (PPL) module for MABSA, in which three Task Affinity-based Pseudo-label Generation (TAPG) modules are proposed for task-specific label expansion. The TAPG modules leverage the task correlations among three MABSA downstream tasks for co-guided teaching, and the PPL module utilises their gradual relationship from aspect term extraction to perform sentiment classification and then joint prediction for progressive learning. Experiments on widely used public datasets demonstrate that the proposed MCPL method achieves excellent performance on the three subtasks of MABSA and outperforms most state-of-the-art methods. The source code is publicly available at https://github.com/qujiaqi-babu/MCPL.

Text-centered cross-sample fusion network for multimodal sentiment analysis

Text-centered cross-sample fusion network for multimodal sentiment analysis

Advanced Multimodal Sentiment Analysis with Enhanced Contextual Fusion and Robustness (AMSA-ECFR): Symmetry in Feature Integration and Data Alignment

Multimodal sentiment analysis, a significant challenge in artificial intelligence, necessitates the integration of various data modalities for accurate human emotion interpretation. This study introduces the Advanced Multimodal Sentiment Analysis with Enhanced Contextual Fusion and Robustness (AMSA-ECFR) framework, addressing the critical challenge of data sparsity in multimodal sentiment analysis. The main components of the proposed approach include a Transformer-based model employing BERT for deep semantic analysis of textual data, coupled with a Long Short-Term Memory (LSTM) network for encoding temporal acoustic features. Innovations in AMSA-ECFR encompass advanced feature encoding for temporal dynamics and an adaptive attention-based model for efficient cross-modal integration, achieving symmetry in the fusion and alignment of asynchronous multimodal data streams. Additionally, the framework employs generative models for intelligent approximation of missing features. It ensures robust alignment of high-level features with multimodal data context, effectively tackling issues of incomplete or noisy inputs. In simulation studies, the AMSA-ECFR model demonstrated superior performance against existing approaches. The symmetrical approach to feature integration and data alignment contributed significantly to the model’s robustness and precision. In simulations, the AMSA-ECFR model demonstrated a 10% higher accuracy and a 15% lower mean absolute error than the current best multimodal sentiment analysis frameworks.

Mual: enhancing multimodal sentiment analysis with cross-modal attention and difference loss

Mual: enhancing multimodal sentiment analysis with cross-modal attention and difference loss

Self-HCL: Self-Supervised Multitask Learning with Hybrid Contrastive Learning Strategy for Multimodal Sentiment Analysis

Self-HCL: Self-Supervised Multitask Learning with Hybrid Contrastive Learning Strategy for Multimodal Sentiment Analysis

Analysis of neural networks in text sentiment

The proliferation of digital media and online platforms has led to a significant increase in the production and distribution of textual data. Consequently, the study of text sentiment analysis has emerged as a crucial area of research within the domain of natural language processing. The objective of sentiment analysis is to detect and categorize emotional inclinations in text, which has substantial practical importance in comprehending public sentiment, monitoring brand reputation, and other related areas. This research does a comprehensive investigation of the utilization of neural network models in text sentiment analysis using literature review methods. It examines the difficulties encountered and the approaches employed to overcome these difficulties. The research encompasses the fundamental principles and techniques of text sentiment analysis utilizing neural networks, as well as the utilization of convolutional neural networks (CNN), recurrent neural networks (RNN)/long short-term memory networks (LSTM), attention mechanisms, and pre-trained models in the field of sentiment analysis. Furthermore, this study delves into the complexities associated with data noise, the interpretability of models, the analysis of long text and multimodal sentiment, as well as the utilization of transfer learning and domain adaptation techniques. The study's importance is in establishing a theoretical basis and practical recommendations for the advancement of sentiment analysis technology, particularly in enhancing model precision, comprehensibility, and range of applications, thereby delivering useful insights.

Multi-task disagreement-reducing multimodal sentiment fusion network

Existing multimodal sentiment analysis models can effectively capture sentimental commonalities between different modalities and possess high sentimental acquisition capability. However, there are still shortcomings in the model's analysis and recognition abilities when dealing with samples that exhibit sentimental polarity disagreement between different modalities. Additionally, the dominance of the text modality in multimodal models, particularly those pre-trained with BERT, can hinder the learning of other modalities due to its richer semantic information. This issue becomes particularly pronounced in cases where there is a conflict between multimodal and textual sentimental polarities, often leading to suboptimal analytical results. Besides, the classification ability of each modality is also suppressed by single-task learning. In this paper, We propose a Multi-Task disagreement-Reducing Multimodal Sentiment Fusion Network (MtDr-MSF), designed to enhance the semantic information of non-text modalities and reduce the dominant impact of the textual modality on the model, and to improve the learning capabilities of unimodal networks. We conducted experiments on multimodal sentiment analysis datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS. The results show that our method outperforms the current SOTA method.

TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion Network

Multimodal sentiment analysis (MSA) has become a popular field of research in recent years. The aim is to combine the three modalities of text, video, and audio to obtain comprehensive emotional information. However, current research often treats these three modalities equally, downplaying the crucial role of text modality in MSA and ignoring the redundant information generated during multimodal fusion. To address these problems, we propose the Text-Centric Hierarchical Fusion Network (TCHFN), employing a hierarchical fusion strategy. In this framework, low-level fusion involves cross-modal interactions between pairs of modalities, while high-level fusion extends these interactions to involve all three modalities. Through the design of the Cross-modal Reinforced Transformer (CRT), we achieve cross-modal enhancement of the target modality, facilitating a nuanced fusion process with text serving as its core. Additionally, we design Text-Centric Contrastive Learning (TCCL) to align non-text modalities with the text modality, emphasising the central role of text in the fusion process. After fusion, a multimodal fusion output gate is employed to mitigate redundant information within the multimodal fusion representation, which is subsequently processed by a linear layer for prediction. Simultaneously, to fully leverage limited labelled datasets, we introduced knowledge distillation. This approach involves preserving the model parameters that yield the best performance during training as a teacher model. The teacher model aids in capturing rich emotional information, enabling the model to transcend local optima and discover more optimal parameters, thereby enhancing overall model performance. Extensive experiments on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate the superiority of our model over state-of-the-art methods in MSA tasks.

Joint training strategy of unimodal and multimodal for multimodal sentiment analysis

With the explosive growth of social media video content, research on multimodal sentiment analysis (MSA) has attracted considerable attention recently. Despite significant progress in MSA, there remains challenges: current research mostly focuses on learning either unimodal features or aspects of multimodal interactions, neglecting the importance of simultaneously considering both unimodal features and intermodal interactions. To address the aforementioned challenges, this paper proposes a fusion strategy called Joint Training of Unimodal and Multimodal (JTUM). Specifically, this strategy combines unimodal label generation module with cross-modal transformer. The unimodal label generation module aims to generate more distinctive labels for each unimodal input, facilitating more effective learning of unimodal representations. Meanwhile, cross-modal transformer is designed to treat each modality as a target modality and optimize it using other modalities as source modalities, thereby learning the interactions between each pair of modalities. By jointly training unimodal and multimodal tasks, our model can focus on individual modality features while learning the interactions between modalities. Finally, to better capture temporal information and make predictions, we also added self-attention transformer as sequence models. Experimental results on the CMU-MOSI and CMU-MOSEI datasets demonstrate that JTUM outperforms current main methods.

Multi-task learning and mutual information maximization with crossmodal transformer for multimodal sentiment analysis

Multi-task learning and mutual information maximization with crossmodal transformer for multimodal sentiment analysis

Siamese capsule gorilla troops network-based multimodal sentiment analysis for car reviews

Siamese capsule gorilla troops network-based multimodal sentiment analysis for car reviews

Multimodal Aspect-Based Sentiment Analysis: A survey of tasks, methods, challenges and future directions

With the development of social media, users increasingly tend to express their sentiments (broadly including sentiment polarities, emotions and sarcasm, etc.) associated with fine-grained aspects (e.g., entities) in multimodal content (mostly encompassing images and texts). Consequently, automated recognition of sentiments within multimodal content over different aspects, namely Multimodal Aspect-Based Sentiment Analysis (MABSA), has recently become an emergent research area. This paper assesses the state-of-the-art methods in MABSA based on a systematic taxonomy over different subtasks of MABSA. It compiles advanced models for each task and offers a concise overview of popular datasets and evaluation standards. Finally, we discuss the limitations of current research and highlight promising future research directions.