Multimodal Sentiment Analysis Research Articles

MST-ARGCN: modality-squeeze transformer with attentional recurrent graph capsule network for multimodal sentiment analysis

MST-ARGCN: modality-squeeze transformer with attentional recurrent graph capsule network for multimodal sentiment analysis

Aspect-aware semantic feature enhanced networks for multimodal aspect-based sentiment analysis

Aspect-aware semantic feature enhanced networks for multimodal aspect-based sentiment analysis

DRSS: a multimodal sentiment analysis approach based on dual representation and self-supervised learning strategy

DRSS: a multimodal sentiment analysis approach based on dual representation and self-supervised learning strategy

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments

Multimodal sentiment analysis models can determine users’ sentiments by utilizing rich information from various sources (e.g., textual, visual, and audio). However, there are two key challenges when deploying the model in real-world environments: (1) the limitations of relying on the performance of automatic speech recognition (ASR) models can lead to errors in recognizing sentiment words, which may mislead the sentiment analysis of the textual modality, and (2) variations in information density across modalities complicate the development of a high-quality fusion framework. To address these challenges, this paper proposes a novel Multimodal Sentiment Word Optimization Module and a heterogeneous hierarchical fusion (MSWOHHF) framework. Specifically, the proposed Multimodal Sentiment Word Optimization Module optimizes the sentiment words extracted from the textual modality by the ASR model, thereby reducing sentiment word recognition errors. In the multimodal fusion phase, a heterogeneous hierarchical fusion network architecture is introduced, which first utilizes a Transformer Aggregation Module to fuse the visual and audio modalities, enhancing the high-level semantic features of each modality. A Cross-Attention Fusion Module then integrates the textual modality with the audiovisual fusion. Next, a Feature-Based Attention Fusion Module is proposed that enables fusion by dynamically tuning the weights of both the combined and unimodal representations. It then predicts sentiment polarity using a nonlinear neural network. Finally, the experimental results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets show that the MSWOHHF outperforms several baselines, demonstrating better performance.

Multi-granularity visual-textual jointly modeling for aspect-level multimodal sentiment analysis

Multi-granularity visual-textual jointly modeling for aspect-level multimodal sentiment analysis

Reconstructing representations using diffusion models for multimodal sentiment analysis through reading comprehension

The primary challenge in multimodal sentiment analysis (MSA), which utilizes textual, audio, and visual information to analyze speakers' emotions, lies in constructing representation vectors that incorporate both unimodal semantic and multimodal interaction information. While existing research has extensively focused on multimodal fusion strategies, there remains insufficient exploration of the intrinsic potential within concurrently enhancing unimodal and multimodal representations. To address this gap, we introduce two additional steps to traditional three-step MSA: text modality enhancement through the machine reading comprehension (MRC) framework and multimodal representation reconstruction via proposing the diverse diffusion denoising autoencoder (D3AE) module. The MRC queries are integrated to locate sentiment-related prior knowledge, thereby deepening textual semantic understanding from a pretrained language model. Meanwhile, D3AE employs a single-step denoising strategy along with diffusion models across multiple time intervals, enabling efficient reconstruction and enhancement of multimodal representations. Extensive experiments conducted on two benchmark datasets, CMU-MOSI and CMU-MOSEI, validate that our model, MRC-D3AE, achieves state-of-the-art performance. The superiority of our model over existing baselines is primarily attributed to integrating MRC for enhancing text modality and D3AE for reconstructing multimodal representations.

Hierarchical graph contrastive learning framework based on quantum neural networks for sentiment analysis

Existing multi-modal sentiment analysis (MSA) methods typically achieve interaction by connecting different layers or designing special structures, but rarely consider the synergistic effects among data. Moreover, most sentiment analysis research tends to focus solely on single sentiment polarity analysis, without considering the intensity and directional attributes of emotions. Addressing these issues, we propose a framework called Hierarchical Graph Contrastive Learning based on Quantum Neural Network (HGCL-QNN) to remedy these shortcomings. Specifically, a graph structure is established within and between modalities. In the quantum fuzzy neural network module, fuzzy quantum encoding is implemented by using complex-valued, then quantum superposition and entanglement are utilized to consider the intensity and directional attributes of emotions while analyzing emotional polarity. In the quantum multi-modal fusion neural network module, methods such as amplitude encoding and quantum entanglement are employed to further integrate information from different modalities, thereby enhancing the model's power to express emotional information. To enhance the model's understanding of fine-grained and global features, and to better align and integrate features from different modalities, hierarchical graph contrastive learning is employed on different levels. The experimental results demonstrate that HGCL-QNN outperforms the existing baseline methods on MOSI and MOSEI datasets, achieving significant efficacy improvements.

AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis

Multimodal sentiment analysis (MSA) involves interpreting sentiment using various sensory data modalities. Traditional MSA models often overlook causality between modalities, resulting in spurious correlations and ineffective cross-modal attention. To address these limitations, we propose the Attention-based Causality-Aware Fusion (AtCAF) network from a causal perspective. To capture a causality-aware representation of text, we introduce the Causality-Aware Text Debiasing Module (CATDM) utilizing the front-door adjustment. Furthermore, we employ the Counterfactual Cross-modal Attention (CCoAt) module integrate causal information in modal fusion, thereby enhancing the quality of aggregation by incorporating more causality-aware cues. AtCAF achieves state-of-the-art performance across three datasets, demonstrating significant improvements in both standard and Out-Of-Distribution (OOD) settings. Specifically, AtCAF outperforms existing models with a 1.5% improvement in ACC-2 on the CMU-MOSI dataset, a 0.95% increase in ACC-7 on the CMU-MOSEI dataset under normal conditions, and a 1.47% enhancement under OOD conditions. CATDM improves category cohesion in feature space, while CCoAt accurately classifies ambiguous samples through context filtering. Overall, AtCAF offers a robust solution for social media sentiment analysis, delivering reliable insights by effectively addressing data imbalance. The code is available at https://github.com/TheShy-Dream/AtCAF.

Multimodal dual perception fusion framework for multimodal affective analysis

The misuse of social platforms and the difficulty in regulating post contents have culminated in a surge of negative sentiments, sarcasms, and the rampant spread of fake news. In response, Multimodal sentiment analysis, sarcasm detection and fake news detection based on image and text have attracted considerable attention recently. Due to that these areas share semantic and sentiment features and confront related fusion challenges in deciphering complex human expressions across different modalities, integrating these multimodal classification tasks that share commonalities across different scenarios into a unified framework is expected to simplify research in sentiment analysis, and enhance the effectiveness of classification tasks involving both semantic and sentiment modeling. Therefore, we consider integral components of a broader spectrum of research known as multimodal affective analysis towards semantics and sentiment, and propose a novel multimodal dual perception fusion framework (MDPF). Specifically, MDPF contains three core procedures: (1) Generating bootstrapping language-image Knowledge to enrich origin modality space, and utilizing cross-modal contrastive learning for aligning text and image modalities to understand underlying semantics and interactions. (2) Designing dynamic connective mechanism to adaptively match image-text pairs and jointly employing gaussian-weighted distribution to intensify semantic sequences. (3) Constructing a cross-modal graph to preserve the structured information of both image and text data and share information between modalities, while introducing sentiment knowledge to refine the edge weights of the graph to capture cross-modal sentiment interaction. We evaluate MDPF on three publicly available datasets across three tasks, and the empirical results demonstrate the superiority of our proposed model.

Hybrid Quantum-Classical Neural Network for Multimodal Multitask Sarcasm, Emotion, and Sentiment Analysis

Hybrid Quantum-Classical Neural Network for Multimodal Multitask Sarcasm, Emotion, and Sentiment Analysis

A Multimodal Sentiment Analysis Method Based on Fuzzy Attention Fusion

A Multimodal Sentiment Analysis Method Based on Fuzzy Attention Fusion

A novel hybrid deep learning IChOA-CNN-LSTM model for modality-enriched and multilingual emotion recognition in social media.

In the rapidly evolving field of artificial intelligence, the importance of multimodal sentiment analysis has never been more evident, especially amid the ongoing COVID-19 pandemic. Our research addresses the critical need to understand public sentiment across various dimensions of this crisis by integrating data from multiple modalities, such as text, images, audio, and videos sourced from platforms like Twitter. Conventional methods, which primarily focus on text analysis, often fall short in capturing the nuanced intricacies of emotional states, necessitating a more comprehensive approach. To tackle this challenge, our proposed framework introduces a novel hybrid model, IChOA-CNN-LSTM, which leverages Convolutional Neural Networks (CNNs) for precise image feature extraction, Long Short-Term Memory (LSTM) networks for sequential data analysis, and an Improved Chimp Optimization Algorithm for effective feature fusion. Remarkably, our model achieves an impressive accuracy rate of 97.8%, outperforming existing approaches in the field. Additionally, by integrating the GeoCoV19 dataset, we facilitate a comprehensive analysis that spans linguistic and geographical boundaries, enriching our understanding of global pandemic discourse and providing critical insights for informed decision-making in public health crises. Through this holistic approach and innovative techniques, our research significantly advances multimodal sentiment analysis, offering a robust framework for deciphering the complex interplay of emotions during unprecedented global challenges like the COVID-19 pandemic.

A deep spatiotemporal interaction network for multimodal sentimental analysis and emotion recognition

One of the challenges of sentiment analysis and emotion recognition is how to effectively fuse the multimodal inputs. The transformer-based models have achieved great success in applications of multimodal sentiment analysis and emotion recognition recently. However, the transformer-based model often neglects the coherence of human emotion due to its parallel structure. Additionally, a low-rank bottleneck created by multi- attention-head causes an inadequate fitting ability of models. To tackle these issues, a Deep Spatiotemporal Interaction Network (DSIN) is proposed in this study. It consists of two main components, i.e., a cross-modal transformer with a cross-talking attention module and a hierarchically temporal fusion module, where the cross-modal transformer is used to model the spatial interactions between different modalities and the hierarchically temporal fusion network is utilized to model the temporal coherence of emotion. Therefore, the DSIN can model the spatiotemporal interactions of multimodal inputs by incorporating the time-dependency into the parallel structure of transformer and decrease the redundancy of embedded features by implanting their spatiotemporal interactions into a hybrid memory network in a hierarchical manner. The experimental results on two benchmark datasets indicate that DSIN achieves superior performance compared with the state-of-the-art models, and some useful insights are derived from the results.

FMCF: Few-shot Multimodal aspect-based sentiment analysis framework based on Contrastive Finetuning

FMCF: Few-shot Multimodal aspect-based sentiment analysis framework based on Contrastive Finetuning

Multimodal PEAR Chain-of-Thought Reasoning for Multimodal Sentiment Analysis

Multimodal sentiment analysis aims to predict sentiments from multimodal signals such as audio, video, and text. Existing methods often rely on Pre-trained Language Models (PLMs) to extract semantic information from textual data, lacking an in-depth understanding of the logical relationships within the text modality . This article introduces the Multimodal PEAR (Preliminaries, quEstion, Answer, Reason) Chain-of-Thought (MM-PEAR-CoT) reasoning for multimodal sentiment analysis. Inspired by the human thought process when solving complex problems, the PEAR CoT prompt is first proposed to induce Large Language Models (LLMs) to generate text-based reasoning processes and zero-shot sentiment prediction results. However, text-based CoT reasoning is not always reliable and might contain irrational steps due to the hallucinations of LLMs . To address this, we further design the Cross-Modal Filtering and Fusion (CMFF) module. The filtering submodule utilizes audio and visual modalities to suppress irrational steps in the CoT, while the fusion submodule integrates high-level reasoning information and cross-modal complementary information in the process of semantic representation learning. Experimental results on two multimodal sentiment analysis benchmark datasets show that high-level reasoning information can help learn discriminative text representation, and cross-modal complementary information can avoid misleading by unreasonable steps in the CoT. MM-PEAR-CoT achieves the best results on both datasets, with improvements of 2.2% and 1.7% in binary classification accuracy on the CMU-MOSI and CMU-MOSEI datasets, respectively. To the best of our knowledge, this is the first study to apply CoT reasoning to multimodal sentiment analysis.

A Multimodal Sentiment Analysis Model for Graphic Texts Based on Deep Feature Interaction Networks

Due to the widespread adoption of social networks, image-text comments have become a prevalent mode of emotional expression compared to traditional text descriptions. However, there are currently two major challenges. The first is the question of how to extract rich representations effectively from both text and images, and the second is the question of how to extract cross-modal shared emotion features. This study proposes a multimodal sentiment analysis method based on a deep feature interaction network (DFINet). It leverages word-to-word graphs and deep attention interaction networks (DAIN) to learn text representations effectively from multiple subspaces. Additionally, it introduces a cross-modal attention interaction network to extract cross-modal shared emotion features efficiently. This approach helps alleviate the difficulties associated with acquiring image-text features and representing cross-modal shared emotion features. Experimental results on the Yelp dataset demonstrate the effectiveness of the DFINet method.

Visual audio and textual triplet fusion network for multi-modal sentiment analysis

Visual audio and textual triplet fusion network for multi-modal sentiment analysis

A short video sentiment analysis model based on multimodal feature fusion

With the development of the internet, the number of short video platform users has increased quickly. People's social entertainment mode has gradually changed from text to short video, generating many multimodal data. Therefore, traditional single-modal sentiment analysis can no longer fully adapt to multimodal data. To address this issue, this study proposes a short video sentiment analysis model based on multimodal feature fusion. This model analyzes the text, speech, and visual content in the video. Meanwhile, the information of the three modalities is integrated through a multi-head attention mechanism to analyze and classify emotions. The experimental results showed that when the training set size was 500, the recognition accuracy of the multimodal sentiment analysis model based on modal contribution recognition and multi-task learning was 0.96. The F1 score was 98, and the average absolute error value was 0.21. When the validation set size was 400, the recognition time of the multimodal sentiment analysis model based on modal contribution recognition and multi-task learning was 2.1 s. When the iterations were 60, the recognition time of the multimodal sentiment analysis model based on modal contribution recognition and multi-task learning was 0.9 s. The experimental results show that the proposed multimodal sentiment analysis model based on modal contribution recognition and multi-task learning has good model performance and can accurately identify emotions in short videos.

A dual-channel multimodal sentiment analysis framework based on three-way decision

In the exploration of human–computer interaction, accurately identifying and understanding the user’s sentimental state helps enhance the user experience. Multimodal sentiment analysis aims to identify sentiments through various signals. However, inconsistencies in sentiments between text and visual–audio signals pose a challenge, as models often struggle to capture these differences adequately. To alleviate this issue, we propose a dual-channel sentiment analysis framework based on three-way decision. This framework includes a multimodal model, a visual–audio model, and a three-way decision module. Specifically, the multimodal model and visual–audio model generate predictions, and the three-way decision module selects an appropriate prediction. We improve the binary cross-entropy loss function for the visual–audio model to integrate the three-way decision module better and improve sentiment prediction accuracy. Additionally, we introduce an adversarial training task to ensure differences in predictions between the two models, thereby enhancing the influence of the visual–audio model. Experimental results on four public datasets demonstrate that our framework significantly improves the accuracy of sentiment recognition and reduces the average decision time.

Frame-level nonverbal feature enhancement based sentiment analysis

Multimodal Sentiment Analysis (MSA) comprehensively utilizing data from multiple modalities to obtain more accurate sentiment attribute, has important applications in other fields, such as social media analysis, user experience evaluation and medical health, etc. It is worth noting that previous studies have paid little attention to the inconsistency of the initial representation granularity between verbal (textual) and nonverbal (acoustic and visual) modalities. As a result, the imbalanced emotional information between them complicates the interaction process, and ultimately affects the model’s performance. To solve this problem, this paper proposes a Frame-level Nonverbal feature Enhancement Network (FNENet) to improve performance on MSA by reducing the gap and integrating asynchronous affective information between modalities. Specifically, Vector Quantization (VQ) is applied to nonverbal modalities to reduce the granularity differences and improve the performance of the model. Additionally, nonverbal information is integrated through the Sequence Fusion mechanism (SF) into a pre-trained language model to enhance the textual representation, which benefits the word-level semantic expression according to the asynchronous affective cues preserved in unaligned frame-level nonverbal features. Extensive experiments on three benchmark datasets demonstrate that FNENet significantly outperforms baseline methods. It indicates that our model has potential application on MSA.