Multimodal Sentiment Research Articles

MahaEmoSen: Towards Emotion-aware Multimodal Marathi Sentiment Analysis

With the advent of the Internet, social media platforms have witnessed an enormous increase in user-generated textual and visual content. Microblogs on platforms such as Twitter are extremely useful for comprehending how individuals feel about a specific issue through their posted texts, images, and videos. Owing to the plethora of content generated, it is necessary to derive an insight of its emotional and sentimental inclination. Individuals express themselves in a variety of languages and, lately, the number of people preferring native languages has been consistently increasing. Marathi language is predominantly spoken in the Indian state of Maharashtra. However, sentiment analysis in Marathi has rarely been addressed. In light of the above, we propose an emotion-aware multimodal Marathi sentiment analysis method (MahaEmoSen). Unlike the existing studies, we leverage emotions embedded in tweets besides assimilating the content-based information from the textual and visual modalities of social media posts to perform a sentiment classification. We mitigate the problem of small training sets by implementing data augmentation techniques. A word-level attention mechanism is applied on the textual modality for contextual inference and filtering out noisy words from tweets. Experimental outcomes on real-world social media datasets demonstrate that our proposed method outperforms the existing methods for Marathi sentiment analysis in resource-constrained circumstances.

Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis

Multi-modal Aspect-based Sentiment Analysis (MABSA) aims to forecast the polarity of sentiment concerning aspects within a given sentence based on the correlation between the sentence and its accompanying image. Comprehending multi-modal sentiment expression requires strong cross-modal alignment and fusion ability. Previous state-of-the-art (SOTA) models fail to explicitly align valuable visual clues with aspect and sentiment information in textual representations and overlook the utilization of syntactic dependency information in the accompanying text modality. We present CoolNet (Cross-modal Fine-grained Alignment and Fusion Network) to boost the performance of visual-language models in seamlessly integrating vision and language information. Specifically, CoolNet starts by transforming an image into a textual caption and a graph structure, then dynamically aligns the semantic and syntactic information from both the input sentence and the generated caption, as well as models the object-level visual features. Finally, a cross-modal transformer is employed to fuse and model the inter-modality dynamics.This network boasts advanced cross-modal fine-grained alignment and fusion capabilities. On standard benchmarks such as Twitter-2015 and Twitter-2017, CoolNet consistently outperforms state-of-the-art algorithm FITE with notable improvements in accuracy and Macro-F1 values. Specifically, we observe an improvement in accuracy and Macro-F1 values by 1.43% and 1.38% for Twitter-2015, and 0.74% and 0.88% for Twitter-2017, respectively, demonstrating the superiority of our CoolNet architecture.

Disentanglement Translation Network for multimodal sentiment analysis

Obtaining an effective joint representation has always been the goal for multimodal tasks. However, distributional gap inevitably exists due to the heterogeneous nature of different modalities, which poses burden on the fusion process and the learning of multimodal representation. The imbalance of modality dominance further aggravates this problem, where inferior modalities may contain much redundancy that introduces additional variations. To address the aforementioned issues, we propose a Disentanglement Translation Network (DTN) with Slack Reconstruction to capture desirable information properties, obtain a unified feature distribution and reduce redundancy. Specifically, the encoder–decoder-based disentanglement framework is adopted to decouple the unimodal representations into modality-common and modality-specific subspaces, which explores the cross-modal commonality and diversity, respectively. In the encoding stage, to narrow down the discrepancy, a two-stage translation is devised to incorporate with the disentanglement learning framework. The first stage targets at learning modality-invariant embedding for modality-common information with adversarial learning strategy, capturing the commonality shared across modalities. The second stage considers the modality-specific information that reveals diversity. To relieve the burden of multimodal fusion, we realize Specific-Common Distribution Matching to further unify the distribution of the desirable information. As for the decoding and reconstruction stage, we propose Slack Reconstruction to seek a balance between retaining discriminative information and reducing redundancy. Although the existing commonly-used reconstruction loss with strict constraint lowers the risk of information loss, it easily leads to the preservation of information redundancy. In contrast, Slack Reconstruction imposes a more relaxed constraint so that the redundancy is not forced to be retained, and simultaneously explores the inter-sample relationships. The proposed method aids multimodal fusion by learning the exact properties and obtaining a more uniform distribution for cross-modal data, and manages to reduce information redundancy to further ensure feature effectiveness. Extensive experiments on the task of multimodal sentiment analysis indicate the effectiveness of the proposed method. The codes are available at https://github.com/zengy268/DTN.

Understanding public opinions on Chinese short video platform by multimodal sentiment analysis using deep learning-based techniques

PurposeWith the rapid development of short videos in China, the public has become accustomed to using short videos to express their opinions. This paper aims to solve problems such as how to represent the features of different modalities and achieve effective cross-modal feature fusion when analyzing the multi-modal sentiment of Chinese short videos (CSVs).Design/methodology/approachThis paper aims to propose a sentiment analysis model MSCNN-CPL-CAFF using multi-scale convolutional neural network and cross attention fusion mechanism to analyze the CSVs. The audio-visual and textual data of CSVs themed on “COVID-19, catering industry” are collected from CSV platform Douyin first, and then a comparative analysis is conducted with advanced baseline models.FindingsThe sample number of the weak negative and neutral sentiment is the largest, and the sample number of the positive and weak positive sentiment is relatively small, accounting for only about 11% of the total samples. The MSCNN-CPL-CAFF model has achieved the Acc-2, Acc-3 and F1 score of 85.01%, 74.16 and 84.84%, respectively, which outperforms the highest value of baseline methods in accuracy and achieves competitive computation speed.Practical implicationsThis research offers some implications regarding the impact of COVID-19 on catering industry in China by focusing on multi-modal sentiment of CSVs. The methodology can be utilized to analyze the opinions of the general public on social media platform and to categorize them accordingly.Originality/valueThis paper presents a novel deep-learning multimodal sentiment analysis model, which provides a new perspective for public opinion research on the short video platform.

Hierarchical Fusion Network with Enhanced Knowledge and Contrastive Learning for Multimodal Aspect-Based Sentiment Analysis on Social Media.

Aspect-based sentiment analysis (ABSA) is a task of fine-grained sentiment analysis that aims to determine the sentiment of a given target. With the increased prevalence of smart devices and social media, diverse data modalities have become more abundant. This fuels interest in multimodal ABSA (MABSA). However, most existing methods for MABSA prioritize analyzing the relationship between aspect-text and aspect-image, overlooking the semantic gap between text and image representations. Moreover, they neglect the rich information in external knowledge, e.g., image captions. To address these limitations, in this paper, we propose a novel hierarchical framework for MABSA, known as HF-EKCL, which also offers perspectives on sensor development within the context of sentiment analysis. Specifically, we generate captions for images to supplement the textual and visual features. The multi-head cross-attention mechanism and graph attention neural network are utilized to capture the interactions between modalities. This enables the construction of multi-level aspect fusion features that incorporate element-level and structure-level information. Furthermore, for this paper, we integrated modality-based and label-based contrastive learning methods into our framework, making the model learn shared features that are relevant to the sentiment of corresponding words in multimodal data. The results, based on two Twitter datasets, demonstrate the effectiveness of our proposed model.

Context-Dependent Multimodal Sentiment Analysis Based on a Complex Attention Mechanism

Multimodal sentiment analysis aims to understand people’s attitudes and opinions from different data forms. Traditional modality fusion methods for multimodal sentiment analysis con-catenate or multiply various modalities without fully utilizing context information and the correlation between modalities. To solve this problem, this article provides a new model based on a multimodal sentiment analysis framework based on a recurrent neural network with a complex attention mechanism. First, after the raw data is preprocessed, the numerical feature representation is obtained using feature extraction. Next, the numerical features are input into the recurrent neural network, and the output results are multimodally fused using a complex attention mechanism layer. The objective of the complex attention mechanism is to leverage enhanced non-linearity to more effectively capture the inter-modal correlations, thereby improving the performance of multimodal sentiment analysis. Finally, the processed results are fed into the classification layer and the sentiment output is obtained using the classification layer. This process can effectively capture the semantic information and contextual relationship of the input sequence and fuse different pieces of modal information. Our model was tested on the CMU-MOSEI datasets, achieving an accuracy of 82.04%.

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

In the real world, multimodal sentiment analysis (MSA) enables the capture and analysis of sentiments by fusing multimodal information, thereby enhancing the understanding of real-world environments. The key challenges lie in handling the noise in the acquired data and achieving effective multimodal fusion. When processing the noise in data, existing methods utilize the combination of multimodal features to mitigate errors in sentiment word recognition caused by the performance limitations of automatic speech recognition (ASR) models. However, there still remains the problem of how to more efficiently utilize and combine different modalities to address the data noise. In multimodal fusion, most existing fusion methods have limited adaptability to the feature differences between modalities, making it difficult to capture the potential complex nonlinear interactions that may exist between modalities. To overcome the aforementioned issues, this paper proposes a new framework named multimodal-word-refinement and cross-modal-hierarchy (MWRCMH) fusion. Specifically, we utilized a multimodal word correction module to reduce sentiment word recognition errors caused by ASR. During multimodal fusion, we designed a cross-modal hierarchical fusion module that employed cross-modal attention mechanisms to fuse features between pairs of modalities, resulting in fused bimodal-feature information. Then, the obtained bimodal information and the unimodal information were fused through the nonlinear layer to obtain the final multimodal sentiment feature information. Experimental results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets demonstrated that the proposed approach outperformed other comparative methods, achieving Has0-F1 scores of 76.43%, 80.15%, and 81.93%, respectively. Our approach exhibited better performance, as compared to multiple baselines.

A structural model of teachers’ teaching competencies based on multimodal affective features of support vector machines

Abstract Based on big data technology, this paper first proposes to study and analyze the structural model of teachers’ teaching ability based on multimodal sentiment features of support vector machines. Then the Mel inverse spectral coefficients, wavelet packet coefficients, Fourier coefficient features, dynamic features, and global features are extracted respectively, and the features are dimensionalized using linear support vector machine and Lagrangian function, and the dimensionalized feature parameters are sent to the classifier for emotion recognition. Finally, the structural model evaluation index system of teachers’ teaching ability is constructed, and the current teaching ability development of college teachers in a province is studied and analyzed from two perspectives of teachers’ self-assessment and students’ evaluation based on multimodal emotional features of support vector machine. The results showed that in terms of teaching ability dimensions, teachers had the highest expectation of their teaching ability (4.38) in each dimension, followed in order by teachers’ self-evaluation of teaching ability (4.15), students’ expectation (4.05), and students’ evaluation of teachers’ teaching ability (3.56) was the lowest. This study reveals that teachers’ perceptions of the teaching competencies they should have are biased, and their awareness of self-development is insufficient from the emotional characteristics.

Modality translation-based multimodal sentiment analysis under uncertain missing modalities

Multimodal sentiment analysis (MSA) with uncertain missing modalities poses a new challenge in sentiment analysis. To address this problem, efficient MSA models that consider missing modalities have been proposed. However, existing studies have only adopted the concatenation operation for feature fusion while ignoring the deep interactions between different modalities. Moreover, existing studies have failed to take advantage of the text modality, which can achieve better accuracy in sentiment analysis. To tackle the above-mentioned issues, we propose a modality translation-based MSA model (MTMSA), which is robust to uncertain missing modalities. First, for multimodal data (text, visual, and audio) with uncertain missing data, the visual and audio are translated to the text modality with a modality translation module, and then the translated visual modality, translated audio, and encoded text are fused into missing joint features (MJFs). Next, the MJFs are encoded by the transformer encoder module under the supervision of a pre-trained model (transformer-based modality translation network, TMTN), thus making the transformer encoder module produce joint features of uncertain missing modalities approximating those of complete modalities. The encoded MJFs are input into the transformer decoder module to learn the long-term dependencies between different modalities. Finally, sentiment classification is performed based on the outputs of the transformer encoder module. Extensive experiments were conducted on two popular benchmark datasets (CMU-MOSI and IEMOCAP), with the experimental results demonstrating that MTMSA outperforms eight representative baseline models.

Integrating color cues to improve multimodal sentiment analysis in social media

Multimodal sentiment analysis is an emerging and critical area of research, particularly in the context of social media where users express their emotions through both text and images. However, existing methods that learn common semantic features between visual and textual modalities for multimodal sentiment analysis often overlook the importance of color information, which plays a crucial role in sentiment expression, according to psychology and art theory. This paper proposes a novel model, named ICCI, which aims to enhance multimodal sentiment analysis in social media by integrating color cues, in order to address the limitation in existing approaches. The proposed model leverages both the semantic information from image-text pairs and the color cues from images to improve the accuracy of sentiment analysis. The model comprises a feature extraction module that extracts semantic features from both images and text, as well as color features from images. Furthermore, the feature interaction module employs a cross-attention mechanism to enable the interaction of information between semantic features and color features. Finally, the label prediction module integrates all attention features to enhance multimodal sentiment analysis. The proposed model was evaluated through experiments on two widely used benchmark datasets, namely MVSA-Single and MVSA-Multiple datasets, demonstrating its effectiveness in outperforming existing methods. On MVSA-Single, ICCI achieved an accuracy of 79.33%. Similarly, on MVSA-Multiple, ICCI achieved an accuracy of 73.29%. The results underscore the importance of integrating color information for more accurate sentiment analysis in social media.

SKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis

Multimodal sentiment analysis is an active research field that aims to recognize the user’s sentiment information from multimodal data. The primary challenge in this field is to develop a high-quality fusion framework that effectively addresses the heterogeneity among different modalities. However, prior research has primarily concentrated on intermodal interactions while neglecting the semantic sentiment information conveyed by words in the text modality. In this paper, we propose the Sentiment Knowledge Enhanced Attention Fusion Network (SKEAFN), a novel end-to-end fusion network that enhances multimodal fusion by incorporating additional sentiment knowledge representations from an external knowledge base. Firstly, we construct an external knowledge enhancement module to acquire additional representations for the text modality. Then, we design a text-guided interaction module that facilitates the interaction between text and the visual/acoustic modality. Finally, we propose a feature-wised attention fusion module that achieves multimodal fusion by dynamically adjusting the weights of the additional and each modality’s representations. We evaluate our method on three challenging multimodal sentiment analysis datasets: CMU-MOSI, CMU-MOSEI, and Twitter2019. The experiment results demonstrate that our model significantly outperforms the state-of-the-art models. The source code is publicly available at https://github.com/doubibobo/SKEAFN.

Joint multimodal sentiment analysis based on information relevance

Joint multimodal sentiment analysis based on information relevance

MHaDiG: A Multilingual Humor-aided Multiparty Dialogue Generation in multimodal conversational setting

Building a multimodal humor-aided dialogue generation system in a multilingual setting is a very challenging task and it could even be more challenging in the Hindi language because of its morphological richness and free word order. Existing works only solved the problem of unimodal humorous dialogue generation, not humor-aided, in a monolingual setting. However, due to the tremendous growth in multimodal and multilingual content, there is a great demand to build multimodal humor-aided systems that support multilingual information access. To this end, in this paper, we propose a deep learning-based multimodal sentiment and emotion-aware humor-aided multiparty dialogue generation in multilingual (Hindi and English) setting. We evaluate our proposed approach (MHaDiG) on the recently released Sentiment, Humor, and Emotion-aware Multilingual Multimodal Multiparty Dataset (SHEMuD). Experimental results show the efficacy of MHaDiG and show the effect of multilingualism over monolingualism in the multimodal conversational setting.

Generating paraphrase sentences for multimodal entity-category-sentiment triple extraction

Multimodal entity-based sentiment analysis (MEBSA) is an emerging task in sentiment analysis that aims to identify three key elements (entity, entity category, and sentiment polarity) from a pair of sentence and image. However, most existing studies have primarily focused on one or two MEBSA subtasks, ignoring the fact that these subtasks are closely related with one another. Moreover, previous studies focused on the detection of coarse-grained entity categories, which failed to provide sufficient information to disambiguate entities. To address these two issues, we introduced a new task called multimodal entity-category-sentiment triple extraction (MECSTE) to extract entities, their corresponding fine-grained entity categories, and sentiment polarities simultaneously. We constructed two datasets for this task based on two existing Twitter corpora. Moreover, we developed a generative multimodal approach based on a pre-trained sequence-to-sequence model that formulates the MECSTE task as a paraphrase generation problem by linearising all the entity-category-sentiment triples into a natural language sentence. Extensive experiments on the annotated Twitter datasets demonstrate the superiority of the proposed method.

A comprehensive survey on deep learning-based approaches for multimodal sentiment analysis

A comprehensive survey on deep learning-based approaches for multimodal sentiment analysis

Moving From Narrative to Interactive Multi-Modal Sentiment Analysis: A Survey

A growing number of individuals are expressing their opinions and engaging in interactive communication with others through various modalities, including natural language (text), facial gestures (vision), acoustic behaviors (audio), and more. Within the realms of natural language processing (NLP) and artificial intelligence (AI), multi-modal sentiment analysis has consistently remained a fundamental research area. Building upon recent advancements, this survey aims to provide researchers with a comprehensive overview of the state-of-the-art techniques in multi-modal sentiment analysis, specifically focusing on various sentiment interaction tasks. It is worth noting that the existing literature on multi-modal sentiment analysis has rarely delved into the realm of sentiment interaction. This survey presents a novel perspective by outlining the progression of multi-modal sentiment analysis from narrative sentiment to interactive sentiment. Furthermore, it discusses the research background, problem definition, and various approaches in multi-modal sentiment analysis. Additionally, this survey provides insights into the development of multi-modal sarcasm recognition, emphasizing the shift from narrativity to interactivity. Lastly, we summarize the current scientific challenges related to interaction modeling and highlight future development trends in the field.

Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges

Sentiment analysis has come long way since it was introduced as a natural language processing task nearly 20 years ago. Sentiment analysis aims to extract the underlying attitudes and opinions toward an entity. It has become a powerful tool used by governments, businesses, medicine, marketing, and others. The traditional sentiment analysis model focuses mainly on text content. However, technological advances have allowed people to express their opinions and feelings through audio, image and video channels. As a result, sentiment analysis is shifting from unimodality to multimodality. Multimodal sentiment analysis brings new opportunities with the rapid increase of sentiment analysis as complementary data streams enable improved and deeper sentiment detection which goes beyond text-based analysis. Audio and video channels are included in multimodal sentiment analysis in terms of broadness. People have been working on different approaches to improve sentiment analysis system performance by employing complex deep neural architectures. Recently, sentiment analysis has achieved significant success using the transformer-based model. This paper presents a comprehensive study of different sentiment analysis approaches, applications, challenges, and resources then concludes that it holds tremendous potential. The primary motivation of this survey is to highlight changing trends in the unimodality to multimodality for solving sentiment analysis tasks.

Learning from the global view: Supervised contrastive learning of multimodal representation

The development of technology enables the availability of abundant multimodal data, which can be utilized in many representation learning tasks. However, most methods ignore the rich modality correlation information stored in each multimodal object and fail to fully exploit the potential of multimodal data. To address the aforementioned issue, cross-modal contrastive learning methods are proposed to learn the similarity score of each modality pair in a self-/weakly-supervised manner and improve the model robustness. Though effective, contrastive learning based on unimodal representations might be, in some cases, inaccurate as unimodal representations fail to reveal the global information of multimodal objects. To this end, we propose a contrastive learning pipeline based on multimodal representations to learn from the global view, and devise multiple techniques to generate negative and positive samples for each anchor. To generate positive samples, we apply the mix-up operation to mix two multimodal representations of different objects that have the maximal label similarity. Moreover, we devise a permutation-invariant fusion mechanism to define the positive samples by permuting the input order of modalities for fusion and sampling various contrastive fusion networks. In this way, we force the multimodal representation to be invariant regarding the order of modalities and the structures of fusion networks, so that the model can capture high-level semantic information of multimodal objects. To define negative samples, for each modality, we randomly replace the unimodal representation with that from another dissimilar object when synthesizing the multimodal representation. By this means, the model is led to capture the high-level concurrence information and correspondence relationship between modalities within each object. We also directly define the multimodal representation from another object as a negative sample, where the chosen object shares the minimal label similarity with the anchor. The label information is leveraged in the proposed framework to learn a more discriminative multimodal embedding space for downstream tasks. Extensive experiments demonstrate that our method outperforms previous state-of-the-art baselines on the tasks of multimodal sentiment analysis and humor detection.

Retraction Note: ICDN: integrating consistency and difference networks by transformer for multimodal sentiment analysis

Retraction Note: ICDN: integrating consistency and difference networks by transformer for multimodal sentiment analysis

Dynamic Weighted Multitask Learning and Contrastive Learning for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) has attracted more and more attention in recent years. This paper focuses on the representation learning of multimodal data to reach higher prediction results. We propose a model to assist in learning modality representations with multitask learning and contrastive learning. In addition, our approach obtains dynamic weights by considering the homoscedastic uncertainty of each task in multitask learning. Specially, we design two groups of subtasks, which predict the sentiment polarity of unimodal and bimodal representations, to assist in learning representation through a hard parameter-sharing mechanism in the upstream neural network. A loss weight is learned according to the homoscedastic uncertainty of each task. Moreover, a training strategy based on contrastive learning is designed to balance the inconsistency between training and inference caused by the randomness of the dropout layer. This method minimizes the MSE between two submodels. Experimental results on the MOSI and MOSEI datasets show our method achieves better performance than the current state-of-the-art methods by comprehensively considering the intramodality and intermodality interaction information.