Multimodal Sentiment Research Articles

Correlations Between Positive or Negative Utterances and Basic Acoustic Features of Voice: a Preliminary Analysis

The major aim of this paper is to establish possible correlations between continuous sentiment scores and four basic acoustic characteristics of voice. In order to achieve this objective, the text of “A Christmas Carol” by Charles Dickens was tokenized at the sentence level. Next, each of the resulting text units was assessed in terms of sentiment polarity and aligned with the corresponding fragment in an audiobook. The results indicate weak but statistically significant correlations between sentiment scores and three acoustic features: the mean F0, the standard deviation of F0 and the mean intensity. These findings may be useful in selecting optimal acoustic features for model training in multimodal sentiment analysis. Also, they are essential from a linguistic point of view and could be applied in studies on such language phenomena as irony.

Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects

Multimodal sentiment analysis benefits various applications, such as human–computer interaction and recommendation systems. It infers the user-based sentiments using modals, enriched by neutrality and ambivalence empirical. Although researchers affirm the association between cognitive cues and emotional manifestations, most current multimodal sentiment analytical approaches disregard user-specific aspects. To tackle this issue, we devise a novel method to perform multimodal sentiment prediction using cognitive cues. Our cognitive-aware framework constructs an adaptive tree by hierarchically dividing users and then trains the Long short-term memory submodels, utilizing an attention-based fusion to transfer cognitive-oriented knowledge within the tree. Subsequently, the framework consumes the conclusive agglomerative knowledge from the adaptive tree to predict final sentiments. We also develop a dynamic dropout method to regularize data of pertinent groups during hierarchical training. The empirical results on real-world datasets determine that our proposed model for sentiment prediction can surpass trending baselines. Moreover, compared to other ensemble approaches, the proposed transfer-based algorithm can better utilize the latent cues and foster the prediction outcomes, benefitting from multiple categorization strategies. Based on the given extrinsic and intrinsic analysis results, we note that compared to other theoretical-based techniques, the proposed hierarchical categorization model can better group the users within the adaptive tree.

PS-Mixer: A Polar-Vector and Strength-Vector Mixer Model for Multimodal Sentiment Analysis

Multimodal sentiment analysis aims to judge the sentiment of multimodal data uploaded by the Internet users on various social media platforms. On one hand, existing studies focus on the fusion mechanism of multimodal data such as text, audio and visual, but ignore the similarity of text and audio, text and visual, and the heterogeneity of audio and visual, resulting in deviation of sentiment analysis. On the other hand, multimodal data brings noise irrelevant to sentiment analysis, which affects the effectness of fusion. In this paper, we propose a Polar-Vector and Strength-Vector mixer model called PS-Mixer, which is based on MLP-Mixer, to achieve better communication between different modal data for multimodal sentiment analysis. Specifically, we design a Polar-Vector (PV) and a Strength-Vector (SV) for judging the polar and strength of sentiment separately. PV is obtained from the communication of text and visual features to decide the sentiment that is positive, negative, or neutral sentiment. SV is gained from the communication between the text and audio features to analyze the sentiment strength in the range of 0 to 3. Furthermore, we devise an MLP-Communication module (MLP-C) composed of several fully connected layers and activation functions to make the different modal features fully interact in both the horizontal and the vertical directions, which is a novel attempt to use MLP for multimodal information communication. Finally, we mix PV and SV to obtain a fusion vector to judge the sentiment state. The proposed PS-Mixer is tested on two publicly available datasets, CMU-MOSEI and CMU-MOSI, which achieves the state-of-the-art (SOTA) performance on CMU-MOSEI compared with baseline methods. The codes are available at: https://github.com/metaphysicser/PS-Mixer.

TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis

Multimodal sentiment analysis (MSA), which aims to recognize sentiment expressed by speakers in videos utilizing textual, visual and acoustic cues, has attracted extensive research attention in recent years. However, textual, visual and acoustic modalities often contribute differently to sentiment analysis. In general, text contains more intuitive sentiment-related information and outperforms nonlinguistic modalities in MSA. Seeking a strategy to take advantage of this property to obtain a fusion representation containing more sentiment-related information and simultaneously preserving inter- and intra-modality relationships becomes a significant challenge. To this end, we propose a novel method named Text Enhanced Transformer Fusion Network (TETFN), which learns text-oriented pairwise cross-modal mappings for obtaining effective unified multimodal representations. In particular, it incorporates textual information in learning sentiment-related nonlinguistic representations through text-based multi-head attention. In addition to preserving consistency information by cross-modal mappings, it also retains the differentiated information among modalities through unimodal label prediction. Furthermore, the vision pre-trained model Vision-Transformer is utilized to extract visual features from the original videos to preserve both global and local information of a human face. Extensive experiments on benchmark datasets CMU-MOSI and CMU-MOSEI demonstrate the superior performance of the proposed TETFN over state-of-the-art methods.

Joint multimodal sentiment analysis based on information relevance

Social media users are increasingly turning to express opinions with both images and text, while the visual content and text description may cover some conflicting information diverse from each other. Information relevance refers to the matching degree between cross-modal features at the emotional semantic level, which is not systematically studied. In order to exploit the discriminative features and the internal correlation among different modalities, the mid-level representation extracted by a visual sentiment concept classifier is used to determine information relevance, with the integration of other features, including attended textual and visual features. Then grid search is applied to tune weighting coefficients of the decision fusion scheme, followed by a multimodal adaptive method for joint sentiment analysis based on image-text relevance. The superiority of our architecture approach is demonstrated experimentally by comparing it with several state-of-the-art baselines, such as the vision-aware language modeling approach and contrastive learning-based model. The results indicate that fused multiple features lead to more precise classification than unimodal ones, while the contributions of each single modality differ obviously in emotional expression. Besides, the performance of every involved model varies markedly per dataset of different content correlation, which proves that it is of extensive theoretical significance and application prospect to introduce the image-text relevance classifier into a multimodal task.

StyleBERT: Text-audio sentiment analysis with Bi-directional Style Enhancement

Recent multimodal sentiment analysis works focus on establishing sophisticated fusion strategies for better performance. However, a major limitation of these works is that they ignore effective modality representation learning before fusion. In this work, we propose a novel text-audio sentiment analysis framework, named StyleBERT, to enhance the emotional information of unimodal representations by learning distinct modality styles, such that the model already obtains an effective unimodal representation before fusion, which mitigates the reliance on fusion. In particular, we propose a Bi-directional Style Enhancement module, which learns one contextualized style representation and two differentiated style representations for each modality, where the relevant semantic information across modalities and the discriminative characteristics of each modality will be captured. Furthermore, to learn fine-grained acoustic representation, we only use the directly available Log-Mel spectrograms as audio modality inputs and encode it with a multi-head self-attention mechanism. Comprehensive experimental results on three widely-used benchmark datasets demonstrate that the proposed StyleBERT is an effective multimodal framework and significantly outperforms the state-of-the-art multimodal baselines. Our code is available at https://github.com/lsq960124/StyleBERT.

Multimodal Sentiment Analysis of Online Product Information Based on Text Mining Under the Influence of Social Media

Currently, with the dramatic increase in social media users and the greater variety of online product information, manual processing of this information is time-consuming and labour-intensive. Therefore, based on the text mining of online information, this paper analyzes the text representation method of online information, discusses the long short-term memory network, and constructs an interactive attention graph convolutional network (IAGCN) model based on graph convolutional neural network (GCNN) and attention mechanism to study the multimodal sentiment analysis (MSA) of online product information. The results show that the IAGCN model improves the accuracy by 4.78% and the F1 value by 29.25% compared with the pure interactive attention network. Meanwhile, it is found that the performance of the model is optimal when the GCNN is two layers and uses syntactic position attention. This research has important practical significance for MSA of online product information in social media.

Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis has a wide range of applications due to its information complementarity in multimodal interactions. Previous works focus more on investigating efficient joint representations, but they rarely consider the insufficient unimodal features extraction and data redundancy of multimodal fusion. In this paper, a Video-based Cross-modal Auxiliary Network (VCAN) is proposed, which is comprised of an audio features map module and a cross-modal selection module. The first module is designed to substantially increase feature diversity in audio feature extraction, aiming to improve classification accuracy by providing more comprehensive acoustic representations. To empower the model to handle redundant visual features, the second module is addressed to efficiently filter the redundant visual frames during integrating audiovisual data. Moreover, a classifier group consisting of several image classification networks is introduced to predict sentiment polarities and emotion categories. Extensive experimental results on RAVDESS, CMU-MOSI, and CMU-MOSEI benchmarks indicate that VCAN is significantly superior to the state-of-the-art methods for improving the classification accuracy of multimodal sentiment analysis.

Visual Enhancement Capsule Network for Aspect-based Multimodal Sentiment Analysis

Multimodal sentiment analysis, which aims to recognize the emotions expressed in multimodal data, has attracted extensive attention in both academia and industry. However, most of the current studies on user-generated reviews classify the overall sentiments of reviews and hardly consider the aspects of user expression. In addition, user-generated reviews on social media are usually dominated by short texts expressing opinions, sometimes attached with images to complement or enhance the emotion. Based on this observation, we propose a visual enhancement capsule network (VECapsNet) based on multimodal fusion for the task of aspect-based sentiment analysis. Firstly, an adaptive mask memory capsule network is designed to extract the local clustering information from opinion text. Then, an aspect-guided visual attention mechanism is constructed to obtain the image information related to the aspect phrases. Finally, a multimodal fusion module based on interactive learning is presented for multimodal sentiment classification, which takes the aspect phrases as the query vectors to continuously capture the multimodal features correlated to the affective entities in multi-round iterative learning. Otherwise, due to the limited number of multimodal aspect-based sentiment review datasets at present, we build a large-scale multimodal aspect-based sentiment dataset of Chinese restaurant reviews, called MTCom. The extensive experiments both on the single-modal and multimodal datasets demonstrate that our model can better capture the local aspect-based sentiment features and is more applicable for general multimodal user reviews than existing methods. The experimental results verify the effectiveness of our proposed VECapsNet.

Multimodal Sentiment Analysis: A Systematic review of History, Datasets, Multimodal Fusion Methods, Applications, Challenges and Future Directions

Multimodal Sentiment Analysis: A Systematic review of History, Datasets, Multimodal Fusion Methods, Applications, Challenges and Future Directions

Learning modality-fused representation based on transformer for emotion analysis

Modality-fused representation is an essential and challenging task in multimodal emotion analysis. Previous studies have already yielded remarkable achievements. However, there are two problems: insufficient feature interaction and rough data fusion. To investigate these two challenges more deeply, first, a hybrid architecture, which consists of convolution and a transformer, is proposed to extract local and global features. Second, for extracting more sufficient mutual features from multimodal datasets, our model is comprised of three parts: (1) the interior transformer encoder (TE) aims to extract the intramodality characteristics from the current monomodality; (2) the between TE aims to extract the intermodality feature between two different modalities; and (3) the enhance TE aims to extract the target modality enhance feature from multimodality. Finally, instead of directly fusing features by a linear function, we employ a popular and widely used multimodal factorized high-order pooling mechanism to obtain a more distinguishable feature representation. Extensive experiments on three multimodal sentiment datasets (CMU-MOSEI, CMU-MOSI, and IEMOCAP) demonstrate that our approach reaches the state-of-the-art in an unaligned version setting. Compared with the mainstream methods, our proposed method shows superiority in both word-aligned and unaligned settings.

AOBERT: All-modalities-in-One BERT for multimodal sentiment analysis

Multimodal sentiment analysis utilizes various modalities such as Text, Vision and Speech to predict sentiment. As these modalities have unique characteristics, methods have been developed for fusing features. However, the overall modality characteristics are not guaranteed, because traditional fusion methods have some loss of intra-modality and inter-modality. To solve this problem, we introduce a single-stream transformer, All-modalities-in-One BERT (AOBERT). The model is pre-trained on two tasks simultaneously: Multimodal Masked Language Modeling (MMLM) and Alignment Prediction (AP). The dependency and relationship between modalities can be determined using two pre-training tasks. AOBERT achieved state-of-the-art results on the CMU-MOSI, CMU-MOSEI, and UR-FUNNY datasets. Furthermore, ablation studies that validated combinations of modalities, effects of MMLM and AP and fusion methods confirmed the effectiveness of the proposed model.

The Application of Interactive Humanoid Robots in the History Education of Museums Under Artificial Intelligence

The purpose is to improve the application of museum robots in museum scenes, enhance the service capabilities of robots in museums, break tourists’ boring concepts of museum environment, manual explanation, services, etc., and promote tourists’ exhibition experience. A method for sentiment analysis of humanoid robots in museums is proposed by studying the transformation of museums with the help of artificial intelligence (AI) technology, as well as the function and significance of museums in history education. First, the function of museums in history education and the role of AI in constructing intelligent museums are described. Second, on account of the multimodal sentiment analysis method of speech and emotion, a scenario model of the visitor museum is established. An uncertain reasoning method for robot service tasks based on Multi-entity Bayesian network (MEBN) is also proposed. Finally, the proposed model is validated by experiments. The results show that compared with the recognition rates of Arousal and Valence dimensions, the consistency correlation coefficient value of the Kalman filter is higher. The Consistency Correlation Coefficient (CCC) value of the Arousal dimension is 0.703, and the CCC value of the Valence dimension is 0.766. Besides, in different tour times, the proportion of services that tourists want to be provided with varies in different emotional states. From time t1 to time t2, the proportion of tourists who want to hear explanations of cultural relics dropped by 11.5%, while the proportion of tourists who want to be provided with tea service increased by 24%. This indicates that when the Kalman filter algorithm performs continuous emotion recognition of a multimodal fusion, the final emotion recognition accuracy is higher, and emotion analysis can help humanoid robots to be more intelligent and humanized. The proposed sentiment analysis based on the multimodal analysis and MEBN’s uncertainty reasoning method for robot service tasks not only broadens the practical application field of intelligent robots under human–computer interaction technology but also has important research significance for the innovative education development of museum history education.

TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis

TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis

Heterogeneous graph convolution based on In-domain Self-supervision for Multimodal Sentiment Analysis

The inability to fully exploit domain-specific knowledge and the lack of an effective integration method have been the difficulties and focus of multimodal sentiment analysis. In this paper, we propose heterogeneous graph convolution with in-domain self-supervised multi-task learning for multimodal sentiment analysis (HIS-MSA) to solve these problems. Firstly, HIS-MSA carries out the second pre-trained with different self-supervised training strategies to fully mine the unique knowledge of the in-domain corpus, and give BERT the awareness of professional field. Secondly, HIS-MSA uses heterogeneous graph, which is good at integrating heterogeneous knowledge, to fuse feature from multiple sources. Finally, a unimodal label generation module is used to jointly guide multimodal tasks and unimodal tasks to balance independent and complementary information between the modalities. We conducted experiments on the datasets MOSI and MOSEI, which have 2199 and 23454 video segments respectively. The results show an average improvement of approximately 1.5 points in all metrics compared to the current state-of-the-art model.

Excavating multimodal correlation for representation learning

A majority of previous methods for multimodal representation learning ignore the rich correlation information inherently stored in each sample, leading to a lack of robustness when trained on small datasets. Although a few contrastive learning frameworks leverage that information in a self-supervised manner, they generally encourage the intra-sample unimodal representations to be identical, neglecting the modality-specific information carried by individual modalities. In contrast, we propose a novel algorithm that learns the correlations between modalities to facilitate downstream multimodal tasks by leveraging the prior information across samples, and we explore the feasibility of the proposed method on elaborately designed unsupervised and supervised auxiliary learning tasks. Specifically, we construct the positive and negative sets for correlation learning as unimodal embeddings from the same sample and from different samples, respectively. A weak predictor is employed on the concatenated unimodal embeddings to learn the correspondence relationship for each set. In this way, the model can correlate unimodal features and discover the shared information across modalities. In contrast to contrastive learning methods, the proposed framework is compatible with any number of modalities and can retain modality-specific information, enabling multimodal representation to capture richer information. Moreover, in the supervised version, one of the main novelties is that the sample labels are further utilized to learn more discriminative features, where the assigned correlation scores of negative sets vary according to the label variations between the associated samples. Extensive experiments suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis, emotion recognition, and humor detection, and can improve the performance of various fusion approaches.

Impact of Annotator Demographics on Sentiment Dataset Labeling

As machine learning methods become more powerful and capture more nuances of human behavior, biases in the dataset can shape what the model learns and is evaluated on. This paper explores and attempts to quantify the uncertainties and biases due to annotator demographics when creating sentiment analysis datasets. We ask >1000 crowdworkers to provide their demographic information and annotations for multimodal sentiment data and its component modalities. We show that demographic differences among annotators impute a significant effect on their ratings, and that these effects also occur in each component modality. We compare predictions of different state-of-the-art multimodal machine learning algorithms against annotations provided by different demographic groups, and find that changing annotator demographics can cause >4.5 in accuracy difference when determining positive versus negative sentiment. Our findings underscore the importance of accounting for crowdworker attributes, such as demographics, when building datasets, evaluating algorithms, and interpreting results for sentiment analysis.

Modality-invariant temporal representation learning for multimodal sentiment classification

Multimodal sentiment classification is a notable research field that aims to refine sentimental information and classify the sentiment tendency from sequential multimodal data. Most existing sentimental recognition algorithms explore multimodal fusion schemes that achieve good performance. However, there are two key challenges to overcome. First, it is essential to effectively extract inter- and intra-modality features prior to fusion, while simultaneously reducing ambiguity. The second challenge is how to learn modality-invariant representations that capture the underlying similarities. In this paper, we present a modality-invariant temporal learning technique and a new gated inter-modality attention mechanism to overcome these issues. For the first challenge, our proposed gated inter-modality attention mechanism performs modality interactions and filters inconsistencies from multiple modalities in an adaptive manner. We also use parallel structures to learn more comprehensive sentimental information in pairs (i.e., acoustic and visual). In addition, to address the second problem, we treat each modality as a multivariate Gaussian distribution (considering each timestamp as a single Gaussian distribution) and use the KL divergence to capture the implicit temporal distribution-level similarities. These strategies are helpful in reducing domain shifts between different modalities and extracting effective sequential modality-invariant representations. We have conducted experiments on several public datasets (i.e., YouTube and MOUD) and the results show that our proposed method outperforms the state-of-the-art multimodal sentiment categorization methods.

Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification

The recent booming of artificial intelligence (AI) applications, e.g., affective robots, human-machine interfaces, autonomous vehicles, and so on, has produced a great number of multi-modal records of human communication. Such data often carry latent subjective users’ attitudes and opinions, which provides a practical and feasible path to realize the connection between human emotion and intelligence services. Sentiment and emotion analysis of multi-modal records is of great value to improve the intelligence level of affective services. However, how to find an optimal manner to learn people’s sentiments and emotional representations has been a difficult problem, since both of them involve subtle mind activity. To solve this problem, a lot of approaches have been published, but most of them are insufficient to mine sentiment and emotion, since they have treated sentiment analysis and emotion recognition as two separate tasks. The interaction between them has been neglected, which limits the efficiency of sentiment and emotion representation learning. In this work, emotion is seen as the external expression of sentiment, while sentiment is the essential nature of emotion. We thus argue that they are strongly related to each other where one’s judgment helps the decision of the other. The key challenges are multi-modal fused representation and the interaction between sentiment and emotion. To solve such issues, we design an external knowledge enhanced multi-task representation learning network, termed KAMT. The major elements contain two attention mechanisms, which are inter-modal and inter-task attentions and an external knowledge augmentation layer. The external knowledge augmentation layer is used to extract the vector of the participant’s gender, age, occupation, and of overall color or shape. The main use of inter-modal attention is to capture effective multi-modal fused features. Inter-task attention is designed to model the correlation between sentiment analysis and emotion classification. We perform experiments on three widely used datasets, and the experimental performance proves the effectiveness of the KAMT model.

MALO-LSTM: Multimodal Sentiment Analysis Using Modified Ant Lion Optimization with Long Short Term Memory Network

MALO-LSTM: Multimodal Sentiment Analysis Using Modified Ant Lion Optimization with Long Short Term Memory Network