Integrating color cues to improve multimodal sentiment analysis in social media

Abstract

Multimodal sentiment analysis is an emerging and critical area of research, particularly in the context of social media where users express their emotions through both text and images. However, existing methods that learn common semantic features between visual and textual modalities for multimodal sentiment analysis often overlook the importance of color information, which plays a crucial role in sentiment expression, according to psychology and art theory. This paper proposes a novel model, named ICCI, which aims to enhance multimodal sentiment analysis in social media by integrating color cues, in order to address the limitation in existing approaches. The proposed model leverages both the semantic information from image-text pairs and the color cues from images to improve the accuracy of sentiment analysis. The model comprises a feature extraction module that extracts semantic features from both images and text, as well as color features from images. Furthermore, the feature interaction module employs a cross-attention mechanism to enable the interaction of information between semantic features and color features. Finally, the label prediction module integrates all attention features to enhance multimodal sentiment analysis. The proposed model was evaluated through experiments on two widely used benchmark datasets, namely MVSA-Single and MVSA-Multiple datasets, demonstrating its effectiveness in outperforming existing methods. On MVSA-Single, ICCI achieved an accuracy of 79.33%. Similarly, on MVSA-Multiple, ICCI achieved an accuracy of 73.29%. The results underscore the importance of integrating color information for more accurate sentiment analysis in social media.