A dual-channel multimodal sentiment analysis framework based on three-way decision

Abstract

In the exploration of human–computer interaction, accurately identifying and understanding the user’s sentimental state helps enhance the user experience. Multimodal sentiment analysis aims to identify sentiments through various signals. However, inconsistencies in sentiments between text and visual–audio signals pose a challenge, as models often struggle to capture these differences adequately. To alleviate this issue, we propose a dual-channel sentiment analysis framework based on three-way decision. This framework includes a multimodal model, a visual–audio model, and a three-way decision module. Specifically, the multimodal model and visual–audio model generate predictions, and the three-way decision module selects an appropriate prediction. We improve the binary cross-entropy loss function for the visual–audio model to integrate the three-way decision module better and improve sentiment prediction accuracy. Additionally, we introduce an adversarial training task to ensure differences in predictions between the two models, thereby enhancing the influence of the visual–audio model. Experimental results on four public datasets demonstrate that our framework significantly improves the accuracy of sentiment recognition and reduces the average decision time.