Capsule network-based deep ensemble transfer learning for multimodal sentiment analysis

Abstract

Understanding the attitudes of users about different topics through their huge amount of comments and opinions on social networks is an emerging hot issue. Analyzing the sentiments of these posts and comments provides useful information utilized for many applications. Multimodal sentiments often contain more information than single-modal sentiments, including text, image, audio and video, which leads to better performance of multimodal sentiment analysis (MSA) compared to single-modal sentiment analysis (SSA). In this paper, a capsule network-based deep ensemble transfer learning approach, called DSY-ETL-MSA, is proposed for MSA on images and texts, the results of which are fused using Yager theory. Capsule networks and ensemble learning methods improve classification performance, and the deep transfer learning approach reduces the training time. A hybrid deep architecture is used for automatic feature extraction in the proposed method. For analyzing the sentiment of image modality, a pre-trained VGG16 model, fine-tuned on the datasets, is used to extract high-level features for image classification. A capsule convolutional neural network (CNN) is also separately used for extracting image features and classifying them. In the text modality, the pre-trained GloVe model is exploited to embed words and 2 separate classifiers are employed for text classifications. The Yager fusion rules are finally used for early and late fusions of the results of the classifiers. The results of the text and image classifiers are combined separately as the early fusion and the final results of them are fused at the decision level as the late fusion. The proposed model is empirically evaluated on the MVSA and T4SA datasets. The significant performance improvement compared to a variety of former methods is shown in the experimental results. The final accuracies obtained by the proposed method on the MVSA and T4SA datasets are 0.9866 and 0.9996, respectively.