Multi-modal sentiment analysis based on graph neural network

Abstract

Thanks to popularity of social media, people are witnessing the rapid proliferation of posts with various modalities. It is worth noting that these multi-modal expressions share certain characteristics, including the interdependence of objects in the posted images, which is sometimes overlooked in previous researches as they focused on single image-text posts and pay little attention on obtaining the global features. In this paper, a neural network with multiple channels for image-text sentiment detection is proposed. The first step is to encode text and images to capture implicit tendencies. Then the introduction of this model obtains multi-modal expressions by collecting the shared characteristics of the dataset. Finally, the attention mechanism provides reliable predictions of the sentiment tendencies of the given pairs of image-text data. The results of experiments conducted on two publicly available datasets crawled from Twitter prove the reliability of the model on multi-modal sentiment detection, since the model precedes previously proposed models in the main evaluating criteria.