SentiMatch: Sentiment Analysis of Multi-modal Social Media Posts

Abstract

Social media users are increasingly using images and videos to communicate their feelings. However, researchers usually rely on textual data for sentiment analysis. In this paper, we focus on Twitter posts having both text and image data and analyze the match between their sentiments, when predicted independently. We also propose a method to predict the sentiment of a social media post as a collective entity from the individual sentiments of text and image. Comparing our proposed model to a naive approach where the sentiment of the text is assigned to the image also, we see an improvement of around 20% in prediction of sentiment of Twitter images associated with text.