Multi-modal topic modeling from social media data using deep transfer learning

Abstract

As social media platforms grow rapidly, multi-modal data is becoming more and more prevalent. A user can better understand events by analyzing multimodal data for topics. Automatic topic detection from multimodal data can potentially have tremendous value for advertising and government agencies for whom public opinion matters for strategic decisions and policy making. However , multimodal topic detection is complicated for two reasons: (1) The nature of the multimodal data varies from one medium to another, and (2) The noisy nature of webdata. Conventional topic models are ineffective in dealing with these two problems. This paper proposes, a framework for multimodal topic modeling for social media data that uses topics extracted using Latent Dirichlet Allocation (LDA) and patterns found from images using transfer learning. The proposed framework makes use of textual as well as visual data for topic detection. The experiments are conducted on the benchmark datasets: Flickr8k, Flickr30k, and MCG WEBV. The proposed work outperformed other techniques in terms of accuracy (0.63), precision (0.75), recall (0.97), F-Measure (0.85), Bleu-1(0.68),METEOR (0.17) , ROUGE-L (0.49), and CIDEr (0.573). The proposed work is compared to state-of-the-art methods to demonstrate its accuracy.