A distributed joint sentiment and topic modeling using Spark for big opinion mining

Abstract

Opinion data are produced rapidly by a large and uncontrolled number of opinion holders in different domains (public, business, politic and etc). The volume, variety and velocity of such data requires an opinion mining model to be also adopted with the ever growing and huge volume of opinions and obtaining the probabilistic generative model advantages. In this paper we propose a parallel implementation of joint sentiment and topic (JST) model for simultaneously discovering topics and sentiments from reviews on Spark. Spark is an open source and fast cluster computing framework for large-scale data processing. Here we discuss the implementation of JST on Spark and also discuss the benefit of using Spark while exploring the challenges encountered. We used different Amazon opinion datasets with different volume such as (reviews of electronic devices, book, restaurants, DVD and kitchen). The results present significant speedup and high efficiency on larger scale dataset in our experiments.