Stock Market Real Time Recommender Model Using Apache Spark Framework

Abstract

The stock market is considered a complicated and nonlinear system. Now stock market prediction is recognized as an attracting point for financial investors. The historical price is not considered as the main factor to predict the stock market trend. There are many other factors such as politics and natural events that affect social media environments like Twitter and Facebook which generate huge datasets needed data analysis to extract the polarity of these data and its effectiveness on the stock market. On the other hand, these data may be unstructured and need special handling on storing and processing. This paper proposes a real-time forecasting of stock market trends based on news, tweets, and historical price. A supervised machine learning algorithms used to build this model. Historical price will be combined with sentiment analysis to build the hybrid model based on Apache Spark and Hadoop HDFS to handle big data (structured and unstructured) generated from social media and news websites. The proposed model works in two modes; the offline mode that works on historical data including today’s data after ending of a stock market session, and real-time mode that works on real-time data during the stock market session. This model increases the accuracy of prediction due to the additional features added by sentiment analysis on StockTwits and market news data. In addition, this model enhances the performance of handling this data set due to parallel processing occurred on data using Apache Spark.