Predicting the Stock Market Behavior Using Historic Data Analysis and News Sentiment Analysis in R

Abstract

Predicting the stock market has always been an attractive topic, mainly due to its vitality in the economic and financial sectors. Yet, predictions of the stock market pose a challenging exercise, even to the brightest and sharpest minds in the business. Prediction of stock market is never an easy task, due to the complexity and dynamic characteristics of the data it deals with. Bulk amount of the data output generated by the stock market is considered to be a treasure house of knowledge for investors; several studies have been conducted in an attempt to predict the stock market trends. Hence, it is imminent to uncover the behavior of the stock market data in order to avoid future investment risks for the investors. Here we tried a different approach for solving this problem by combining two different components: sentiment analysis on stock-related news reports and historic data analysis. The primary aim of this study was to construct an efficient model to predict trends in the stock market, with minimum error ratio and with maximum accuracy possible for the prediction. This model achieved notably better accuracy as compared to the models created in the previous studies. Two datasets were used in this study. A historical dataset containing the stock values of over ten 11, xxxx companies in the previous years, and a sentiment dataset containing the stock market news reports from social media and other online sources. The first step was to analyze the stock reports and classify them either as a positive or a negative sentiment. Lexicon method of text sentiment classification was used for this purpose. Predictions at this stage achieved an accuracy of 67.14%. The second step of this study used ts and ARIMA functions to predict stock trend, using the historical dataset. In the final step, results from both the components were combined together, to predict stock prices in future. This improved the prediction accuracy up to 89.80%.