Enhancing Stock Trend Prediction Using BERT-Based Sentiment Analysis and Machine Learning Techniques

Abstract

Predicting stock trends with precision in the ever-evolving financial markets continues to be a formidable challenge. This research investigates an innovative approach that amalgamates the capabilities of BERT (Bidirectional Encoder Representations from Transformers) for sentiment classification (Pang et al., 2002; ?) with supervised machine learning techniques to elevate the accuracy of stock trend prediction. By harnessing the natural language processing process of BERT and its capacity to understand context and sentiment in textual data, coupled with established machine learning methodologies, we aim to provide a robust solution to the intricacies of stock market prediction. By leveraging BERT's natural language processing capabilities, we extract sentiment features from financial news articles. These sentiment scores, combined with traditional financial indicators, form a comprehensive set of features for our predictive model. We aggregate daily net sentiment, among other metrics, and demonstrate its statistically significant predictive efficacy concerning subsequent movements in the stock market. We employed a machine learning model to establish a quantitative relationship between the aggregation of daily net sentiment and trends in stock market movements. Which improved the state-of-the-art performance by 15 percentage points. This research contributes to the ongoing effort to improve stock trend prediction methods, ultimately aiding market participants in making informed investment choices.