A novel hierarchical feature selection with local shuffling and models reweighting for stock price forecasting

Abstract

Stock price forecasting is a challenging task due to the complexity of financial markets and the high volatility of stocks. Because of the strong nonlinear representation ability of neural network models, such as long-short memory network (LSTM) and deep learning, they are used extensively in recent years for stock price forecasting. However, due to the volatility of financial data, neural network models often suffer from overfitting or instability problems. In addition, quantitative trading with feature mining tools can generate a growing number of features for financial data. Therefore, selecting effective features for financial data is an urgent problem. To address these problems, we propose a novel hierarchical feature selection with local shuffling (HFSLS) and models reweighting (MR) based on LSTM, named HFSLSMR-LSTM, for stock price forecasting. Specifically, for each layer, local shuffling perturbs each feature to re-predict, and its predicted value is compared with the true value to calculate the feature importance, and the important features are selected and returned to the next layer. Besides, a proximity reweighting scheme is presented to adjust the weight for each layer model that learns from hierarchical features. The HFSLSMR-LSTM model is still effective for financial data with multiple features and frequent fluctuations. Experimental results on stock index dataset and Dow Jones Industrial Average dataset demonstrate that the HFSLSMR-LSTM outperforms Informer, DoubleEnsemble, LSTM, GRU, BI-GRU and BI-LSTM on the metrics MSE, RMSE, MAE, MAPE and R2.