A novel trading system for the stock market using Deep Q-Network action and instance selection

Abstract

Stock trading is a complex decision-making process that involves predicting market price movements. Many investors attempt to buy at low prices and sell at high prices, which can be difficult due to numerous factors that can influence stock prices. Consequently, researchers have developed trading systems designed to generate high stock market returns using machine learning or deep learning techniques. However, building a trading system using supervised learning can be challenging in real-time due to noisy stock price data. Reinforcement learning trading systems can play an active role in the stock market and their performance depends heavily on how the learning environment is constructed. Therefore, this study proposes a Deep Q-Network (DQN) Action Instance Selection Trading System (DAIS) to improve the limitations of both supervised learning and reinforcement learning trading systems. DAIS learns effective trading timing by labeling the behavior of DQN models and using instance selection to effectively handle noise in stock price data. Our proposed model is evaluated by testing 72 companies listed on Korean Composite Stock Price Indexes (KOSPI) and comparing the results with those obtained in previous studies using supervised and reinforcement learning trading systems. The results indicate that the DAIS trading system is more efficient than the compared systems.