Deep Reinforcement Learning in Agent Based Financial Market Simulation

Iwao Maeda,Hiroyasu Matsushima,David Degraw,Michiharu Kitano,Kiyoshi Izumi,Atsuo Kato,Hiroki Sakaji

doi:10.3390/jrfm13040071

Iwao Maeda, Hiroyasu Matsushima + Show 5 more

Open Access

https://doi.org/10.3390/jrfm13040071

Copy DOI

Abstract

Prediction of financial market data with deep learning models has achieved some level of recent success. However, historical financial data suffer from an unknowable state space, limited observations, and the inability to model the impact of your own actions on the market can often be prohibitive when trying to find investment strategies using deep reinforcement learning. One way to overcome these limitations is to augment real market data with agent based artificial market simulation. Artificial market simulations designed to reproduce realistic market features may be used to create unobserved market states, to model the impact of your own investment actions on the market itself, and train models with as much data as necessary. In this study we propose a framework for training deep reinforcement learning models in agent based artificial price-order-book simulations that yield non-trivial policies under diverse conditions with market impact. Our simulations confirm that the proposed deep reinforcement learning model with unique task-specific reward function was able to learn a robust investment strategy with an attractive risk-return profile.

Highlights

In recent years, applications of deep learning to predicting financial market data have achieved some level of success (Chong et al 2017; Long et al 2019; Lahmiri and Bekiros 2019, 2020)
We can see that the liquidation-value reward function was instrumental for the deep reinforcement learning (DRL) agent to learn a profitable trading strategy that simultaneously mitigates inventory risk
It was confirmed that the learning efficiency greatly differs depending on the reward functions, and the action probability distributions of well-trained strategies were consistent with investment strategies used in real markets

Summary

Introduction

Applications of deep learning to predicting financial market data have achieved some level of success (Chong et al 2017; Long et al 2019; Lahmiri and Bekiros 2019, 2020). Issues such as heteroskedasticity Nelson (1991), low signal-to-noise Aaker and Jacobson (1994), and the large observer effect seen in market impact, make their use in real-world applications challenging. Financial practitioners have traditionally been limited to training models with past data and do not have many options for improving their predictive models (Bailey et al 2014) Such “back-testing” of models cannot account for transaction costs or market impact, both of which can be of comparable magnitude to the forecast returns (Hill and Faff 2010)

Methods

Results

Conclusion