Deep reinforcement learning for the optimal placement of cryptocurrency limit orders

Abstract

This paper presents the first large-scale application of deep reinforcement learning to optimize execution at cryptocurrency exchanges by learning optimal limit order placement strategies. Execution optimization is highly relevant for both professional asset managers and private investors as execution quality affects portfolio performance at economically significant levels and is the target of regulatory supervision. To optimize execution with deep reinforcement learning, we design a problem-specific training environment that introduces a purpose-built reward function, hand-crafted market state features and a virtual limit order exchange. We empirically compare state-of-the-art deep reinforcement learning algorithms to several benchmarks with market data from major cryptocurrency exchanges, which represent an ideal test bed for our study as liquidity costs are relatively high. In total, we leverage 18 months of high-frequency data for several currency pairs with 300 million trades and more than 3.5 million order book states. We find proximal policy optimization to reliably learn superior order placement strategies. By interacting with our simulated limit order exchange, it learns cryptocurrency execution strategies that are empirically known from established markets. Order placement becomes more aggressive in anticipation of lower execution probabilities, which is indicated by trade and order imbalances.