An adaptive dual-level reinforcement learning approach for optimal trade execution

Abstract

The purpose of this research is to devise a tactic that can closely track the daily cumulative volume-weighted average price (VWAP) using reinforcement learning while minimizing the deviation from the VWAP. Previous studies often choose a relatively short trading horizon to implement their models, making it difficult to accurately track the daily cumulative VWAP since the stock price movement is often insignificant within the short trading horizon. On the other hand, training reinforcement learning models directly over a longer, daily horizon is burdensome due to extensive sequence length. Hence, there is a need for a method that can divide the long daily horizon into smaller, more manageable segments. We propose a method that leverages the U-shaped pattern of intraday stock trade volumes and uses Proximal Policy Optimization (PPO) as the learning algorithm. Our method follows a dual-level approach: a Transformer model that captures the overall (global) distribution of daily volumes in a U-shape, and a LSTM model that handles the distribution of orders within smaller (local) time intervals. The results from our experiments suggest that this dual-level architecture improves cumulative VWAP tracking accuracy compared to previous reinforcement learning approaches. The key finding is that explicitly accounting for the U-shaped intraday volume pattern leads to better performance in approximating the cumulative daily VWAP. This has implications for developing trading strategies that need to efficiently track VWAP over a full trading day.