TBDQN: A novel two-branch deep Q-network for crude oil and natural gas futures trading

Abstract

Algorithmic trading plays a significant role in the trade of crude oil and natural gas futures. In this paper we propose a novel deep reinforcement learning (DRL) algorithm, dubbed two-branch deep Q-network (TBDQN), to automatically produce consistently profitable and robust trading signals in crude oil and natural gas futures markets. The first branch exploits long-short-term memory (LSTM) module to discover potential features hidden behind many technical indicators; the second branch extracts intrinsic features from futures contracts, trading positions and OHLCV using deep neural network. The extracted features are fused together to form the state vector of Q-leaning. In order to facilitate the training of the TBDQN model learning, we design a novel reward function by incorporating both immediate and long-term rewards. Compared to other popular methods, the proposed algorithm demonstrates excellent performance on the evaluation criteria of annualized return and Sharpe ratio in the oil and gas futures markets, which proves the effectiveness of our method.