Optimization of steam injection in SAGD using reinforcement learning

Abstract

Conventional steam injection strategies or policies in SAGD are typically not a result of a formal optimization process but rather empirically found. However, finding the steam injection policy that maximizes cumulative performance over the entire production horizon represents a major challenge due to the complex dynamics of the physical phenomena. To address this challenge one alternative is to use a reinforcement learning (RL) approach; here an agent is trained to find the optimal policy only by continuous interactions with the reservoir simulation model (environment). In this work, the RL-SARSA on-line on-policy learning algorithm is used to find the optimal steam injection policy and offer insight in the physics of the SAGD process. Net Present Value (NPV) is used as a performance measure of objective function, the action-value function is approximated using a stochastic gradient regression strategy, and the state vector is featurized using radial basis kernels. The action space is discrete, three (3) possible actions: increase/decrease/no change on the steam injection rate w.r.t the previous value, and the state space is continuous and is made up of cumulative values for water and oil production and steam injection. Additionally, the environment is represented by a one well pair reservoir simulation model built using data from a reservoir located in northern Alberta and a production horizon of 250 days is considered. For the considered case study, the agent showed a significant improvement on the cumulative NPV value after 200 episodes or simulations and the optimal policies are characterized by three distinct regions, (1) constant rate to increase the average reservoir pressure and allow the steam chamber reach the overburden, (2) sharp increase to increase average reservoir temperature and abrupt decrease and (3) constant minimum value to maintain the average reservoir temperature. • Reinforcement learning with function approximation has been successfully applied to the optimization of steam injection in a SAGD process. • The optimal steam injection policy is characterized by three distinct regions: constant rate, sharp increase and abrupt decrease, constant minimum rate. • For optimum SAGD operations, pressure plays a key role until the steam chamber reaches the overburden, afterwards temperature dominates.