Application of self-learning enhancing Huff and Puff gas injection optimization in shale reservoirs through sequence-based proxy reservoir simulation and reinforcement learning

Abstract

This study introduces a novel approach to optimize Huff and Puff (H-n-P) gas injection in complex shale reservoirs by integrating sequence-based Proxy reservoir simulation and reinforcement learning (RL). The aim is to drastically reduce simulation time while enhancing H-n-P gas injection project decisions. The method involves three main steps: (1) validating simulation outcomes using the GFREE-SIM in-house multi-porosity simulator against actual data; (2) training and validating a Proxy sequence-based machine learning (ML) model with simulation data; (3) implementing a customized RL model that interacts with the environment (e.g., sequence-based Proxy reservoir simulation) to learn optimal actions (e.g., injection/production periods, gas injection rate) based on cumulative rewards (net present value), discussed in the subsequent article.The Proxy reservoir simulator serves as a rapid numerical tool, generating multiple scenarios in minutes. Despite minor accuracy trade-offs, this innovative method accelerates simulations up to 20,000X. Operating as the environment for RL interactions, the Proxy model's reliability and speed play a crucial role.To address the complexities of cyclic enhanced oil recovery, the study employs a self-learning Sequential Data-Informed Degenerate Deep Reinforcement Learning Optimization (SDI-DgRL) framework. This framework leverages RL techniques to identify optimal gas injection strategies. An example is presented where the SDI-DgRL framework investigates injection/production periods and gas injection rate, achieving robust optimization in fewer than 1,000 scenarios. Results indicate that hyperparameter tuning significantly enhances the stability and performance of RL models. After several experiments, it is recommended that injecting 2.0–2.2 MMscfd of gas over 2–2.2 months, followed by production periods of 2.5–3 months, yields the highest net present value (NPV).