Parallel Reservoir Simulation Research Articles

An insight into the effect of different parameters for optimizing parallel reservoir simulation performance

Parallel simulation significantly reduces runtime in oil reservoir simulations, enabling higher-resolution analysis and a better understanding of reservoir behavior. This study comprehensively evaluates how simulation time, hardware configuration, and various physical parameters impact the efficiency, scalability, and performance metrics such as run time reduction (RTR) and number of optimal cores (NOC) in parallel simulations. This study introduced the reduced simulation time (RST) concept, an innovative approach that enhances parallel simulation efficiency, and dynamic load balance. In this approach, simulations were performed using only 5% of the total simulation time. The findings reveal that in 80% of the models, switching to RST mode has no effect on NOC but significantly reduces overall execution time in most cases. In complex models, this time reduction can be as high as 60% compared to serial mode. In the remaining 20%, the number of cores in the total simulation time (TST) mode may be higher than in the RST mode, but this difference can be disregarded due to limited hardware resources and no change in RTR.

Multilevel Schur-complement algorithms for scalable parallel reservoir simulation with temperature variation

In reservoir simulation, the non-isothermal multiphase flow problem introduces the temperature variable to account for thermal effects, simultaneously posing challenges in efficiently solving the nonlinear systems for large-scale simulations. In this paper, we introduce and investigate a family of Schur-complement-based field-split algorithms designed for addressing non-isothermal multiphase flow problems, particularly those characterized by high heterogeneity. This algorithm involves decomposing a large system into smaller, more manageable sub-systems for solving non-isothermal multiphase flow problems with multiple physical fields, which enables parallel computation and makes it suitable for high-performance computing environments. Furthermore, a multilevel Schur-complement preconditioner, which involves applying the Schur-complement technique at each level of the hierarchy by capturing the coupling between different fields and physics, is proposed to enhance the efficiency and robustness of the parallel simulator. Large-scale simulations for both benchmark and realistic problems are conducted on a supercomputer, showcasing the method's efficacy in managing heat diffusion, significantly reducing linear iterations, and demonstrating a good parallel scalability.

Inverse Modeling of Reservoirs with Tilted Fluid Contacts

Summary In this paper, we present a new approach for simulating reservoirs with tilted fluid contacts produced by hydrodynamics. The proposed method solves a nonlinear inverse problem to determine the aquifer flow field that best reproduces the observed contact tilt. The computational effort required to solve this inverse problem is reduced by choosing a pressure-based objective function and applying gradient-based optimization. This approach is entirely automated, in contrast to previous works that have used laborious trial-and-error methods to estimate the aquifer flow field. In addition, the proposed method introduces no additional physics beyond hydrodynamics to model reservoirs with tilted contacts. The proposed method is integrated into a parallel reservoir simulator. A synthetic reservoir is constructed by introducing an artificial tilt, and the new approach is applied to estimate the aquifer flow field. The estimate produced by the proposed method matches the true flow field well and is able to prevent large fluid motions near the contact surface when simulating production from the reservoir. The proposed method is compared with an existing approach that uses capillary pressure adjustments to hold the tilted contact in place. The proposed method is shown to outperform the existing approach without significantly impacting the simulation results.

Multiphase transient analysis of horizontal wells during CO2-EOR

We apply a numerical well test model that considers the transient flow in well and the complex displacement mechanisms for the multiphase transient analysis of horizontal wells during CO 2 -EOR. Aimed to perform a systematic and reliable analysis, we run the model on a high-resolution non-uniform grid to accurately capture the transient flow in the near wellbore region as well as the complex displacement process. In this work, we interpret the pressure response curve in two steps to find the root causes of the particular transient behaviors. First, we identify five typical flow regimes through the traditional pressure transient analysis method for horizontal wells which gives us a basic understanding of the characteristics of the pressure response curve. Second, assisted with the corresponding analysis method, we figure out the durations on the curve that correspond to different component banks. By taking the complex displacement mechanisms into consideration, we find that the component banks have a large influence on the curve and identify the root cause of each unique characteristic. Besides, we conduct a systematic sensitivity analysis with respect to multiple parameters such as miscible condition, wellbore storage coefficient, skin factor, horizontal well length, anisotropy, and amount of injected CO 2 . Finally, we have a better understanding of the transient pressure behavior of horizontal wells during CO 2 -EOR, find a way to determine the miscibility underground, and feel more confident in applying the pressure transient data for analysis and parameter estimation. • The EoS based compositional model and wellbore storage model are coupled in a fully-implicit parallel reservoir simulator. • A new interpretation method is proposed for the pressure transient analysis of horizontal wells during CO 2 -EOR. • The effects of flow patterns and complex displacement mechanisms on transient behavior are analyzed in detail. • The results point out a cheap and reliable way to monitor the CO 2 displacement process in horizontal well patterns.

Parallel multilevel domain decomposition preconditioners for monolithic solution of non-isothermal flow in reservoir simulation

In reservoir simulation, the non-isothermal flow adds a conservation of the energy equation with the involvement of a temperature variable to the complicated physics dynamics, which brings a higher nonlinearity of the corresponding PDE system and therefore exhibits significantly additional challenges on the linear preconditioner of the simulator. However, the commonly used block preconditioning techniques mostly focus on the isothermal petroleum model and do not have a specific treatment of the energy conservation equation for the non-isothermal flow. In this study, we propose and develop a family of multilevel restricted additive Schwarz (MRAS) preconditioners on parallel computers for the fully implicit solution of the non-isothermal flow in porous media. The proposed parallel reservoir simulator incorporates the restricted additive Schwarz preconditioner into a general multilevel preconditioning framework with the use of the domain decomposition technique and the multigrid method, which allows us to flexibly construct several types of multilevel Schwarz preconditioners by using the additive or multiplicative strategies. In particular, it follows the fully implicit discretization scheme for the monolithic solution, and therefore is unconditionally stable with the relaxation of the time step size by the stability condition. We numerically show that the proposed approach is highly robust and efficient for solving some non-isothermal flow problems with high nonlinearity, and good parallel scalabilities are obtained on a supercomputer with a large number of processors.

On the impact of heterogeneity-aware mesh partitioning and non-contributing computation removal on parallel reservoir simulations

Parallel computations have become standard practice for simulating the complicated multi-phase flow in a petroleum reservoir. Increasingly sophisticated numerical techniques have been developed in this context. During the chase of algorithmic superiority, however, there is a risk of forgetting the ultimate goal, namely, to efficiently simulate real-world reservoirs on realistic parallel hardware platforms. In this paper, we quantitatively analyse the negative performance impact caused by non-contributing computations that are associated with the “ghost computational cells” per subdomain, which is an insufficiently studied subject in parallel reservoir simulation. We also show how these non-contributing computations can be avoided by reordering the computational cells of each subdomain, such that the ghost cells are grouped together. Moreover, we propose a new graph-edge weighting scheme that can improve the mesh partitioning quality, aiming at a balance between handling the heterogeneity of geological properties and restricting the communication overhead. To put the study in a realistic setting, we enhance the open-source Flow simulator from the OPM framework, and provide comparisons with industrial-standard simulators for real-world reservoir models.

A 6M digital twin for modeling and simulation in subsurface reservoirs

A 6M digital twin for modeling and simulation in subsurface reservoirs

Parallel reservoir simulators for fully implicit complementarity formulation of multicomponent compressible flows

The numerical simulation of multicomponent compressible flow in porous media is an important research topic in reservoir modeling. Traditional semi-implicit methods for such problems are usually conditionally stable, suffer from large splitting errors, and may accompany with violations of the boundedness requirement of the numerical solution. In this study we reformulate the original multicomponent equations into a nonlinear complementarity problem and discretize it using a fully implicit finite element method. We solve the resultant nonsmooth nonlinear system of equations arising at each time step by a parallel, scalable, and nonlinearly preconditioned semismooth Newton algorithm, which is able to preserve the boundedness of the solution and meanwhile treats the possibly imbalanced nonlinearity of the system. Some numerical results are presented to demonstrate the robustness and efficiency of the proposed algorithm on the Tianhe-2 supercomputer for both standard benchmarks as well as realistic problems in highly heterogeneous media.

Parallel Reservoir Simulation with OpenACC and Domain Decomposition

Parallel reservoir simulation is an important approach to solving real-time reservoir management problems. Recently, there is a new trend of using a graphics processing unit (GPU) to parallelize the reservoir simulations. Current GPU-aided reservoir simulations focus on compute unified device architecture (CUDA). Nevertheless, CUDA is not functionally portable across devices and incurs high amount of code. Meanwhile, domain decomposition is not well used for GPU-based reservoir simulations. In order to address the problems, we propose a parallel method with OpenACC to accelerate serial code and reduce the time and effort during porting an application to GPU. Furthermore, the GPU-aided domain decomposition is developed to accelerate the efficiency of reservoir simulation. The experimental results indicate that (1) the proposed GPU-aided approach can outperform the CPU-based one up to about two times, meanwhile with the help of OpenACC, the workload of the transplant code was reduced significantly by about 22 percent of the source code, (2) the domain decomposition method can further improve the execution efficiency up to 1.7×. The proposed parallel reservoir simulation method is a efficient tool to accelerate reservoir simulation.

Development of a framework for parallel reservoir simulation

The development of a parallel reservoir simulator is a more complex task than nonparallel simulators. Problems related to parallel implementation such as parallel communication, model division among processors, and the management of data distributed among processors, and other issues should be addressed and solved in addition to the already complex task of reservoir simulator development. Hence, the development of parallel reservoir simulators is more time intensive than the traditional development on single processor computers. In this work, a framework was developed to aid the development of parallel reservoir simulators. The framework developed in this work implements and handles the parallel processing complexity and provides easy-to-use programming interfaces to accelerate the development of new parallel reservoir simulators or the parallelization of existing ones. Additionally, The University of Texas Compositional Simulator was used in conjunction with the framework developed in this work to create a parallel reservoir simulator. Case studies are used to verify accuracy and to test parallel performance of our new parallel reservoir simulator.

Development of a framework for parallel reservoir simulation

The development of a parallel reservoir simulator is a more complex task than nonparallel simulators. Problems related to parallel implementation such as parallel communication, model division amon...

Efficient CPR-type preconditioner and its adaptive strategies for large-scale parallel reservoir simulations

With the rapid development of high performance parallel computers and the increasing demands of large-scale reservoir simulation, the development of effective and fast solvers and preconditioners for parallel reservoir simulation has become increasingly important. The incomplete LU factorization (ILU) preconditioners are the most commonly used preconditioners in reservoir simulations due to their low computational cost. They have been implemented in commercial simulators as the default preconditioners. However, for reservoirs with highly heterogeneous permeability, the CPR (constrained pressure residual)-type preconditioners are more effective than the ILU preconditioners, especially when parallel computing is employed. The CPR-type preconditioners consist of multi-stages of preconditioning processes, which lead to huge computational costs. Employing the algebraic multigrid (AMG) method to solve a pressure system and the ILU method to solve the whole system has two indispensable stages. For parallel computation, due to the serial nature of the ILU method, the ILU method is usually combined with the restricted additive Schwarz (RAS) method and used as a subdomain solver. Different settings of the AMG and RAS methods, such as a grid coarsening strategy, interpolation and smoothing in the AMG method, and a subdomain solver and an overlap level in the RAS method, as well as different CPR-type preconditioners, yield significantly different performance. This paper first gives an overview of the CPR-type preconditioners and the AMG and RAS methods. Then a detailed comparison between different CPR-type preconditioners with different settings of the AMG and RAS methods is presented. Based on the comparison, the efficient settings of the AMG and RAS methods and the most efficient CPR-type preconditioner are given. Furthermore, an adaptive preconditioning strategy is designed to save the computational time. The adaptive preconditioning strategy introduces an indicator to measure the difficulty level of solving a linear system. According to difficulties, a RAS–ILU preconditioner or a CPR-type preconditioner is automatically and dynamically selected to achieve both efficiency and effectiveness.

Development of a parallel chemical flooding reservoir simulator

Simulation of large-scale and complicated reservoirs requires a large number of gridblocks, which consumes a considerable amount of memory and is computationally expensive. One solution to remedy the computational problem is to take advantage of clusters of CPUs and high-performance computing widely available nowadays. We can run large-scale simulations faster and more efficiently by using parallel processing on these clusters. In this study, we developed a parallel version of an in-house comprehensive chemical flooding reservoir simulator called UTCHEM using MPI functions. Several case studies were used to validate the simulator. The speedup results also show very good parallel efficiency. To the best of our knowledge this is the first time that such a comprehensive chemical reservoir simulator is parallelised, which can be effective in running large-scale cases involving various type of chemical EOR processes, with millions of gridblocks, in few hours. [Received: November 25, 2015; Accepted: June 18, 2016]

High-Performance Modeling of Carbon Dioxide Sequestration by Coupling Reservoir Simulation and Molecular Dynamics

Summary The present work describes a parallel computational framework for carbon dioxide (CO2) sequestration simulation by coupling reservoir simulation and molecular dynamics (MD) on massively parallel high-performance-computing (HPC) systems. In this framework, a parallel reservoir simulator, reservoir-simulation toolbox (RST), solves the flow and transport equations that describe the subsurface flow behavior, whereas the MD simulations are performed to provide the required physical parameters. Technologies from several different fields are used to make this novel coupled system work efficiently. One of the major applications of the framework is the modeling of large-scale CO2 sequestration for long-term storage in subsurface geological formations, such as depleted oil and gas reservoirs and deep saline aquifers, which has been proposed as one of the few attractive and practical solutions to reduce CO2 emissions and address the global-warming threat. Fine grids and accurate prediction of the properties of fluid mixtures under geological conditions are essential for accurate simulations. In this work, CO2 sequestration is presented as a first example for coupling reservoir simulation and MD, although the framework can be extended naturally to the full multiphase multicomponent compositional flow simulation to handle more complicated physical processes in the future. Accuracy and scalability analysis are performed on an IBM BlueGene/P and on an IBM BlueGene/Q, the latest IBM supercomputer. Results show good accuracy of our MD simulations compared with published data, and good scalability is observed with the massively parallel HPC systems. The performance and capacity of the proposed framework are well-demonstrated with several experiments with hundreds of millions to one billion cells. To the best of our knowledge, the present work represents the first attempt to couple reservoir simulation and molecular simulation for large-scale modeling. Because of the complexity of subsurface systems, fluid thermodynamic properties over a broad range of temperature, pressure, and composition under different geological conditions are required, although the experimental results are limited. Although equations of state can reproduce the existing experimental data within certain ranges of conditions, their extrapolation out of the experimental data range is still limited. The present framework will definitely provide better flexibility and predictability compared with conventional methods.

The application of static load balancers in parallel compositional reservoir simulation on distributed memory system

Compositional reservoir simulation depicts the complex behaviors of all the components in gaseous, liquid, and oil phases. It helps to understand the dynamic changes in reservoirs. Parallel computing is implemented to speed up simulation in large scale fields. However, there is still many challenges in obtaining efficient and cost-effective parallel reservoir simulation. Load imbalance on processors in the parallel machine is a major problem and it severely affects the performance of parallel implementation in compositional reservoir simulators. This article presents a new approach to the reduction of load imbalance among processors in large scale parallel compositional reservoir simulation. The approach is based on graph partitioning techniques: Metis partitioning and spectral partitioning. These techniques treat the simulation grid, or the mesh, as a graph constituted by vertices and edges, and then partition the graph into smaller domains. Metis and spectral partitioning techniques are advantageous because they take into account the potential computational load of each grid block in the mesh and generate smaller partitions for heavy computational load areas and larger partitions for light computational load areas. In our case, the computational load is represented by transmissibility. After new partitions are generated, each of them is assigned to a processor in the parallel machine and new parallel reservoir simulation can be conducted. Traditionally, the intuitive 2D decomposition is frequently used to partition the simulation grid into small rectangles, and this is a major source of load imbalance. The performance of parallel compositional simulation based on our partitioning techniques is compared with the most commonly used 2D decomposition and it is found that load imbalance in our new simulations is reduced when compared with the traditional 2D decomposition. This study improves the efficiency of compositional simulation and eventually makes it more cost-effective for hydrocarbon simulation on mega-scale reservoir models.

A family of constrained pressure residual preconditioners for parallel reservoir simulations

SummaryLarge‐scale reservoir simulations are extremely time‐consuming because of the solution of large‐scale linear systems arising from the Newton or Newton–Raphson iterations. The problem becomes even worse when highly heterogeneous geological models are employed. This paper introduces a family of multi‐stage preconditioners for parallel black oil simulations, which are based on the famous constrained pressure residual preconditioner. Numerical experiments demonstrate that our preconditioners are robust, efficient, and scalable. Copyright © 2015 John Wiley & Sons, Ltd.

A New Approach to Load Balance for Parallel/Compositional Simulation Based on Reservoir-Model Overdecomposition

Summary The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high-performance computing architectures. Among the various challenges of efficiency and scalability, load imbalance is a major obstacle that has not been fully addressed and solved. The causes of load imbalance in parallel reservoir simulation are both static and dynamic. Robust graph-partitioning algorithms are capable of handling static load imbalance by decomposing the underlying reservoir geometry to distribute a roughly equal load to each processor. However, these loads that are determined by a static load balancer seldom remain unchanged as the simulation proceeds in time. This so-called dynamic imbalance can be exacerbated further in parallel compositional simulations. The flash calculations for equations of state (EOSs) in complex compositional simulations not only can consume more than half of the total execution time but also are difficult to balance merely by a static load balancer. The computational cost of flash calculations in each gridblock heavily depends on the dynamic data such as pressure, temperature, and hydrocarbon composition. Thus, any static assignment of gridblocks may lead to dynamic load imbalance in unpredictable manners. A dynamic load balancer can often provide solutions for this difficulty. However, traditional techniques are inflexible and tedious to implement in legacy reservoir simulators. In this paper, we present a new approach to address dynamic load imbalance in parallel compositional simulation. It overdecomposes the reservoir model to assign each processor a bundle of subdomains. Processors treat these bundles of subdomains as virtual processes or user-level migratable threads that can be dynamically migrated across processors in the run-time system. This technique is shown to be capable of achieving better overlap between computation and communication for cache efficiency. We use this approach in a legacy reservoir simulator and demonstrate a reduction in the execution time of parallel compositional simulations while requiring minimal changes to the source code. Finally, it is shown that domain overdecomposition, together with a load balancer, can improve speedup from 29.27 to 62.38 on 64 physical processors for a realistic simulation problem.

Coupling Equation-of-State Compositional and Surfactant Models in a Fully Implicit Parallel Reservoir Simulator Using the Equivalent-Alkane-Carbon-Number Concept

Summary Equation-of-state (EOS) compositional and surfactant models are coupled in a fully implicit parallel reservoir simulator using the equivalent alkane carbon number (EACN) of the oleic phase. The EACN of the oleic phase is computed using a mole-fraction-weighted carbon number for each component present in the oleic phase. Important microemulsion properties such as optimum salinity and optimum solubilization parameter as a function of the EACN of the oleic phase are implemented on the basis of known correlations. Type II(-) surfactant phase behavior is considered in this study. The simulator developed is validated using our implicit-pressure/explicit-concentration (IMPEC) chemical-flooding simulator. Case studies, including a large-scale simulation, emphasize that surfactant floods should be modeled carefully, taking the EACN of crude oil into consideration for more realistic and accurate oil-recovery predictions.

A Fully Implicit, Compositional, Parallel Simulator for IOR Processes in Fractured Reservoirs

Summary Naturally fractured reservoirs contain a significant amount of world oil reserves. A number of these reservoirs contain several billion barrels of oil. Accurate and efficient reservoir simulation of naturally fractured reservoirs is one of the most important, challenging, and computationally intensive problems in reservoir engineering. Parallel reservoir simulators developed for naturally fractured reservoirs can effectively address the computational problem. A new accurate parallel simulator for large-scale naturally fractured reservoirs, capable of modeling fluid flow in both rock matrix and fractures, has been developed. The simulator is a parallel, 3D, fully implicit, equation-of-state compositional model that solves very large, sparse linear systems arising from discretization of the governing partial differential equations. A generalized dual-porosity model, the multiple-interacting-continua (MINC), has been implemented in this simulator. The matrix blocks are dis-cretized into subgrids in both horizontal and vertical directions to offer a more accurate transient flow description in matrix blocks. We believe this implementation has led to a unique and powerful reservoir simulator that can be used by small and large oil producers to help them in the design and prediction of complex gas and waterflooding processes on their desktops or a cluster of computers. Some features of this simulator, such as modeling both gas and water processes and the ability of 2D matrix subgridding are not available in any commercial simulator to the best of our knowledge. The code was developed on a cluster of processors, which has proven to be a very efficient and convenient resource for developing parallel programs. The results were successfully verified against analytical solutions and commercial simulators (ECLIPSE and GEM). Excellent results were achieved for a variety of reservoir case studies. Applications of this model for several IOR processes (including gas and water injection) are demonstrated. Results from using the simulator on a cluster of processors are also presented. Excellent speedup ratios were obtained.

Iterative Solution Methods for Modeling Multiphase Flow in Porous Media Fully Implicitly

We discuss several fully implicit techniques for solving the nonlinear algebraic system arising in an expanded mixed finite element or cell-centered finite difference discretization of two- and three-phase porous media flow. Every outer nonlinear Newton iteration requires solution of a nonsymmetric Jacobian linear system. Two major types of preconditioners, supercoarsening multigrid (SCMG) and two-stage, are developed for the GMRES iteration applied to the solution of the Jacobian system. The SCMG reduces the three-dimensional system to two dimensions using a vertical aggregation followed by a two-dimensional multigrid. The two-stage preconditioners are based on decoupling the system into a pressure and concentration equations. Several pressure preconditioners of different types are described. Extensive numerical results are presented using the integrated parallel reservoir simulator (IPARS) and indicate that these methods have low arithmetical complexity per iteration and good convergence rates.