Development and Optimization of Parallel Code for Large-Scale Petroleum Reservoir Simulation

Abstract

Abstract This paper discusses the use of large field-scale reservoir simulation to model multiphase fluid flow processes that occur in giant oil reservoirs. The goal of large-scale studies is to model fluid flow with sufficient details to account for reservoir heterogeneity, various reservoir-wellbore configurations, and complex fluid-rock interactions. In this paper, we developed a black-oil reservoir simulator for distributed-memory parallel environment. We ported serial code of a black-oil model to the CRAY T3E and IBM SP2 systems. We analysed the code and benchmarked the performance. To parallelize the code, we used a domain decomposition algorithm, whereby the reservoir is divided into several subdomains, with each subdomain assigned to a separate processor element (PE). The message-passing interface (MPI) is used to exchange information across subdomains. We validated the parallel simulator using data from the Society of Petroleum Engineers comparative solution projects. Because the linear equation solver accounts for over 90% of the CPU time in a typical reservoir simulation run, we evaluated the performance of several parallel algorithms in the project, including LSOR, Gauss-Siedel (GS) and strongly implicit procedure (SIP). We found that the convergence behaviour of the solver depends on the number of processors and on the permeability anisotropy of the reservoir. For the problems we tested, the SIP algorithms provided the best performance. We compared the computational efficiency of the parallel code against the serial code using models containing up to 350,000 grid blocks in 4-, 8-, 16- 32-, 64-, and 80-PE environments. The paper discusses programming and computational performance issues in large-scale reservoir simulation for parallel systems. Introduction The objective of large field-scale reservoir simulation is to model multiphase fluid flow processes that occur in giant oil reservoirs with sufficient details to account for reservoir heterogeneity, various reservoir-wellbore configurations, and complex fluid-rock interactions. The simulations often require using over one million grid blocks to accurately capture reservoir heterogeneity and multiphase flow behaviour in the full-field models. In the past, limited computer resources forced engineers to use coarse grid blocks for field-scale studies at the expense of accuracy. Advances in parallel computing technology provide engineers modern tools to solve previously computationally expensive problems at competitive costs. In this paper, we developed a black-oil reservoir simulator for distributed-memory parallel environment. The serial code of a black-oil model was ported to the CRAY T3E. After pre-parallelization analysis and optimization of the code for a singleprocessor performance(1), we used a multi-level, domain decomposition algorithm to parallelize the code. Domain decomposition involves dividing the reservoir into several subdomains, with each subdomain assigned to a separate processor element (PE). The message-passing interface (MPI) is used to exchange information between the multiple processors. The use of MPI permits the portability of the reservoir simulator from the massively parallel environments to network of workstations. We validated the code and analysed the performance of the parallel simulator using data from the Society of Petroleum Engineers comparative solution projects. The paper is organized in the following sections. First, we discuss the simulator development.