A Scalable Parallel Multi-Purpose Reservoir Simulator

Abstract

Abstract This paper describes the algorithms and implementation of a parallel reservoir simulator designed for, but not limited to, distributed-memory computational platforms that can solve previously prohibitive problems efficiently. The parallel simulator inherits the multipurpose features of the in-house sequential simulator, which is at the core of the new capability. As a result, black-oil, miscible, compositional, and thermal problems can be solved efficiently using this new simulator. A multilevel domain decomposition approach is used. First, the original reservoir is decomposed into several domains, each of which is given to a separate processing node. All nodes then execute computations in parallel, each node on its associated subdomain. The parallel computations include initialization, coefficient generation, linear solution on the sub-domain, and input/output. To enhance the convergence rate, we solve a coarse global problem which is generated via a multigrid-like coarsening procedure. This solution serves as a preconditioner of an outer parallel GMRES loop. The exchange of information across the subdomains, or processors, is achieved using the message passing interface standard, MPI. The use of MPI ensures portability across different computing platforms ranging from massively parallel machines to clusters of workstations. Results indicate the simulator exhibits excellent scalability of the simulator for up to 32 processors on the IBM SP2 system. Scalability results are also presented for a cluster of IBM workstations connected via an ATM (Asynchronous Transfer Mode) communication. The use of ATM for interprocessor communication was found to have a small, but measurable, impact on scaling performance. Introduction The predictive capacity of a reservoir simulator depends first on the quality of the information used, and then it depends on the ability of the computational grid and solution method to describe the flow behavior accurately. The injection of more detail into reservoir description is producing very large models. Scale up technology can be applied to reduce the overall size of the models while preserving the important details of the flow. For large-scale reservoir displacements, the scaled up model itself could consist of millions of gridblocks Flow simulation using models of that size is beyond the current capability of uniprocessor, or even shared-memory multiprocessor, compute platforms. In this work, we describe the development of a parallel multi-purpose reservoir simulator that can solve previously prohibitive problems efficiently. In addition, the parallel simulator provides the means to validate and help improve the scaled-up model by comparing its flow predictions with detailed simulations using the original finer scale description from which it is derived. In the following sections, we give a brief overview of the parallel computing landscape and we adopt a definition for scalability. That is followed by a description of our parallel simulation development strategy and implementation details. Performance results for the parallel simulator are then presented and analyzed. We close with key conclusions. Background The development of application codes for distributed-memory parallel platforms has been, until recently, a high-risk investment, both in terms of capital and manpower. This high-risk environment was due, in large part, to (1) an unstable landscape of parallel computing vendors/machines and (2) a lack of software portability across the various platforms. The focus had been on massively-parallel machines with proprietary architectures that link hundreds, or even thousands, of specially designed processors. Because the processing nodes in these machines tended to have limited computing power and small local memory, massive parallelism, both in terms of total memory and compute power, was achieved by employing thousands of such processors.