Falcon: A Production Quality Distributed Memory Reservoir Simulator

Abstract

Abstract We describe a new production model (Falcon) which has achieved speeds on parallel computers that are 100 times faster than current production models on a vector computer, on real world problems. Falcon has been used to conduct the largest, geostatistical reservoir study ever conducted within Amoco. In the paper we discuss:–Falcon's data parallel paradigm using FORTRAN 90 and High Performance FORTRAN (HPF),–its Single Program, Multiple Data (SPMD) paradigm using message passing,–efficient memory management that enables simulation of enormous studies,–a numerical formulation that reconciles the generalized compositional approach (based on component masses and pressure) with earlier approaches (based on pressures and saturations), in a more general and more efficient approach, - Falcon's scalability up to 512 processor nodes and performance (timings and memory) achieved on a number of parallel platforms. These include Cray Research's T3D and T3E, SGI's Power Challenge and Origin 2000, Thinking Machines' CMS, IBM's SP2. Falcon also runs on single processor computers, such as, PC's and IBM's RS6000.–a new parallel linear solver technology based on a fully parallel scalable implementation of ILU preconditioning coupled with a GMRES or Orthomin iteration process. This naturally ordered global ILU preconditioner is scalable to hundreds of processors, efficiently solving the matrix problems arising from large scale simulations. The use of the techniques described in this paper has enabled us to run problem sizes of up to 16.5 million grid blocks. Falcon was used to simulate fifty geostatistically derived realizations of a large, black oil waterflood system. The realizations, each with 2.3 million cells and 1039 wells, took an average of 4.2 hours to execute on a 128 node CM5 computer, thus enabling the simulation study to finish in less than a month. In this field study, we bypassed upscaling through the use of fine vertical resolution gridding. The focus throughout has been the applicability to real world problems. Falcon can be used for modeling both small and very large reservoirs, including reservoirs characterized by geostatistics. It can be used to simulate black oil, gas-water and dry gas reservoirs. A fully compositional feature is being developed. Introduction The advances in reservoir characterization techniques such as geostatistics, and the increasing emphasis on better representation of reservoir geology, have placed enormous demands on reservoir flow simulators. Fortunately, recent years have seen unprecedented improvements in computing technology. An important trend in supercomputing has been the emergence of powerful computers equipped with multiple processors, i.e., parallel computing. Parallel computing holds the promise of being able to flow simulate extremely large problems, and run them efficiently. Amdahl's law says in effect that computational speedup due to parallelization(or vectorization) is limited by the scalar "overhead" that is a necessary part of all simulations. Turning this argument around, larger problems have far smaller overheads on a proportionate basis and are therefore excellent candidates for parallelization. Three years ago, Amoco initiated a strategic project named Falcon to build a reservoir simulator capable of effectively utilizing multiprocessing technology. The simulator has been developed by a team consisting of two reservoir engineers from Amoco, scientists from the Los Alamos National Labs, and Cray Research. In this paper, we describe the development of Falcon, and performance achieved on a number of platforms. P. 7