Distributed Memory Parallel Computers Research Articles

Field-split preconditioned active-set reduced-space algorithm for complex black oil reservoir simulation at large-scale

Accurate black oil reservoir simulation is important for the understanding of underground resources in hydrology and petroleum reservoir engineering. The success of fast prediction relies on a robust and scalable black oil simulator, which can handle the complex subsurface fluid dynamics with coupling several different physical processes, and meanwhile run efficiently on distributed memory parallel computers. In this work, we introduce a highly parallel simulator to accurately compute the reservoir flow coupling with wells, including some adaptive fully implicit schemes for the temporal discretization, the improved active-set reduced-space algorithm for the nonlinear algebraic system, and several types of field-split methods for preconditioning. This parallel simulator is applied on the domain decomposition framework, which is designed to overcome the computational issues of traditional simulators implemented for personal computers or small workstations. High-resolution reservoir simulation results on the Society of Petroleum Engineers (SPE) comparative solution project or real-field cases are obtained and analyzed, and the parallel performance of the algorithm is studied on a supercomputer with scaling up to 27440 processor cores. The numerical experiments indicate that the proposed black oil simulator is capable of accurately predicting the highly complex physical processes and interactions between wells and the reservoir for full field scale simulation.

Alya toward exascale: algorithmic scalability using PSCToolkit

In this paper, we describe an upgrade of the Alya code with up-to-date parallel linear solvers capable of achieving reliability, efficiency and scalability in the computation of the pressure field at each time step of the numerical procedure for solving a Large Eddy Simulation formulation of the incompressible Navier–Stokes equations. We developed a software module in the Alya’s kernel to interface the libraries included in the current version of PSCToolkit, a framework for the iterative solution of sparse linear systems, on parallel distributed-memory computers, by Krylov methods coupled to Algebraic MultiGrid preconditioners. The Toolkit has undergone various extensions within the EoCoE-II project with the primary goal of facing the exascale challenge. Results on a realistic benchmark for airflow simulations in wind farm applications show that the PSCToolkit solvers significantly outperform the original versions of the Conjugate Gradient method available in the Alya’s kernel in terms of scalability and parallel efficiency and represent a very promising software layer to move the Alya code toward exascale.

A numerical solver for active hydrodynamics in three dimensions and its application to active turbulence

We present a higher-order convergent numerical solver for active polar hydrodynamics in three-dimensional domains of arbitrary shape, along with a scalable open-source software implementation for shared- and distributed-memory parallel computers. This enables the computational study of the nonlinear dynamics of out-of-equilibrium materials from first principles. We numerically solve the nonlinear active Ericksen–Leslie hydrodynamic equations of three-dimensional (3D) active nematics using both a meshfree and a hybrid particle-mesh method in either the Eulerian or Lagrangian frame of reference. The solver is validated against a newly derived analytical solution in 3D and implemented using the OpenFPM software library for scalable scientific computing. We then apply the presented method to studying the transition of 3D active polar fluids to spatiotemporal chaos, the emergence of coherent angular motion in a 3D annulus, and chiral vortices in symmetric and asymmetric 3D shapes resembling dividing cells. Overall, this provides a robust and efficient open-source simulation framework for 3D active matter with verified numerical convergence and scalability on parallel computers.

Adaptive density-based robust topology optimization under uncertain loads using parallel computing

This work presents an efficient parallel implementation of density-based robust topology optimization (RTO) using adaptive mesh refinement (AMR) schemes permitting us to address the problem with modest computational resources. We use sparse grid stochastic collocation methods (SCMs) for transforming the RTO problem into a weighted multiple-loading deterministic problem at the collocation points. The calculation of these deterministic problems and the functional sensitivity is computationally expensive. We combine distributed-memory parallel computing and AMR techniques to address the problem efficiently. The former allows us to exploit the computational resources available, whereas the latter permits us to increase performance significantly. We propose the parallel incremental calculation of the deterministic problems and the contribution to the functional sensitivity maintaining a similar memory allocation to the one used in the deterministic counterpart. The cumulative computing uses buffers to adapt the evaluation at the collocation points to the parallel computing resources permitting the exploitation of the embarrassing parallelism of SCMs. We evaluate the deterministic problems in a coarse mesh generated for each topology optimization iteration to increase the performance. We perform the regularization and design variable update in a fine mesh to obtain an equivalent design to the one generated in such a mesh. We evaluate the proposal in two- and three-dimensional problems to test its feasibility and scalability. We also check the performance improvement using computational buffers in parallel computing nodes. Finally, we compare the proposal to the same approach using different preconditioners without AMR schemes showing significant performance improvements.

Introducing MPEC: Massively parallel electron correlation.

We have developed a new program for carrying out improved internally contracted Multi-reference Configuration Interaction Singles and Doubles (i2cMRCISD) calculations. It is designed from the ground up to be used on distributed memory parallel computers. Tests show good scaling properties with the number of cores per node and the number nodes. This program features Gaussian basis sets with ℓ > 6; scalar special relativity via the spin-free method; convergence to C∞v, D∞v, or spherical electronic states; special code to determine Rydberg orbitals; both uncontracted and contracted MRCISD wavefunctions; one and two electron properties, including full spin-orbit matrix elements with the Breit interaction; analytic calculation of Born-Oppenheimer diagonal correction for multi-configuration Hartree-Fock wavefunctions; and analytic calculation of second order Born-Oppenheimer corrections for Hartree-Fock wavefunctions. The program can be obtained from software.nasa.gov.

ReMo3D – an open-source Python package for 2D and 3D simulation of normal and lateral resistivity logs

An open-source Python package is presented, ReMo3D, which allows the generation of synthetic normal and lateral resistivity logs for 2D and 3D models. The package is built around a finite element mesh generator Gmsh and a high-performance multiphysics finite element software Netgen/NGSolve and supports distributed-memory parallel computation. The examples included in the paper show that the developed software can accurately simulate the measurement process and produce detailed synthetic normal and lateral resistivity logs. In addition, basic information about normal and lateral tools such as tool configurations, measurement principles, nomenclature and a brief history of utilization is included in the paper.

The cycle-structure connectivity of crossed cubes

Crossed cube is a kind of network structure as a basis for distributed memory parallel computer architecture. Connectivity plays an important role in measuring the fault-tolerance of an interconnection network. As a generalization, Lin et al. [6] introduced the concept of structure connectivity in 2016. For connected graphs G and T, the T-structure connectivity κ(G;T) of G is the minimum cardinality of a set of subgraphs F of G that each member is isomorphic to T so that G−F is a disconnected or a trivial graph. In this paper, we study cycle-structure connectivity of n-dimensional crossed cubes CQn. We show that κ(CQn;C2k−1)=⌈nk−1⌉ for 4⩽k⩽n and κ(CQn;C2k)=⌊nk⌋+1 for even k with 4⩽k⩽n. Furthermore, we also obtain that κ(CQn;C6)=⌊n4⌋+⌈n−3⌊n4⌋2⌉ for n⩾4.

An Efficient Algorithm for Embedding Two-Dimensional Tori into Balanced Hypercubes

Balanced hypercube is a variant of the hypercube structure; it offers desirable properties such as connectivity, diameter, and fault-tolerance. Cycles and tori are popular interconnection topologies and have been widely used in distributed-memory parallel computers. Scholars have developed diverse and numerous parallel algorithms of cycles and tori. This paper proposes an efficient algorithm, called reflected edge label sequence, to generate cycle structures in balanced hypercubes. With this algorithm, we can embed cycles of length [Formula: see text], [Formula: see text], into an [Formula: see text]-dimensional balanced hypercube. The corresponding algorithm for embedding 2-dimensional [Formula: see text] tori can be constructed on the basis of this approach, and the time complexity is linear with respect to the size of a balanced hypercube.

Development of a Scalable Thermal Reservoir Simulator on Distributed-Memory Parallel Computers

Reservoir simulation is to solve a set of fluid flow equations through porous media, which are partial differential equations from the petroleum engineering industry and described by Darcy’s law. This paper introduces the model, numerical methods, algorithms and parallel implementation of a thermal reservoir simulator that is designed for numerical simulations of a thermal reservoir with multiple components in three-dimensional domain using distributed-memory parallel computers. Its full mathematical model is introduced with correlations for important properties and well modeling. Efficient numerical methods (discretization scheme, matrix decoupling methods, and preconditioners), parallel computing technologies, and implementation details are presented. The numerical methods applied in this paper are suitable for large-scale thermal reservoir simulations with dozens of thousands of CPU cores (MPI processes), which are efficient and scalable. The simulator is designed for giant models with billions or even trillions of grid blocks using hundreds of thousands of CPUs, which is our main focus. The validation part is compared with CMG STARS, which is one of the most popular and mature commercial thermal simulators. Numerical experiments show that our results match commercial simulators, which confirms the correctness of our methods and implementations. SAGD simulation with 7406 well pairs is also presented to study the effectiveness of our numerical methods. Scalability testings demonstrate that our simulator can handle giant models with billions of grid blocks using 100,800 CPU cores and the simulator has good scalability.

Distributed memory parallel computing of three-dimensional variable-density groundwater flow and salt transport

Fresh groundwater reserves, being of vital importance for more than a billion of people living in the coastal zone, are being threatened by saltwater intrusion due to anthropogenic activities and climate change. High resolution three-dimensional (3D), variable-density (VD), groundwater flow and salt transport (FT) numerical models are increasingly being used to support water managers and decision makers in their strategic planning and measures for dealing with the problem of fresh water shortages. However, these computer models typically require long runtimes and large memory usage, making them impractical to use without parallelization. Here, we parallelize SEAWAT, and show that with our parallelization 3D-VD-FT modeling is now feasible for a wide range of hydrogeologists, since a) speedups of more than two orders of magnitude can be obtained as illustrated in this paper, and b) large 3D-VD-FT models are feasible with memory requirements far exceeding single machine memory.

GPU-acceleration of the ELPA2 distributed eigensolver for dense symmetric and hermitian eigenproblems

The solution of eigenproblems is often a key computational bottleneck that limits the tractable system size of numerical algorithms, among them electronic structure theory in chemistry and in condensed matter physics. Large eigenproblems can easily exceed the capacity of a single compute node, thus must be solved on distributed-memory parallel computers. We here present GPU-oriented optimizations of the ELPA two-stage tridiagonalization eigensolver (ELPA2). On top of cuBLAS-based GPU offloading, we add a CUDA kernel to speed up the back-transformation of eigenvectors, which can be the computationally most expensive part of the two-stage tridiagonalization algorithm. We benchmark the performance of this GPU-accelerated eigensolver on two hybrid CPU–GPU architectures, namely a compute cluster based on Intel Xeon Gold CPUs and NVIDIA Volta GPUs, and the Summit supercomputer based on IBM POWER9 CPUs and NVIDIA Volta GPUs. Consistent with previous benchmarks on CPU-only architectures, the GPU-accelerated two-stage solver exhibits a parallel performance superior to the one-stage counterpart. Finally, we demonstrate the performance of the GPU-accelerated eigensolver developed in this work for routine semi-local KS-DFT calculations comprising thousands of atoms.

PaKman: A Scalable Algorithm for Generating Genomic Contigs on Distributed Memory Machines

De novo genome assembly is a fundamental problem in the field of bioinformatics, that aims to assemble the DNA sequence of an unknown genome from numerous short DNA fragments (aka reads) obtained from it. With the advent of high-throughput sequencing technologies, billions of reads can be generated in a matter of hours, necessitating efficient parallelization of the assembly process. While multiple parallel solutions have been proposed in the past, conducting a large-scale assembly at scale remains a challenging problem because of the inherent complexities associated with data movement, and irregular access footprints of memory and I/O operations. In this article, we present a novel algorithm, called PaKman , to address the problem of performing large-scale genome assemblies on a distributed memory parallel computer. Our approach focuses on improving performance through a combination of novel data structures and algorithmic strategies for reducing the communication and I/O footprint during the assembly process. PaKman presents a solution for the two most time-consuming phases in the full genome assembly pipeline, namely, k-mer counting and contig generation . A key aspect of our algorithm is its graph data structure (PaK-Graph), which comprises fat nodes (or what we call “macro-nodes”) that reduce the communication burden during contig generation. We present an extensive performance and qualitative evaluation of our algorithm across a wide range of genomes (varying in both size and species group), including comparisons to other state-of-the-art parallel assemblers. Our results demonstrate the ability to achieve near-linear speedups on up to 16K cores (tested) on the NERSC Cori supercomputer; perform better than or comparable to other state-of-the-art distributed memory and shared memory tools in terms of performance while delivering comparable (if not better) quality; and reduce time to solution significantly. For instance, PaKman is able to generate a high-quality set of assembled contigs for complex genomes such as the human and bread wheat genomes in under a minute on 16K cores. In addition, PaKman was able to successfully process a 3.1 TB simulated dataset of one of the largest known genomes (to date)- Ambystoma mexicanum (the axolotl), in just over 200 seconds on 16K cores.

Performance Metrics of Heterogeneous Distributed Memory Parallel Computer System Using Recursive Models

Performance Metrics of Heterogeneous Distributed Memory Parallel Computer System Using Recursive Models

A numerical study of the pollution error and DPG adaptivity for long waveguide simulations

High-frequency wave propagation has many important applications in acoustics, elastodynamics, and electromagnetics. Unfortunately, the finite element discretization for these problems suffers from significant numerical pollution errors that increase with the wavenumber. It is critical to control these errors to obtain a stable and accurate method. We study the effect of pollution for very long waveguide problems in the context of robust discontinuous Petrov–Galerkin (DPG) finite element discretizations. Our numerical experiments show that the pollution primarily has a diffusive effect causing energy loss in the DPG method while phase errors appear less significant. We report results for 3D vectorial time-harmonic Maxwell problems in waveguides with more than 8000 wavelengths. Our results corroborate previous analysis for the Galerkin discretization of the Helmholtz operator by Melenk and Sauter (2011). Additionally, we discuss adaptive refinement strategies for multi-mode fiber waveguides where the propagating transverse modes must be resolved sufficiently. Our study shows the applicability of the DPG error indicator to this class of problems. Finally, we illustrate the importance of load balancing in these simulations for distributed-memory parallel computing.

The scalability analysis of a parallel tree search algorithm

Increasing the number of computational cores is a primary way of achieving the high performance of contemporary supercomputers. However, developing parallel applications capable to harness the enormous amount of cores is a challenging task. It is very important to understand the principle limitations of the scalability of parallel applications imposed by the algorithm’s structure. The tree search addressed in this paper has an irregular structure unknown prior to computations. That is why such algorithms are challenging for parallel implementation especially on distributed memory systems. In this paper, we propose a parallel tree search algorithm aimed at distributed memory parallel computers. For this parallel algorithm, we analyze its scalability and show that it is close to the theoretical maximum.

PSpatiocyte: a high-performance simulator for intracellular reaction-diffusion systems

BackgroundStudies using quantitative experimental methods have shown that intracellular spatial distribution of molecules plays a central role in many cellular systems. Spatially resolved computer simulations can integrate quantitative data from these experiments to construct physically accurate models of the systems. Although computationally expensive, microscopic resolution reaction-diffusion simulators, such as Spatiocyte can directly capture intracellular effects comprising diffusion-limited reactions and volume exclusion from crowded molecules by explicitly representing individual diffusing molecules in space. To alleviate the steep computational cost typically associated with the simulation of large or crowded intracellular compartments, we present a parallelized Spatiocyte method called pSpatiocyte.ResultsThe new high-performance method employs unique parallelization schemes on hexagonal close-packed (HCP) lattice to efficiently exploit the resources of common workstations and large distributed memory parallel computers. We introduce a coordinate system for fast accesses to HCP lattice voxels, a parallelized event scheduler, a parallelized Gillespie’s direct-method for unimolecular reactions, and a parallelized event for diffusion and bimolecular reaction processes. We verified the correctness of pSpatiocyte reaction and diffusion processes by comparison to theory. To evaluate the performance of pSpatiocyte, we performed a series of parallelized diffusion runs on the RIKEN K computer. In the case of fine lattice discretization with low voxel occupancy, pSpatiocyte exhibited 74% parallel efficiency and achieved a speedup of 7686 times with 663552 cores compared to the runtime with 64 cores. In the weak scaling performance, pSpatiocyte obtained efficiencies of at least 60% with up to 663552 cores. When executing the Michaelis-Menten benchmark model on an eight-core workstation, pSpatiocyte required 45- and 55-fold shorter runtimes than Smoldyn and the parallel version of ReaDDy, respectively. As a high-performance application example, we study the dual phosphorylation-dephosphorylation cycle of the MAPK system, a typical reaction network motif in cell signaling pathways.ConclusionspSpatiocyte demonstrates good accuracies, fast runtimes and a significant performance advantage over well-known microscopic particle methods in large-scale simulations of intracellular reaction-diffusion systems. The source code of pSpatiocyte is available at https://spatiocyte.org.

Legacy code and parallel computing: updating and parallelizing a numerical model

In this paper, we present several important details in the process of legacy code parallelization, mostly related to the problem of maintaining numerical output of a legacy code while obtaining a balanced workload for parallel processing. Since we maintained the non-uniform mesh imposed by the original finite element code, we have to develop a specially designed data distribution among processors so that data restrictions are met in the finite element method. In particular, we introduce a data distribution method that is initially used in shared memory parallel processing and obtain better performance than the previous parallel program version. Besides, this method can be extended to other parallel platforms such as distributed memory parallel computers. We present results including several problems related to performance profiling on different (development and production) parallel platforms. The use of new and old parallel computing architectures leads to different behavior of the same code, which in all cases provides better performance in multiprocessor hardware.

Techniques for high-performance construction of Fock matrices.

This paper presents techniques for Fock matrix construction that are designed for high performance on shared and distributed memory parallel computers when using Gaussian basis sets. Four main techniques are considered. (1) To calculate electron repulsion integrals, we demonstrate batching together the calculation of multiple shell quartets of the same angular momentum class so that the calculation of large sets of primitive integrals can be efficiently vectorized. (2) For multithreaded summation of entries into the Fock matrix, we investigate using a combination of atomic operations and thread-local copies of the Fock matrix. (3) For distributed memory parallel computers, we present a globally accessible matrix class for accessing distributed Fock and density matrices. The new matrix class introduces a batched mode for remote memory access that can reduce the synchronization cost. (4) For density fitting, we exploit both symmetry (of the Coulomb and exchange matrices) and sparsity (of 3-index tensors) and give a performance comparison of density fitting and the conventional direct calculation approach. The techniques are implemented in an open-source software library called GTFock.

The Effect of Shear Sliding of Vertical Contraction Joints on Seismic Response of High Arch Dams with Fine Finite Element Model

The contraction joints of arch dams with and without shear keys are simplified to be with no‐slip condition and with relative sliding condition, respectively. Based on the Lagrange multiplier method, a contact model considering the manner of independent cantilever dead load type with no‐slip condition and relative sliding condition is proposed to model the nonlinearities of vertical contraction joins, which is special to the nonlinear analysis of arch dams considering the manner of dead load type. Different from the conventional Gauss iterative method, the strategy of the alternating iterative solution of normal force and tangential force is employed. The parallelization based on overlapping domain decomposition method (ODDM) and explicit message passing using distributed memory parallel computers is employed to improve the computational efficiency. An existing high arch dam with fine finite element model is analyzed to investigate the effect of shear sliding of vertical joints on seismic response of the arch dam. The result shows that the values of maximum principal tensile stress under relative sliding condition are significantly greater than those under no‐slip condition.

Numerical simulations of polymer flooding process in porous media on distributed-memory parallel computers

Polymer flooding is a mature enhanced oil recovery technique that has been successfully applied in many field projects. By injecting polymer into a reservoir, the viscosity of water is increased, and the efficiency of water flooding is improved. As a result, more oil can be recovered. This paper presents numerical simulations of a polymer flooding process using parallel computers, where the numerical modeling of polymer retention, inaccessible pore volumes, a permeability reduction and polymer absorption are considered. Darcy's law is employed to model the behavior of a fluid in porous media, and the upstream finite difference (volume) method is applied to discretize the mass conservation equations. Numerical methods, including discretization schemes, linear solver methods, nonlinearization methods and parallel techniques are introduced. Numerical experiments show that, on one hand, computed results match those from the commercial simulator, Eclipse, Schlumberger, which is widely applied by the petroleum industry, and, on the other hand, our simulator has excellent speedup, which is demonstrated by large-scale applications with up to 27 million grid blocks using up to 2048 CPU cores.