Leadership-class Supercomputer Research Articles

Massively parallel modeling and inversion of electrical resistivity tomography data using PFLOTRAN

Abstract. Electrical resistivity tomography (ERT) is a broadly accepted geophysical method for subsurface investigations. Interpretation of field ERT data usually requires the application of computationally intensive forward modeling and inversion algorithms. For large-scale ERT data, the efficiency of these algorithms depends on the robustness, accuracy, and scalability on high-performance computing resources. In this regard, we present a robust and highly scalable implementation of forward modeling and inversion algorithms for ERT data. The implementation is publicly available and developed within the framework of PFLOTRAN, an open-source, state-of-the-art massively parallel subsurface flow and transport simulation code. The forward modeling is based on a finite-volume discretization of the governing differential equations, and the inversion uses a Gauss–Newton optimization scheme. To evaluate the accuracy of the forward modeling, two examples are first presented by considering layered (1D) and 3D earth conductivity models. The computed numerical results show good agreement with the analytical solutions for the layered earth model and results from a well-established code for the 3D model. Inversion of ERT data, simulated for a 3D model, is then performed to demonstrate the inversion capability by recovering the conductivity of the model. To demonstrate the parallel performance of PFLOTRAN's ERT process model and inversion capabilities, large-scale scalability tests are performed by using up to 131 072 processes on a leadership class supercomputer. These tests are performed for the two most computationally intensive steps of the ERT inversion: forward modeling and Jacobian computation. For the forward modeling, we consider models with up to 122 ×106 degrees of freedom (DOFs) in the resulting system of linear equations and demonstrate that the code exhibits almost linear scalability on up to 10 000 DOFs per process. On the other hand, the code shows superlinear scalability for the Jacobian computation, mainly because all computations are fairly evenly distributed over each process with no parallel communication.

Optimal checkpointing for adjoint multistage time-stepping schemes

We consider checkpointing strategies that minimize the number of recomputations needed when performing discrete adjoint computations using multistage time-stepping schemes that require computing several substeps within one complete time step. Specifically, we propose two algorithms that can generate optimal checkpointing schedules under weak assumptions. The first is an extension of the seminal Revolve algorithm adapted to multistage schemes. The second algorithm, named CAMS, is developed based on dynamic programming, and it requires the least number of recomputations when compared with other algorithms. The CAMS algorithm is made publicly available in a library with bindings to C and Python. Numerical results show that the proposed algorithms can deliver up to two times the speedup compared with that of classical Revolve. Moreover, we discuss the utilization of the CAMS library in mature scientific computing libraries and demonstrate the ease of using it in an adjoint workflow. The proposed algorithms have been adopted by the PETScTSAdjoint library. Their performance has been demonstrated with a large-scale PDE-constrained optimization problem on a leadership-class supercomputer. This work is a significant extension of the authors’ conference paper (Zhang and Constantinescu, 2021).

AI-based design of a nuclear reactor core

The authors developed an artificial intelligence (AI)-based algorithm for the design and optimization of a nuclear reactor core based on a flexible geometry and demonstrated a 3× improvement in the selected performance metric: temperature peaking factor. The rapid development of advanced, and specifically, additive manufacturing (3-D printing) and its introduction into advanced nuclear core design through the Transformational Challenge Reactor program have presented the opportunity to explore the arbitrary geometry design of nuclear-heated structures. The primary challenge is that the arbitrary geometry design space is vast and requires the computational evaluation of many candidate designs, and the multiphysics simulation of nuclear systems is very time-intensive. Therefore, the authors developed a machine learning-based multiphysics emulator and evaluated thousands of candidate geometries on Summit, Oak Ridge National Laboratory’s leadership class supercomputer. The results presented in this work demonstrate temperature distribution smoothing in a nuclear reactor core through the manipulation of the geometry, which is traditionally achieved in light water reactors through variable assembly loading in the axial direction and fuel shuffling during refueling in the radial direction. The conclusions discuss the future implications for nuclear systems design with arbitrary geometry and the potential for AI-based autonomous design algorithms.

PittPack: An open-source Poisson’s equation solver for extreme-scale computing with accelerators

We present a parallel implementation of a direct solver for Poisson’s equation on extreme-scale supercomputers with accelerators. We introduce a chunked-pencil decomposition as the domain-decomposition strategy to distribute work among processing elements to achieve improved scalability at high counts of accelerators. Chunked-pencil decomposition enables overlapping MPI communication and data transfer between the central processing units (CPUs) and the graphics processing units (GPUs). It enables contiguous message transfer among the nodes and improves data locality by keeping neighboring elements in adjacent memory locations while permitting the use of shared memory for certain segments of the algorithm when possible. We study two different communication patterns within the chunked-pencil decomposition. The first pattern fully overlaps the communication with data transfer and aims to speedup the overall turnaround time. The second pattern concentrates on low memory usage and is more network friendly than the first pattern for computations at extreme scale. In our parallel implementation, we interleave OpenACC with MPI to support computations on the GPU or the CPU. The numerical solution and its formal second order of accuracy is verified using the method of manufactured solutions for various combinations of boundary conditions. Additionally, we used PittPack within an incompressible flow solver to further validate its accuracy and as well as demonstrate its versatility as a software package. We performed weak scaling analysis with up to 1.1 trillion Cartesian mesh points distributed over 16384 GPUs on a petascale leadership class supercomputer. Program SummaryProgram Title: PittPackProgram Files doi:http://dx.doi.org/10.17632/59ktwdby4r.1Licensing provisions: GNU General Public License 3Programming language: C++Nature of problem: An elliptic Poisson’s equation for pressure arises in the numerical solution of incompressible fluid flow. This equation needs to be solved accurately to enforce conservation of mass principle.Solution method: The Poisson’s equation is discretized with a second order accurate finite difference scheme. The resulting system of linear equations is solved with a direct method that employs fast Fourier transforms in the horizontal plane and a tridiagonal matrix solver in the third direction.Additional comments including restrictions and unusual features: The mesh has to be directionally uniform. Due to adoption of FFT in the horizontal plane, the boundary conditions in each direction must be of the same type. Each direction in the horizontal plane has to be partitioned by an equal integer number such that n2 parallel processes are deployed at run-time.