Distributed Memory Parallel Research Articles

Proven Distributed Memory Parallelization of Particle Methods

We provide a mathematically proven parallelization scheme for particle methods on distributed-memory computer systems. Particle methods are a versatile and widely used class of algorithms for computer simulations and numerical predictions in various applications, ranging from continuum fluid dynamics to discrete granular flows and molecular dynamics simulations. Particle methods naturally lend themselves to implementation on parallel computing systems. So far, however, a mathematical proof of correctness and equivalence to sequential implementations was only available for shared-memory parallelism. Here, we leverage a formal definition of the algorithmic class of particle methods to provide a proven parallelization scheme for distributed-memory computers. We prove that thus parallelized particle methods on distributed-memory computers are formally equivalent to their sequential counterpart for a well-defined class of particle methods, and we provide analytical expressions for the speed-up and scalability bounds of this class of algorithms in function of their parameters. The parallelization scheme analyzed here is the basis of many real-world software designs for parallel particle methods. The present analysis is, therefore, of direct relevance to existing and new parallel implementations of particle methods and places them on solid theoretical grounds, rationalizing best practices and providing useful scalability and speedup bounds for benchmarking.

Accelerating Fortran codes: A method for integrating Coarray Fortran with CUDA Fortran and OpenMP

Fortran's prominence in scientific computing requires strategies to ensure both that legacy codes are efficient on high-performance computing systems, and that the language remains attractive for the development of new high-performance codes. Coarray Fortran (CAF), part of the Fortran 2008 standard introduced for parallel programming, facilitates distributed memory parallelism with a syntax familiar to Fortran programmers, simplifying the transition from single-processor to multi-processor coding. This research focuses on innovating and refining a parallel programming methodology that fuses the strengths of Intel Coarray Fortran, Nvidia CUDA Fortran, and OpenMP for distributed memory parallelism, high-speed GPU acceleration and shared memory parallelism respectively. We consider the management of pageable and pinned memory, CPU-GPU affinity in NUMA multiprocessors, and robust compiler interfacing with speed optimisation. We demonstrate our method through its application to a parallelised Poisson solver and compare the methodology, implementation, and scaling performance to that of the Message Passing Interface (MPI), finding CAF offers similar speeds with easier implementation. For new codes, this approach offers a faster route to optimised parallel computing. For legacy codes, it eases the transition to parallel computing, allowing their transformation into scalable, high-performance computing applications without the need for extensive re-design or additional syntax.

Local Second Order Mo̷ller-Plesset Theory with a Single Threshold Using Orthogonal Virtual Orbitals: A Distributed Memory Implementation.

In order to alleviate the computational burden associated with superlinear compute scalings with molecular size in electron correlation methods, researchers have developed local correlation methods that wisely treat relatively small contributions as zeros but still yield accurate energy approximation. Such local correlation techniques can also be combined with parallel computing resources to obtain further efficiency and scalability. This work focuses on the distributed memory parallel implementation of a local correlation method for second order Mo̷ller-Plesset (MP2) theory. This method also only has a single threshold to control the dropping of terms and accuracy of different computing kernels in the algorithm. The process partitioning strategy and distributed parallel implementation with the message passing interface (MPI) are discussed. In particular, the algorithm relies on a fixed sparsity pattern matrix multiplication and a corresponding distributed conjugate gradient solver, which exhibits almost linear scaling in both strong and weak scaling analyses. Numerical experiments on a range of molecules, including linear chains and molecules with 2 and 3-dimensional characters, are reported. For example, with only 32 MPI ranks, this MP2 implementation can calculate the correlation energy of vancomycin in def2-TZVP basis within 0.003% accuracy (10-6.5 threshold) in half an hour, where the same problem is unfeasible to solve with sequential or pure shared memory implementations.

A distributed memory parallel randomized Kaczmarz for sparse system of equations

AbstractKaczmarz algorithm is an iterative projection method for solving system of linear equations that arise in science and engineering problems in various application domains. In addition to classical Kaczmarz, there are randomized and parallel variants. The main challenge of the parallel implementation is the dependency of each Kaczmarz iteration on its predecessor. Because of this dependency, frequent communication is required which results in a substantial overhead. In this study, a new distributed parallel method that reduces the communication overhead is proposed. The proposed method partitions the problem so that the Kaczmarz iterations on different blocks are less dependent. A frequency parameter is introduced to see the effect of communication frequency on the performance. The communication overhead is also decreased by allowing communication between processes only if they have shared non‐zero columns. The experiments are performed using problems from various domains to compare the effects of different partitioning methods on the communication overhead and performance. Finally, parallel speedups of the proposed method on larger problems are presented.

Field-split preconditioned active-set reduced-space algorithm for complex black oil reservoir simulation at large-scale

Accurate black oil reservoir simulation is important for the understanding of underground resources in hydrology and petroleum reservoir engineering. The success of fast prediction relies on a robust and scalable black oil simulator, which can handle the complex subsurface fluid dynamics with coupling several different physical processes, and meanwhile run efficiently on distributed memory parallel computers. In this work, we introduce a highly parallel simulator to accurately compute the reservoir flow coupling with wells, including some adaptive fully implicit schemes for the temporal discretization, the improved active-set reduced-space algorithm for the nonlinear algebraic system, and several types of field-split methods for preconditioning. This parallel simulator is applied on the domain decomposition framework, which is designed to overcome the computational issues of traditional simulators implemented for personal computers or small workstations. High-resolution reservoir simulation results on the Society of Petroleum Engineers (SPE) comparative solution project or real-field cases are obtained and analyzed, and the parallel performance of the algorithm is studied on a supercomputer with scaling up to 27440 processor cores. The numerical experiments indicate that the proposed black oil simulator is capable of accurately predicting the highly complex physical processes and interactions between wells and the reservoir for full field scale simulation.

TTK is Getting MPI-Ready.

This system paper documents the technical foundations for the extension of the Topology ToolKit (TTK) to distributed-memory parallelism with the Message Passing Interface (MPI). While several recent papers introduced topology-based approaches for distributed-memory environments, these were reporting experiments obtained with tailored, mono-algorithm implementations. In contrast, we describe in this paper a versatile approach (supporting both triangulated domains and regular grids) for the support of topological analysis pipelines, i.e., a sequence of topological algorithms interacting together, possibly on distinct numbers of processes. While developing this extension, we faced several algorithmic and software engineering challenges, which we document in this paper. Specifically, we describe an MPI extension of TTK's data structure for triangulation representation and traversal, a central component to the global performance and generality of TTK's topological implementations. We also introduce an intermediate interface between TTK and MPI, both at the global pipeline level, and at the fine-grain algorithmic level. We provide a taxonomy for the distributed-memory topological algorithms supported by TTK, depending on their communication needs and provide examples of hybrid MPI+thread parallelizations. Detailed performance analyses show that parallel efficiencies range from 20% to 80% (depending on the algorithms), and that the MPI-specific preconditioning introduced by our framework induces a negligible computation time overhead. We illustrate the new distributed-memory capabilities of TTK with an example of advanced analysis pipeline, combining multiple algorithms, run on the largest publicly available dataset we have found (120 billion vertices) on a standard cluster with 64 nodes (for a total of 1536 cores). Finally, we provide a roadmap for the completion of TTK's MPI extension, along with generic recommendations for each algorithm communication category.

An implementation of a plasma physics application for distributed-memory supercomputers using a directive-based programming framework

To extract performance from supercomputers, programmers in the High Performance Computing (HPC) community are often required to use a combination of frameworks to take advantage of the multiple levels of parallelism. However, over the years, efforts have been made to simplify this situation by creating frameworks that can take advantage of multiple levels. This often means that the programmer has to learn a new library. On the other hand, there are frameworks that were created by extending the capabilities of established paradigms. In this paper, we explore one of this libraries, OpenMP Cluster. As its name implies, it extends the OpenMP API, which allows seasoned programmers to take advantage of their experience to use just one API to program in sharedmemory and distributed-memory parallelism. In this paper, we took an existing plasma physics code that was programmed with MPI+OpenMP and ported it over to OpenMP Cluster. We also show that under certain conditions, the performance of OpenMP Cluster is similar to that of the MPI+OpenMP code.

PPB-MCTS: A novel distributed-memory parallel partial-backpropagation Monte Carlo tree search algorithm

Monte-Carlo Tree Search (MCTS) is an adaptive and heuristic tree-search algorithm designed to uncover sub-optimal actions at each decision-making point. This method progressively constructs a search tree by gathering samples throughout its execution. Predominantly applied within the realm of gaming, MCTS has exhibited exceptional achievements. Additionally, it has displayed promising outcomes when employed to solve NP-hard combinatorial optimization problems. MCTS has been adapted for distributed-memory parallel platforms. The primary challenges associated with distributed-memory parallel MCTS are the substantial communication overhead and the necessity to balance the computational load among various processes. In this work, we introduce a novel distributed-memory parallel MCTS algorithm with partial backpropagations, referred to as Parallel Partial-Backpropagation MCTS (PPB-MCTS). Our design approach aims to significantly reduce the communication overhead while maintaining, or even slightly improving, the performance in the context of combinatorial optimization problems. To address the communication overhead challenge, we propose a strategy involving transmitting an additional backpropagation message. This strategy avoids attaching an information table to the communication messages exchanged by the processes, thus reducing the communication overhead. Furthermore, this approach contributes to enhancing the decision-making accuracy during the selection phase. The load balancing issue is also effectively addressed by implementing a shared transposition table among the parallel processes. Furthermore, we introduce two primary methods for managing duplicate states within distributed-memory parallel MCTS, drawing upon techniques utilized in addressing duplicate states within sequential MCTS. Duplicate states can transform the conventional search tree into a Directed Acyclic Graph (DAG). To evaluate the performance of our proposed parallel algorithm, we conduct an extensive series of experiments on solving instances of the Job-Shop Scheduling Problem (JSSP) and the Weighted Set-Cover Problem (WSCP). These problems are recognized for their complexity and classified as NP-hard combinatorial optimization problems with considerable relevance within industrial applications. The experiments are performed on a cluster of computers with many cores. The empirical results highlight the enhanced scalability of our algorithm compared to that of the existing distributed-memory parallel MCTS algorithms. As the number of processes increases, our algorithm demonstrates increased rollout efficiency while maintaining an improved load balance across processes.

Swift : a modern highly parallel gravity and smoothed particle hydrodynamics solver for astrophysical and cosmological applications

ABSTRACT Numerical simulations have become one of the key tools used by theorists in all the fields of astrophysics and cosmology. The development of modern tools that target the largest existing computing systems and exploit state-of-the-art numerical methods and algorithms is thus crucial. In this paper, we introduce the fully open-source highly-parallel, versatile, and modular coupled hydrodynamics, gravity, cosmology, and galaxy-formation code Swift. The software package exploits hybrid shared- and distributed-memory task-based parallelism, asynchronous communications, and domain-decomposition algorithms based on balancing the workload, rather than the data, to efficiently exploit modern high-performance computing cluster architectures. Gravity is solved for using a fast-multipole-method, optionally coupled to a particle mesh solver in Fourier space to handle periodic volumes. For gas evolution, multiple modern flavours of Smoothed Particle Hydrodynamics are implemented. Swift also evolves neutrinos using a state-of-the-art particle-based method. Two complementary networks of sub-grid models for galaxy formation as well as extensions to simulate planetary physics are also released as part of the code. An extensive set of output options, including snapshots, light-cones, power spectra, and a coupling to structure finders are also included. We describe the overall code architecture, summarize the consistency and accuracy tests that were performed, and demonstrate the excellent weak-scaling performance of the code using a representative cosmological hydrodynamical problem with ≈300 billion particles. The code is released to the community alongside extensive documentation for both users and developers, a large selection of example test problems, and a suite of tools to aid in the analysis of large simulations run with Swift.

Scalable implicit solvers with dynamic mesh adaptation for a relativistic drift-kinetic Fokker–Planck–Boltzmann model

In this work we consider a relativistic drift-kinetic model for runaway electrons along with a Fokker–Planck operator for small-angle Coulomb collisions, a radiation damping operator, and a secondary knock-on (Boltzmann) collision source. We develop a new scalable fully implicit solver utilizing finite volume and conservative finite difference schemes and dynamic mesh adaptivity. A new data management framework in the PETSc library based on the p4est library is developed to enable simulations with dynamic adaptive mesh refinement (AMR), distributed memory parallelization, and dynamic load balancing of computational work. This framework and the runaway electron solver building on the framework are able to dynamically capture both bulk Maxwellian at the low-energy region and a runaway tail at the high-energy region. To effectively capture features via the AMR algorithm, a new AMR indicator prediction strategy is proposed that is performed alongside the implicit time evolution of the solution. This strategy is complemented by the introduction of computationally cheap feature-based AMR indicators that are analyzed theoretically. Numerical results quantify the advantages of the prediction strategy in better capturing features compared with nonpredictive strategies; and we demonstrate trade-offs regarding computational costs. The robustness with respect to model parameters, algorithmic scalability, and parallel scalability are demonstrated through several benchmark problems including manufactured solutions and solutions of different physics models. We focus on demonstrating the advantages of using implicit time stepping and AMR for runaway electron simulations.

Flux Flattening Large Heavy Water Power Reactors with Accelerator-Driven Photoneutron Source

Flux flattening and power uprating of large heavy water power reactors (HWRs) are demonstrated as an application of an accelerator-driven photoneutron source (ADS) in the ADS-CANDU concept where an array of electron linear accelerators is configured around the periphery of a subcritical CANDU-6 core. The localized ADS generated through (e−,γ,n) reactions in the HWR lattice perturbs the reactor power distribution by increasing the power of low-power bundles and depressing the power at the core center relative to the fundamental mode power distribution. Gross power uprating is feasible when the system is near critical, but the ADS array consumes tens of megawatts electric exceeding the power gained by a factor of more than 2 for the conservative ADS performance specifications assumed in the analysis. Several important challenges of fixed-source Monte Carlo simulations of near-critical multiplying media are investigated including severe load imbalance issues with distributed-memory parallel computing architecture and correlated local tallies in nonanalog (implicit absorption) Monte Carlo radiation transport. All subcritical fixed-source simulations in the study readily exceed the default random number stride used in most production Monte Carlo codes, and the stride exceedance causes both bias in local tally results (bundle powers) and spatial autocorrelation of these errors/biases in the large core. A legacy stride exceedance is critically reviewed, and the conclusions and subsequent interpretations of those conclusions are rejected. Several classes of radiation transport Monte Carlo problems are likely to be susceptible to stride exceedance, and this issue needs to be promptly addressed by the Monte Carlo analyst and code developer communities.

GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model

Abstract. We discuss the various performance aspects of parallelizing our transient global-scale groundwater model at 30′′ resolution (30 arcsec; ∼ 1 km at the Equator) on large distributed memory parallel clusters. This model, referred to as GLOBGM, is the successor of our 5′ (5 arcmin; ∼ 10 km at the Equator) PCR-GLOBWB 2 (PCRaster Global Water Balance model) groundwater model, based on MODFLOW having two model layers. The current version of GLOBGM (v1.0) used in this study also has two model layers, is uncalibrated, and uses available 30′′ PCR-GLOBWB data. Increasing the model resolution from 5′ to 30′′ creates challenges, including increased runtime, memory usage, and data storage that exceed the capacity of a single computer. We show that our parallelization tackles these problems with relatively low parallel hardware requirements to meet the needs of users or modelers who do not have exclusive access to hundreds or thousands of nodes within a supercomputer. For our simulation, we use unstructured grids and a prototype version of MODFLOW 6 that we have parallelized using the message-passing interface. We construct independent unstructured grids with a total of 278 million active cells to cancel all redundant sea and land cells, while satisfying all necessary boundary conditions, and distribute them over three continental-scale groundwater models (168 million – Afro–Eurasia; 77 million – the Americas; 16 million – Australia) and one remaining model for the smaller islands (17 million). Each of the four groundwater models is partitioned into multiple non-overlapping submodels that are tightly coupled within the MODFLOW linear solver, where each submodel is uniquely assigned to one processor core, and associated submodel data are written in parallel during the pre-processing, using data tiles. For balancing the parallel workload in advance, we apply the widely used METIS graph partitioner in two ways: it is straightforwardly applied to all (lateral) model grid cells, and it is applied in an area-based manner to HydroBASINS catchments that are assigned to submodels for pre-sorting to a future coupling with surface water. We consider an experiment for simulating the years 1958–2015 with daily time steps and monthly input, including a 20-year spin-up, on the Dutch national supercomputer Snellius. Given that the serial simulation would require ∼ 4.5 months of runtime, we set a hypothetical target of a maximum of 16 h of simulation runtime. We show that 12 nodes (32 cores per node; 384 cores in total) are sufficient to achieve this target, resulting in a speedup of 138 for the largest Afro–Eurasia model when using 7 nodes (224 cores) in parallel. A limited evaluation of the model output using the United States Geological Survey (USGS) National Water Information System (NWIS) head observations for the contiguous United States was conducted. This showed that increasing the resolution from 5′ to 30′′ results in a significant improvement with GLOBGM for the steady-state simulation when compared to the 5′ PCR-GLOBWB groundwater model. However, results for the transient simulation are quite similar, and there is much room for improvement. Monthly and multi-year total terrestrial water storage anomalies derived from the GLOBGM and PCR-GLOBWB models, however, compared favorably with observations from the GRACE satellite. For the next versions of GLOBGM, further improvements require a more detailed (hydro)geological schematization and better information on the locations, depths, and pumping rates of abstraction wells.

Coupling finite and boundary element methods to solve the Poisson-Boltzmann equation for electrostatics in molecular solvation.

The Poisson-Boltzmann equation is widely used to model electrostatics in molecular systems. Available software packages solve it using finite difference, finite element, and boundary element methods, where the latter is attractive due to the accurate representation of the molecular surface and partial charges, and exact enforcement of the boundary conditions at infinity. However, the boundary element method is limited to linear equations and piecewise constant variations of the material properties. In this work, we present a scheme that couples finite and boundary elements for the linearised Poisson-Boltzmann equation, where the finite element method is applied in a confined solute region and the boundary element method in the external solvent region. As a proof-of-concept exercise, we use the simplest methods available: Johnson-Nédélec coupling with mass matrix and diagonal preconditioning, implemented using the Bempp-cl and FEniCSx libraries via their Python interfaces. We showcase our implementation by computing the polar component of the solvation free energy of a set of molecules using a constant and a Gaussian-varying permittivity. As validation, we compare against well-established finite difference solvers for an extensive binding energy data set, and with the finite difference code APBS (to 0.5%) for Gaussian permittivities. We also show scaling results from protein G B1 (955 atoms) up to immunoglobulin G (20,148 atoms). For small problems, the coupled method was efficient, outperforming a purely boundary integral approach. For Gaussian-varying permittivities, which are beyond the applicability of boundary elements alone, we were able to run medium to large-sized problems on a single workstation. The development of better preconditioning techniques and the use of distributed memory parallelism for larger systems remains an area for future work. We hope this work will serve as inspiration for future developments that consider space-varying field parameters, and mixed linear-nonlinear schemes for molecular electrostatics with implicit solvent models.

Lifex-cfd: An open-source computational fluid dynamics solver for cardiovascular applications

Computational fluid dynamics (CFD) is an important tool for the simulation of the cardiovascular function and dysfunction. Due to the complexity of the anatomy, the transitional regime of blood flow in the heart, and the strong mutual influence between the flow and the physical processes involved in the heart function, the development of accurate and efficient CFD solvers for cardiovascular flows is still a challenging task. In this paper we present life▪-cfd, an open-source CFD solver for cardiovascular simulations based on the life▪ finite element library, written in modern C++ and exploiting distributed memory parallelism. We model blood flow in both physiological and pathological conditions via the incompressible Navier-Stokes equations, accounting for moving cardiac valves, moving domains, and transition-to-turbulence regimes. In this paper, we provide an overview of the underlying mathematical formulation, numerical discretization, implementation details and examples on how to use life▪-cfd. We verify the code through rigorous convergence analyses, and we show its almost ideal parallel speedup. We demonstrate the accuracy and reliability of the numerical methods implemented through a series of idealized and patient-specific vascular and cardiac simulations, in different physiological flow regimes. The life▪-cfd source code is available under the LGPLv3 license, to ensure its accessibility and transparency to the scientific community, and to facilitate collaboration and further developments. Program summaryProgram Title:life▪-cfdCPC Library link to program files:https://doi.org/10.17632/hzsnc3jgds.1Developer's repository link:https://gitlab.com/lifex/lifex-cfdLicensing provisions:LGPLv3Programming language: C++ (standard ≥17)Supplementary material:https://doi.org/10.5281/zenodo.7852088 contains the application executable in binary form, compatible with any recent enough x86-64 Linux system, assuming that glibc version ≥ 2.28 is installed. Data and parameter files necessary to replicate the test cases described in this manuscript are also available.Nature of problem: The program allows to run computational fluid dynamics simulations of cardiovascular blood flows in physiological and pathological conditions, modeled through incompressible Navier-Stokes equations, including moving cardiac valves, moving domains (such as contracting cardiac chambers) in the arbitrary Lagrangian-Eulerian framework, and transition-to-turbulence flow. Given the scale of the typical applications, the program is designed for parallel execution.Solution method: The equations are discretized using the Finite Element method, on either tetrahedral or hexahedral meshes. The software builds on top of deal.II, implementing the mathematical models and numerical methods specific for CFD cardiovascular simulations. Parallel execution exploits the MPI paradigm. The software supports both Trilinos and PETSc as linear algebra backends.Additional comments including restrictions and unusual features: The program provides a general-purpose executable that can be used to run CFD simulations without having to access or modify the source code. The program allows to setup simulations through a user-friendly yet flexible interface, by means of readable and self-documenting parameter files. On top of that, more advanced users can modify the source code to implement more sophisticated test cases. life▪-cfd supports checkpointing, i.e. simulations can be stopped and restarted at a later time.

Open-source complex-geometry 3D fluid dynamics for applications with unpredictable heterogeneous dynamic high-performance-computing loads

We discuss the software implementation of a finite element method, intended for the simulation of complex-geometry 3D flows, using hardware resources effectively, including problems with a priori unknown, heterogeneous, and dynamic parallel computational load. Our fundamental choices of data structures, algorithm, and software design specifically target engineering resolution and accuracy requirements. Some of these choices are: unstructured grids (to explicitly resolve complex 3D geometries), tetrahedra-only computational elements (to enable automatic mesh generation), edge-based finite element scheme (to reduce indirect addressing for increased performance), distributed-memory parallel computing paradigm (to enable large problems), and Charm++ (https://charmplusplus.org) as the runtime system (to effectively use computing resources even in the presence of hardware heterogeneities and dynamic application requirements). We discuss aspects of the implementation that enable exercising unique features of the runtime system, e.g., the single Charm++ programming abstraction, overdecomposition, asynchronous execution, latency-hiding parallel communication and computation, task-parallelism, and dynamic load balancing via object migratability. Multiple test problems are used to verify and validate the numerical solutions and computational performance and scalability to high-performance computing environments are discussed. For maximum transparency and reproducibility, and to encourage future research, development, and use, the full source code, together with regression tests and documentation, is publicly available at https://xyst.cc.

A numerical solver for active hydrodynamics in three dimensions and its application to active turbulence

We present a higher-order convergent numerical solver for active polar hydrodynamics in three-dimensional domains of arbitrary shape, along with a scalable open-source software implementation for shared- and distributed-memory parallel computers. This enables the computational study of the nonlinear dynamics of out-of-equilibrium materials from first principles. We numerically solve the nonlinear active Ericksen–Leslie hydrodynamic equations of three-dimensional (3D) active nematics using both a meshfree and a hybrid particle-mesh method in either the Eulerian or Lagrangian frame of reference. The solver is validated against a newly derived analytical solution in 3D and implemented using the OpenFPM software library for scalable scientific computing. We then apply the presented method to studying the transition of 3D active polar fluids to spatiotemporal chaos, the emergence of coherent angular motion in a 3D annulus, and chiral vortices in symmetric and asymmetric 3D shapes resembling dividing cells. Overall, this provides a robust and efficient open-source simulation framework for 3D active matter with verified numerical convergence and scalability on parallel computers.

Task-based Parallel Programming for Scalable Matrix Product Algorithms

Task-based programming models have succeeded in gaining the interest of the high-performance mathematical software community because they relieve part of the burden of developing and implementing distributed-memory parallel algorithms in an efficient and portable way.In increasingly larger, more heterogeneous clusters of computers, these models appear as a way to maintain and enhance more complex algorithms. However, task-based programming models lack the flexibility and the features that are necessary to express in an elegant and compact way scalable algorithms that rely on advanced communication patterns. We show that the Sequential Task Flow paradigm can be extended to write compact yet efficient and scalable routines for linear algebra computations. Although, this work focuses on dense General Matrix Multiplication, the proposed features enable the implementation of more complex algorithms. We describe the implementation of these features and of the resulting GEMM operation. Finally, we present an experimental analysis on two homogeneous supercomputers showing that our approach is competitive up to 32,768 CPU cores with state-of-the-art libraries and may outperform them for some problem dimensions. Although our code can use GPUs straightforwardly, we do not deal with this case because it implies other issues which are out of the scope of this work.

Newly Released Capabilities in the Distributed-Memory SuperLU Sparse Direct Solver

We present the new features available in the recent release of SuperLU_DIST , Version 8.1.1. SuperLU_DIST is a distributed-memory parallel sparse direct solver. The new features include (1) a 3D communication-avoiding algorithm framework that trades off inter-process communication for selective memory duplication, (2) multi-GPU support for both NVIDIA GPUs and AMD GPUs, and (3) mixed-precision routines that perform single-precision LU factorization and double-precision iterative refinement. Apart from the algorithm improvements, we also modernized the software build system to use CMake and Spack package installation tools to simplify the installation procedure. Throughout the article, we describe in detail the pertinent performance-sensitive parameters associated with each new algorithmic feature, show how they are exposed to the users, and give general guidance of how to set these parameters. We illustrate that the solver’s performance both in time and memory can be greatly improved after systematic tuning of the parameters, depending on the input sparse matrix and underlying hardware.

Introducing MPEC: Massively parallel electron correlation.

We have developed a new program for carrying out improved internally contracted Multi-reference Configuration Interaction Singles and Doubles (i2cMRCISD) calculations. It is designed from the ground up to be used on distributed memory parallel computers. Tests show good scaling properties with the number of cores per node and the number nodes. This program features Gaussian basis sets with ℓ > 6; scalar special relativity via the spin-free method; convergence to C∞v, D∞v, or spherical electronic states; special code to determine Rydberg orbitals; both uncontracted and contracted MRCISD wavefunctions; one and two electron properties, including full spin-orbit matrix elements with the Breit interaction; analytic calculation of Born-Oppenheimer diagonal correction for multi-configuration Hartree-Fock wavefunctions; and analytic calculation of second order Born-Oppenheimer corrections for Hartree-Fock wavefunctions. The program can be obtained from software.nasa.gov.

Exploiting underlying crystal lattice for efficient computation of Coulomb matrix elements in multi-million atoms nanostructures

Atomistic modeling of nanostructures often leads to computationally challenging problems involving millions of atoms and tens of thousands of Coulomb matrix elements. In our previous work, we presented a practical solution to this problem, where quasi-linear efficiency, both in time and memory, was obtained by utilizing the fast Fourier transform. Here, we present an updated version of our highly-parallelized computer program, named Coulombo-Lattice, that eliminates the necessity of introducing an auxiliary basis set for the wave-function transfer to the computational grid. Here, we instead exploit the properties of the underlying crystal lattice and run calculations on a regular three-dimensional grid superimposed on the original, lower-symmetry lattice. Due to removal of spurious interactions from other supercells, the resulting Coulomb matrix elements are, up to numerical precision, identical to those obtained by the direct summation O(N2) method, yet our code maintains O(Nlog⁡N) scaling. We illustrate our approach by calculations involving up to 1.7 million integrals, and number of atoms reaching up to 2.8 million, for the problem of dopant charging energy for a single phosphorus dopant embedded in a silicon lattice. Next, to emphasize the broad applicability of our code, we show the results for mixed zinc-blend/wurtzite lattice systems, also known as crystal phase quantum dots. Program summaryProgram Title: Coulombo-LatticeCPC Library link to program files:https://doi.org/10.17632/jwkvh5ycbf.1Licensing provisions: CC BY 4.0Programming language: C++Nature of problem: Computing the Coulomb matrix elements (Coulomb and exchange integrals), while being a demanding computational task, is a necessary step in a range of quantum mechanical calculations. For example, in the field of nanostructures, such as quantum dots and nanowires or novel single dopant devices, this stage of calculation (even after a series of approximations) is at least an O(N2) operation (a summation over all pairs of N atoms or grid points in the analyzed system). Moreover, calculating the full Coulomb matrix usually requires the computation of thousands of such elements, thus presenting a formidable computational challenge.Solution method: We provide a ready-to-use implementation for calculating Coulomb matrix elements for a given set of input wavefunctions in LCAO (linear combination of atomic orbitals) form resulting from tight-binding calculation. This implementation is based on the approach introduced in [1,2], by using a fast Fourier transform. In this work, we further significantly improved the method by eliminating the need to introduce wave-function transfer via auxiliary basis set.Additional comments including Restrictions and Unusual features: The implementation is fully parallelized in a distributed-memory model, using MPI and parallel routines from FFTW [3].