Parallel Supercomputers Research Articles

MILC Code Performance on High End CPU and GPU Supercomputer Clusters

With recent developments in parallel supercomputing architecture, many core, multi-core, and GPU processors are now commonplace, resulting in more levels of parallelism, memory hierarchy, and programming complexity. It has been necessary to adapt the MILC code to these new processors starting with NVIDIA GPUs, and more recently, the Intel Xeon Phi processors. We report on our efforts to port and optimize our code for the Intel Knights Landing architecture. We consider performance of the MILC code with MPI and OpenMP, and optimizations with QOPQDP and QPhiX. For the latter approach, we concentrate on the staggered conjugate gradient and gauge force. We also consider performance on recent NVIDIA GPUs using the QUDA library.

A scalable parallel framework for microstructure analysis of large-scale molecular dynamics simulations data

Molecular dynamics (MD) simulation is a fundamental tool in computational materials science. With the development of parallel supercomputers, researchers can access the detail atomic responses of materials with MD simulations at unprecedented scales of physical and time. However, the size of generated output datasets is also growing rapidly, which poses a serious challenge for traditional data analysis methods. Therefore, parallel analysis methods to support faster and more scalable manipulation of atomic data are desperately needed. In this paper, we present a scalable parallel framework to meet the requirements. It allows users to implement a parallel analysis program using a simple interface, make use of the existing sequential analysis codes, and carry out distributed-memory post-simulation data analysis. We have integrated three popular microstructure characterization methods (lattice structure identification, Voronoi analysis, and Wigner-Seitz defect analysis) based on this framework. Performance evaluations run on massively parallel process computers with 109 atoms on up to 1024 processor cores demonstrate the scalability and efficiency of the proposed framework. The proposed framework is helping accelerate large-scale MD data analysis to a new level.

Smilei : A collaborative, open-source, multi-purpose particle-in-cell code for plasma simulation

Smilei is a collaborative, open-source, object-oriented (C++) particle-in-cell code. To benefit from the latest advances in high-performance computing (HPC), Smilei is co-developed by both physicists and HPC experts. The code’s structures, capabilities, parallelization strategy and performances are discussed. Additional modules (e.g. to treat ionization or collisions), benchmarks and physics highlights are also presented. Multi-purpose and evolutive, Smilei is applied today to a wide range of physics studies, from relativistic laser–plasma interaction to astrophysical plasmas. Program summaryProgram title:Smilei (version 3.2)Program Files doi:http://dx.doi.org/10.17632/gsn4x6mbrg.1Licensing provisions: This version of the code is distributed under the GNU General Public License v3Programming language: C++11, Python 2.7Nature of the problem: The kinetic simulation of plasmas is at the center of various physics studies, from laser–plasma interaction to astrophysics. To address today’s challenges, a versatile simulation tool requires high-performance computing on massively parallel super-computers.Solution method: The Vlasov–Maxwell system describing the self-consistent evolution of a collisionless plasma is solved using the Particle-In-Cell (PIC) method. Additional physics modules allow to account for additional effects such as collisions and/or ionization. A hybrid MPI-OpenMP strategy, based on a patch-based super-decomposition, allows for efficient cache-use, dynamic load balancing and high-performance on massively parallel super-computers.Additional comments: Repository https://github.com/SmileiPIC/Smilei

Challenges in scaling NLO generators to leadership computers

Exascale computing resources are roughly a decade away and will be capable of 100 times more computing than current supercomputers. In the last year, Energy Frontier experiments crossed a milestone of 100 million core-hours used at the Argonne Leadership Computing Facility, Oak Ridge Leadership Computing Facility, and NERSC. The Fortran-based leading-order parton generator called Alpgen was successfully scaled to millions of threads to achieve this level of usage on Mira. Sherpa and MadGraph are next-to-leading order generators used heavily by LHC experiments for simulation. Integration times for high-multiplicity or rare processes can take a week or more on standard Grid machines, even using all 16-cores. We will describe our ongoing work to scale the Sherpa generator to thousands of threads on leadership-class machines and reduce run-times to less than a day. This work allows the experiments to leverage large-scale parallel supercomputers for event generation today, freeing tens of millions of grid hours for other work, and paving the way for future applications (simulation, reconstruction) on these and future supercomputers.

Discovery of Pb-Free Perovskite Solar Cells via High-Throughput Simulation on the K Computer.

We performed a systematic high-throughput simulation with density functional theory for 11 025 compositions of hybrid organic-inorganic halide compounds in ABX3 and A2BB'X6 forms, where A is an organic or inorganic component, B/B' is a metal atom, and X is a halogen atom. The computational results were compiled as a materials database. We performed massive computational simulation by using the K computer, which is a massively parallel many-core supercomputer in Japan. By applying the screening procedure to all the compounds in the materials database, we discovered novel candidates for environmentally friendly lead-free perovskite solar cells and propose 51 low-toxic halide single and double perovskites, most of which are newly proposed in this study. The proposed low-toxic halide double perovskites are classified under six families: group-14-group-14, group-13-group-15, group-11-group-11, group-9-group-13, group-11-group-13, and group-11-group-15 double perovskites.

Accelerating nuclear configuration interaction calculations through a preconditioned block iterative eigensolver

We describe a number of recently developed techniques for improving the performance of large-scale nuclear configuration interaction calculations on high performance parallel computers. We show the benefit of using a preconditioned block iterative method to replace the Lanczos algorithm that has traditionally been used to perform this type of computation. The rapid convergence of the block iterative method is achieved by a proper choice of starting guesses of the eigenvectors and the construction of an effective preconditioner. These acceleration techniques take advantage of special structure of the nuclear configuration interaction problem which we discuss in detail. The use of a block method also allows us to improve the concurrency of the computation, and take advantage of the memory hierarchy of modern microprocessors to increase the arithmetic intensity of the computation relative to data movement. We also discuss the implementation details that are critical to achieving high performance on massively parallel multi-core supercomputers, and demonstrate that the new block iterative solver is two to three times faster than the Lanczos based algorithm for problems of moderate sizes on a Cray XC30 system.

Development and Integration of an In-Situ Framework for Flow Visualization of Large-Scale, Unsteady Phenomena

With large-scale simulation models on massively parallel supercomputers generating increasingly large data sets, in-situ visualization is a promising way to avoid bottlenecks. Enabling in-situ visualization in a simulation model asks for special attention to the interface between a parallel simulation model and the data analysis part of the visualization, and to presentation and interaction scenarios. Modifications to scientific workflows would potentially result in a paradigm shift, which affects compute and data intensive applications generally. We present our approach for enabling in-situ visualization within the highly parallelized climate model ICON using the DSVR visualization framework. We focus on the requirements for generalized grid and data structures, and for universal, scalable algorithms for volume and flow visualization of time series. In-situ pathline extraction as a technique for the visualization of unsteady flows has been integrated in the climate simulation model ICON and verified in first studies.

Entropy-limited hydrodynamics: a novel approach to relativistic hydrodynamics

We present entropy-limited hydrodynamics (ELH): a new approach for the computation of numerical fluxes arising in the discretization of hyperbolic equations in conservation form. ELH is based on the hybridisation of an unfiltered high-order scheme with the first-order Lax-Friedrichs method. The activation of the low-order part of the scheme is driven by a measure of the locally generated entropy inspired by the artificial-viscosity method proposed by Guermond et al. (J. Comput. Phys. 230(11):4248-4267, 2011, doi:10.1016/j.jcp.2010.11.043). Here, we present ELH in the context of high-order finite-differencing methods and of the equations of general-relativistic hydrodynamics. We study the performance of ELH in a series of classical astrophysical tests in general relativity involving isolated, rotating and nonrotating neutron stars, and including a case of gravitational collapse to black hole. We present a detailed comparison of ELH with the fifth-order monotonicity preserving method MP5 (Suresh and Huynh in J. Comput. Phys. 136(1):83-99, 1997, doi:10.1006/jcph.1997.5745), one of the most common high-order schemes currently employed in numerical-relativity simulations. We find that ELH achieves comparable and, in many of the cases studied here, better accuracy than more traditional methods at a fraction of the computational cost (up to {sim}50% speedup). Given its accuracy and its simplicity of implementation, ELH is a promising framework for the development of new special- and general-relativistic hydrodynamics codes well adapted for massively parallel supercomputers.

Near-surface coherent structures explored by large eddy simulation of entire tropical cyclones

Taking advantage of the huge computational power of a massive parallel supercomputer (K-supercomputer), this study conducts large eddy simulations of entire tropical cyclones by employing a numerical weather prediction model, and explores near-surface coherent structures. The maximum of the near-surface wind changes little from that simulated based on coarse-resolution runs. Three kinds of coherent structures appeared inside the boundary layer. The first is a Type-A roll, which is caused by an inflection-point instability of the radial flow and prevails outside the radius of maximum wind. The second is a Type-B roll that also appears to be caused by an inflection-point instability but of both radial and tangential winds. Its roll axis is almost orthogonal to the Type-A roll. The third is a Type-C roll, which occurs inside the radius of maximum wind and only near the surface. It transports horizontal momentum in an up-gradient sense and causes the largest gusts.

Multidimensional Chebyshev interpolation for warm and hot dense matter.

We propose a scheme based on a multidimensional Chebyshev interpolation to approximate smooth functions that depend on more than one variable. The present method generalizes the one dimensional Chebyshev approximation. The multidimensional approach can be used for generating databases like equation of state in the warm and hot dense matter. It is well suited to the present advance of massively parallel supercomputers.

Large-scale atomistic calculations of clusters in intense x-ray pulses

We present the methodology of our recently developed Monte Carlo/molecular-dynamics method for studying the fundamental ultrafast dynamics induced by high-fluence, high-intensity x-ray free electron laser (XFEL) pulses in clusters. The quantum nature of the initiating ionization process is accounted for by a Monte Carlo method to calculate probabilities of electronic transitions, including photo absorption, inner-shell relaxation, photon scattering, electron collision and recombination dynamics, and thus track the transient electronic configurations (EC) explicitly. The freed electrons, atoms and ions are followed by classical particle trajectories using a molecular dynamics algorithm. Our calculations reveal the role of electron–ion recombination processes that lead to the development of nonuniform spatial charge density profiles in x-ray excited clusters over femtosecond timescales at XFEL intensities exceeding 1020 W cm−2. The Ar cluster fluorescence spectrum is found to be very different from the Ar atom spectrum, in which recombination processes enable additional pathways to reach the required EC for fluorescence transitions. In the high-intensity limit, recombination dynamics can play an important role in the calculated scattering response even for a 2 fs pulse. We demonstrate that our numerical codes and algorithms can make efficient use of the computational power of massively parallel supercomputers to investigate the intense-field dynamics in systems with increasing complexity and size at the ultrafast timescale and in nonlinear x-ray interaction regimes. In particular, picosecond trajectories of XFEL clusters with attosecond time resolution containing millions of particles can be efficiently computed on upwards of 262 144 processes.

Distributed Shortcut Networks: Low-Latency Low-Degree Non-Random Topologies Targeting the Diameter and Cable Length Trade-Off

Low communication latency becomes a main concern in highly parallel computers and supercomputers that reach millions of processing cores. Random network topologies are better suited to achieve low average shortest path length and low diameter in terms of the hop counts between nodes. However, random topologies lead to two problems: (1) increased aggregate cable length on a machine room floor that would become dominant for communication latency in next-generation custom supercomputers, and (2) high routing complexity that typically requires a routing table at each node (e.g., topology-agnostic deadlock-free routing). In this context, we first propose low-degree non-random topologies that exploit the small-world effect, which has been well modeled by some random network models. Our main idea is to carefully design a set of various-length shortcuts that keep the diameter small while maintaining a short cable length for economical passive electric cables. We also propose custom routing that uses the regularity of the various-length shortcuts. Our experimental graph analyses show that our proposed topology has low diameter and low average shortest path length, which are considerably better than those of the counterpart 3-D torus and are near to those of a random topology with the same average degree. The proposed topology has average cable length drastically shorter than that of the counterpart random topology, which leads to low cost of interconnection networks. Our custom routing takes non-minimal paths to provide lower zero-load latency than the minimal custom routings on different counterpart topologies. Our discrete-event simulation results using SimGrid show that our proposed topology is suitable for applications that have irregular communication patterns or non-nearest neighbor collective communication patterns.

A parallel algorithm of multi-variant evolutionary synthesis of nonlinear models

A parallel algorithm has been proposed for solving the problem of construction of nonlinear models (mathematical expressions, functions, algorithms, and programs) using given experimental data, set of variables, basic functions and operations. The designed algorithm of multivariant evolutionary synthesis of nonlinear models includes linear representation of a chromosome, modular operations in decoding of a genotype into a phenotype for interpreting a chromosome as a sequence of instructions, and a multi-variant method for presenting a set of models (expressions) using a single chromosome. A sequential version of the algorithm is compared with a standard genetic programming (GP) algorithm and a Cartesian genetic programming (CGP) one. The algorithm proposed was shown to excel the GP and CGP algorithms both in the time required for search for a solution (more than by an order of magnitude in most cases) and in the probability of finding a given function (model). Experiments have been carried out on parallel supercomputer systems, and estimates of the efficiency of the parallel algorithm offered have been obtained; the estimates demonstrate linear acceleration and scalability.

Asynchronous optimized Schwarz methods with and without overlap

An asynchronous version of the optimized Schwarz method for the solution of differential equations on a parallel computational environment is studied. In a one-way subdivision of the computational domain, with and without overlap, the method is shown to converge when the optimal artificial interface conditions are used. Convergence is also proved for the Laplacian operator under very mild conditions on the size of the subdomains, when approximate (non-optimal) interface conditions are utilized. Numerical results are presented on a large three-dimensional problem on modern parallel clusters and supercomputers illustrating the efficiency of the asynchronous approach.

Gromex: Electrostatics with Chemical Variability for Realistic Molecular Simulations on the Exascale

Molecular electrostatics, notably in proteins and other macromolecules, is complicated by the presence of titratable sites which occur in different forms whose charge distribution differs, e.g., due to protonation or reduction in response to changes in their environment. This variability is often crucial for molecular function and interaction properties and thus has to be included for a realistic description of electrostatics. The computation of electrostatic interactions is also the computationally most demanding part of a molecular dynamics (MD) simulation. To address these issues, we combine a fast multipole method (FMM) with λ-dynamics for the open source molecular dynamics package GROMACS.λ-dynamics bridges discrete, physical site forms via continuous λ-variables to allow the variable sites to interconvert between their forms, thus adding the variability of the charge distribution as physical detail missing in standard MD. The FMM ideally complements by enabling efficient, non-redundant computation of the electrostatic forces needed also in standard MD and of the additional electrostatic interaction energies required to compute the forces that drive the interconversion between the different site forms. In addition, the FMM allows us to take full advantage of highly parallel supercomputers beyond 10,000s of CPU cores, as predicted for the upcoming exascale machines, by avoiding time consuming communication between compute nodes.Our λ-dynamics method extends existing constant-pH λ-dynamics beyond protonation by allowing for titratable sites with any number of forms of potentially entirely different chemical structure, including binding of different ligand types. Allowing for many different site forms, and thus λ-variables, increases the size of the configuration space that needs be sampled. While interconversion between the site forms via continuous λ-variables increases sampling efficiency, it also leads to the occurrence of unphysical intermediate states. A new bias potential that explicitly controls the occurrence of unphysical intermediate states avoids their overrepresentation and ensures sampling of physically relevant configurations.

MPI/OpenMP hybrid parallel algorithm for resolution of identity second-order Møller-Plesset perturbation calculation of analytical energy gradient for massively parallel multicore supercomputers.

A massively parallel algorithm of the analytical energy gradient calculations based the resolution of identity Møller-Plesset perturbation (RI-MP2) method from the restricted Hartree-Fock reference is presented for geometry optimization calculations and one-electron property calculations of large molecules. This algorithm is designed for massively parallel computation on multicore supercomputers applying the Message Passing Interface (MPI) and Open Multi-Processing (OpenMP) hybrid parallel programming model. In this algorithm, the two-dimensional hierarchical MP2 parallelization scheme is applied using a huge number of MPI processes (more than 1000 MPI processes) for acceleration of the computationally demanding O(N5 ) step such as calculations of occupied-occupied and virtual-virtual blocks of MP2 one-particle density matrix and MP2 two-particle density matrices. The new parallel algorithm performance is assessed using test calculations of several large molecules such as buckycatcher C60 @C60 H28 (144 atoms, 1820 atomic orbitals (AOs) for def2-SVP basis set, and 3888 AOs for def2-TZVP), nanographene dimer (C96 H24 )2 (240 atoms, 2928 AOs for def2-SVP, and 6432 AOs for cc-pVTZ), and trp-cage protein 1L2Y (304 atoms and 2906 AOs for def2-SVP) using up to 32,768 nodes and 262,144 central processing unit (CPU) cores of the K computer. The results of geometry optimization calculations of trp-cage protein 1L2Y at the RI-MP2/def2-SVP level using the 3072 nodes and 24,576 cores of the K computer are presented and discussed to assess the efficiency of the proposed algorithm. © 2017 Wiley Periodicals, Inc.

Copernicus, a hybrid dataflow and peer-to-peer scientific computing platform for efficient large-scale ensemble sampling

Compute-intensive applications have gradually changed focus from massively parallel supercomputers to capacity as a resource obtained on-demand. This is particularly true for the large-scale adoption of cloud computing and MapReduce in industry, while it has been difficult for traditional high-performance computing (HPC) usage in scientific and engineering computing to exploit this type of resources. However, with the strong trend of increasing parallelism rather than faster processors, a growing number of applications target parallelism already on the algorithm level with loosely coupled approaches based on sampling and ensembles. While these cannot trivially be formulated as MapReduce, they are highly amenable to throughput computing. There are many general and powerful frameworks, but in particular for sampling-based algorithms in scientific computing there are some clear advantages from having a platform and scheduler that are highly aware of the underlying physical problem. Here, we present how these challenges are addressed with combinations of dataflow programming, peer-to-peer techniques and peer-to-peer networks in the Copernicus platform. This allows automation of sampling-focused workflows, task generation, dependency tracking, and not least distributing these to a diverse set of compute resources ranging from supercomputers to clouds and distributed computing (across firewalls and fragile networks). Workflows are defined from modules using existing programs, which makes them reusable without programming requirements. The system achieves resiliency by handling node failures transparently with minimal loss of computing time due to checkpointing, and a single server can manage hundreds of thousands of cores e.g. for computational chemistry applications.

Multiple program/multiple data molecular dynamics method with multiple time step integrator for large biological systems.

Parallelization of molecular dynamics (MD) simulation is essential for investigating conformational dynamics of large biological systems, such as ribosomes, viruses, and multiple proteins in cellular environments. To improve efficiency in the parallel computation, we have to reduce the amount of data transfer between processors by introducing domain decomposition schemes. Also, it is important to optimize the computational balance between real-space non-bonded interactions and reciprocal-space interactions for long-range electrostatic interactions. Here, we introduce a novel parallelization scheme for large-scale MD simulations on massively parallel supercomputers consisting of only CPUs. We make use of a multiple program/multiple data (MPMD) approach for separating the real-space and reciprocal-space computations on different processors. We also utilize the r-RESPA multiple time step integrator on the framework of the MPMD approach in an efficient way: when the reciprocal-space computations are skipped in r-RESPA, processors assigned for them are utilized for half of the real-space computations. The new scheme allows us to use twice as many as processors that are available in the conventional single program approach. The best performances of all-atom MD simulations for 1 million (STMV), 8.5 million (8_STMV), and 28.8 million (27_STMV) atom systems on K computer are 65, 36, and 24 ns/day, respectively. The MPMD scheme can accelerate 23.4, 10.2, and 9.2 ns/day from the maximum performance of single-program approach for STMV, 8_STMV, and 27_STMV systems, respectively, which correspond to 57%, 39%, and 60% speed up. This suggests significant speedups by increasing the number of processors without losing parallel computational efficiency. © 2016 Wiley Periodicals, Inc.

Graphics Processing Unit Acceleration and Parallelization of GENESIS for Large-Scale Molecular Dynamics Simulations.

The graphics processing unit (GPU) has become a popular computational platform for molecular dynamics (MD) simulations of biomolecules. A significant speedup in the simulations of small- or medium-size systems using only a few computer nodes with a single or multiple GPUs has been reported. Because of GPU memory limitation and slow communication between GPUs on different computer nodes, it is not straightforward to accelerate MD simulations of large biological systems that contain a few million or more atoms on massively parallel supercomputers with GPUs. In this study, we develop a new scheme in our MD software, GENESIS, to reduce the total computational time on such computers. Computationally intensive real-space nonbonded interactions are computed mainly on GPUs in the scheme, while less intensive bonded interactions and communication-intensive reciprocal-space interactions are performed on CPUs. On the basis of the midpoint cell method as a domain decomposition scheme, we invent the single particle interaction list for reducing the GPU memory usage. Since total computational time is limited by the reciprocal-space computation, we utilize the RESPA multiple time-step integration and reduce the CPU resting time by assigning a subset of nonbonded interactions on CPUs as well as on GPUs when the reciprocal-space computation is skipped. We validated our GPU implementations in GENESIS on BPTI and a membrane protein, porin, by MD simulations and an alanine-tripeptide by REMD simulations. Benchmark calculations on TSUBAME supercomputer showed that an MD simulation of a million atoms system was scalable up to 256 computer nodes with GPUs.

A new mode of radio wave diffraction via the terrestrial surface plasmon on mountain range

AbstractIt is shown that a new mode of radio wave diffraction occurs at the peak of mountains mediated via the terrestrial surface plasmon. If mobile electrical charges exist on the Earth's surface, the electromagnetic theory predicts strong coupling between the radio wave and the surface plasmon on the ground. If sufficient amount of electrical charges of the same sign appear on the ground as a consequence of some underground preseismic activity, they will be subject to the electrical repulsive forces. The surface electrical charges will then move toward topographic highs of nearby mountain peaks. Radio waves are then shown to interact with such electrical charges and create collective oscillations of the surface charges to induce a surface plasmon. Here it is clarified with numerical analyses on a massively parallel supercomputer that such interactions occur on the peak of mountains, hence causing peculiar phenomena of random diffraction. Depending on the density of the electrical charges on the ground surface, the interaction becomes strong enough to cause intense and random scattering and diffraction of the radio wave from the rough surface of the mountain topography. Mountain peaks thus act as a secondary source of radio waves; unexpectedly, radio waves are reradiated from the peaks into various directions by the anomalous diffraction and scattering, and the reradiated wave can propagate beyond the line of sight over mountains to reach distant locations. Such effects may arise randomly but concurrently with some preseismic activity in the crustal rocks, of which observation may allow statistical analysis of the critical state of crustal rocks over a broad area of a few hundred kilometers.