Program Multiple Data Research Articles

MiMiC: A high-performance framework for multiscale molecular dynamics simulations.

MiMiC is a framework for performing multiscale simulations in which loosely coupled external programs describe individual subsystems at different resolutions and levels of theory. To make it highly efficient and flexible, we adopt an interoperable approach based on a multiple-program multiple-data (MPMD) paradigm, serving as an intermediary responsible for fast data exchange and interactions between the subsystems. The main goal of MiMiC is to avoid interfering with the underlying parallelization of the external programs, including the operability on hybrid architectures (e.g., CPU/GPU), and keep their setup and execution as close as possible to the original. At the moment, MiMiC offers an efficient implementation of electrostatic embedding quantum mechanics/molecular mechanics (QM/MM) that has demonstrated unprecedented parallel scaling in simulations of large biomolecules using CPMD and GROMACS as QM and MM engines, respectively. However, as it is designed for high flexibility with general multiscale models in mind, it can be straightforwardly extended beyond QM/MM. In this article, we illustrate the software design and the features of the framework, which make it a compelling choice for multiscale simulations in the upcoming era of exascale high-performance computing.

An MPMD approach coupling electromagnetic continuum mechanics approximations in ALEGRA

Two complementary approximations for describing aspects of continuum electromagnetics in moving media are discussed: electroquasistatic and magnetoquasistatic. Each has been implemented in the finite element shock code ALEGRA for modeling dynamic electromechanical phenomena on typical engineering time scales, with fully integrated circuit coupling (Niederhaus et al. 2023). The approximations can be obtained by consistent asymptotic balancing of Maxwell’s equations relative to timescales associated with magnetic diffusion, charge relaxation, and electromagnetic wave propagation. In ALEGRA, the electroquasistatic approximation is used for ferroelectric (FE) modeling, while the magnetoquasistatic approximation is used for magnetohydrodynamic (MHD) modeling. In this paper we introduce for the first time a detailed derivation of a useful quasi-steady “low-Rm” variant of the MHD approximation applicable for cases, such as with detonators, where the thermodynamic pressure arising from Joule heating dominates over magnetic forces. An additional purpose of this paper is to present a coupling mode using Multiple Program-Multiple Data (MPMD) message passing communication that allows the user to run 3D FE problems together with 2D and/or 3D MHD problems with the respective simulation domains coupled through a common circuit equation. The MPMD coupling capability is used here to model the dynamic coupling of a notional ferroelectric generator with an RP-87 exploding bridgewire detonator. The simulated bridgewire heats up and bursts under current generated by simulated depoling of the ferroelectric generator, as a demonstration of the MPMD capability.

Extreme Scalability of DFT-Based QM/MM MD Simulations Using MiMiC.

We present a highly scalable DFT-based QM/MM implementation developed within MiMiC, a recently introduced multiscale modeling framework that uses a loose-coupling strategy in conjunction with a multiple-program multiple-data (MPMD) approach. The computation of electrostatic QM/MM interactions is parallelized exploiting both distributed- and shared-memory strategies. Here, we use the efficient CPMD and GROMACS programs as QM and MM engines, respectively. The scalability is demonstrated through large-scale benchmark simulations of realistic biomolecular systems employing non-hybrid and hybrid GGA exchange-correlation functionals. We show that the loose-coupling strategy adopted in MiMiC, with its inherent high flexibility, does not carry any significant computational overhead compared to a tight-coupling scheme. Furthermore, we demonstrate that the adopted parallelization strategy enables scaling up to 13,000 CPU cores with efficiency above 70%, thus making DFT-based QM/MM MD simulations using hybrid functionals at the nanosecond scale accessible.

MiMiC: A Novel Framework for Multiscale Modeling in Computational Chemistry.

We present a flexible and efficient framework for multiscale modeling in computational chemistry (MiMiC). It is based on a multiple-program multiple-data (MPMD) model with loosely coupled programs. Fast data exchange between programs is achieved through the use of MPI intercommunicators. This allows exploiting the existing parallelization strategies used by the coupled programs while maintaining a high degree of flexibility. MiMiC has been used in a new electrostatic embedding quantum mechanics/molecular mechanics (QM/MM) implementation coupling the highly efficient CPMD and GROMACS programs, but it can also be extended to use other programs. The framework can also be utilized to extend the partitioning of the system into several domains that can be treated using different models, such as models based on wave function or density functional theory as well as coarse-graining and continuum models. The new QM/MM implementation treats long-range electrostatic QM-MM interactions through the multipoles of the QM subsystem which substantially reduces the computational cost without loss of accuracy compared to an exact treatment. This enables QM/MM molecular dynamics (MD) simulations of very large systems.

Matrix-Free Nodal Domain Decomposition With Relaxation For Massively Parallel Finite-Element Computation of EM Apparatus

In this paper, the nodal domain decomposition with relaxation (NDDR) scheme is proposed to solve the nonlinear finite-element (FE) problem in electromagnetic apparatus without assembling the global system of equations. Each sub-domain contains only one node with unknown magnetic vector potential, and the calculation of each sub-domain can be massively parallelized to utilize the prevalent parallel computing architectures. The sub-domain solver has excellent modularity for single instruction multiple data programming with a specific data structure, and the required memory shows a linear increase with the problem size. The NDDR scheme is implemented on both multi-core CPUs and many-core GPUs, and the accuracy and efficiency are discussed for different problem sizes. Result comparison with a commercial FE package shows a speedup of more than 30 times for a magnetostatic case and an average speedup of more than 53 times for a time-domain nonlinear FE case with different time steps while maintaining an error of less than 0.85%.

Massively parallel structural acoustics in Sierra-SD

Sierra-SD is a massively parallel finite element application for structural dynamics and acoustics. Problems on the order of 2 billion degrees of freedom and running on as up to one hundred thousand distributed memory cores have been solved. Sierra-SD offers a wide range of modern acoustic capabilities. Unbounded problems can be solved with either an absorbing boundary condition, infinite elements, or perfectly matched layers. Problems can be solved in the time domain, frequency domain, or as a linear or quadratic eigenvalue problem. Structural acoustics problems can be solved with monolithic strong coupling, or a Multiple Program Multiple Data (MPMD) coupling with the linear and nonlinear structural Sierra applications. One way handoff is available from other applications in the Sandia National Laboratories’ Sierra Mechanics suite, enabling loading through the Lighthill tensor from incompressible fluid codes to calculate far field noise. Real world applications include ship shock loading and vibration of reentry bodies. [Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA-0003525.]

HPCG and HPGMG benchmark tests on multiple program, multiple data (MPMD) mode on Blue Waters—A Cray XE6/XK7 hybrid system

SummaryThe high‐performance conjugate gradients (HPCG) and high‐performance geometric multi‐grid (HPGMG) benchmarks are alternatives to the traditional LINPACK benchmark (HPL) in measuring the performance of modern HPC platforms. We performed HPCG and HPGMG benchmark tests on a Cray XE6/XK7 hybrid supercomputer, Blue Waters at National Center for Supercomputing Applications (NCSA). The benchmarks were tested on CPU‐based and GPU‐enabled nodes separately, and then we analyzed characteristic parameters that affect their performance. Based on our analyses, we performed HPCG and HPGMG runs in multiple program, multiple data (MPMD) mode in Cray Linux Environment in order to measure their hybrid performance on both CPU‐based and GPU‐enabled nodes. We observed and analyzed several performance issues during those tests. Based on lessons learned from this study, we provide recommendations about how to optimize science applications on modern hybrid HPC platforms.

Multiple program/multiple data molecular dynamics method with multiple time step integrator for large biological systems.

Parallelization of molecular dynamics (MD) simulation is essential for investigating conformational dynamics of large biological systems, such as ribosomes, viruses, and multiple proteins in cellular environments. To improve efficiency in the parallel computation, we have to reduce the amount of data transfer between processors by introducing domain decomposition schemes. Also, it is important to optimize the computational balance between real-space non-bonded interactions and reciprocal-space interactions for long-range electrostatic interactions. Here, we introduce a novel parallelization scheme for large-scale MD simulations on massively parallel supercomputers consisting of only CPUs. We make use of a multiple program/multiple data (MPMD) approach for separating the real-space and reciprocal-space computations on different processors. We also utilize the r-RESPA multiple time step integrator on the framework of the MPMD approach in an efficient way: when the reciprocal-space computations are skipped in r-RESPA, processors assigned for them are utilized for half of the real-space computations. The new scheme allows us to use twice as many as processors that are available in the conventional single program approach. The best performances of all-atom MD simulations for 1 million (STMV), 8.5 million (8_STMV), and 28.8 million (27_STMV) atom systems on K computer are 65, 36, and 24 ns/day, respectively. The MPMD scheme can accelerate 23.4, 10.2, and 9.2 ns/day from the maximum performance of single-program approach for STMV, 8_STMV, and 27_STMV systems, respectively, which correspond to 57%, 39%, and 60% speed up. This suggests significant speedups by increasing the number of processors without losing parallel computational efficiency. © 2016 Wiley Periodicals, Inc.

SIMD studies in the LHCb reconstruction software

During the data taking process in the LHC at CERN, millions of collisions are recorded every second by the LHCb Detector. The LHCb Online computing farm, counting around 15000 cores, is dedicated to the reconstruction of the events in real-time, in order to filter those with interesting Physics. The ones kept are later analysed Offline in a more precise fashion on the Grid. This imposes very stringent requirements on the reconstruction software, which has to be as efficient as possible.Modern CPUs support so-called vector-extensions, which extend their Instruction Sets, allowing for concurrent execution across functional units. Several libraries expose the Single Instruction Multiple Data programming paradigm to issue these instructions. The use of vectorisation in our codebase can provide performance boosts, leading ultimately to Physics reconstruction enhancements.In this paper, we present vectorisation studies of significant reconstruction algorithms. A variety of vectorisation libraries are analysed and compared in terms of design, maintainability and performance. We also present the steps taken to systematically measure the performance of the released software, to ensure the consistency of the run-time of the vectorised software.

A parallel computational model based on mobile agents for high performance computing

Abstract The goal of this paper is to present a new massively parallel virtual machine model, designed for parallel and high performance computing on distributed systems. The proposed model allows us to build a polymorphic grid computing assigned to solve fine grained parallel problems over different well known, SIMD (Single Instruction Multiple Data), SPMD (Single Program Multiple Data), MIMD (Multiple Instruction Multiple Data) ) and MPMD (Multiple Program Multiple Data) machine structures. This model is built using dynamic distributed Virtual Processors Units (VPU). Each VPU corresponds to a mobile agent deployed in a physical processing unit. Each physical processing unit is assigned to a node of the considered distributed system. These units may be heterogeneous machines, they are characterized by their physical performance indices. The proposed model is able to include different computing unit technologies, such as supercomputers, simple desktop or smart phones etc. The VPUs are designed to communicate with each other asynchronously by exchanging, in local or remote manner, the ACL messages (Agent Communication Language) containing data, instructions or any task to be performed. In this model a special agent is designed to represent the host of the parallel virtual machine and to manage the following activities: the life cycle of the VPUs, the load balancing control and the parallel programs to run. In this model VPUs can also use a virtual shared memory represented by hierarchical mobile agents. All the properties offered to the proposed model, are easily designed thanks to the flexibility and the mobility of the multi agent systems. This leads to a strong computation model for high perform-

Orchestrator: Guarding Against Voltage Emergencies in Multithreaded Applications

Voltage emergency (VE) has become a critical challenge with decreasing feature size and increasing power capacity. Destructive core interference is one main source of VE in multicore processors. We observed that the applications following single program and multiple data programming model tend to spark domain-wide destructive core interference because multiple threads exhibit similar power activity. We analyze and quantify this effect and propose one low-cost solution, Orchestrator, to avoid voltage droop synergy among cores. Orchestrator leverages the thread diversity to smooth voltage droops in multicore architectures based on thread scheduling. The thread migration impact on performance is also considered. Experimental results show that Orchestrator can significantly reduce VEs, thereby improving performance.

Implementation and scaling of the fully coupled Terrestrial Systems Modeling Platform (TerrSysMP v1.0) in a massively parallel supercomputing environment – a case study on JUQUEEN (IBM Blue Gene/Q)

Abstract. Continental-scale hyper-resolution simulations constitute a grand challenge in characterizing nonlinear feedbacks of states and fluxes of the coupled water, energy, and biogeochemical cycles of terrestrial systems. Tackling this challenge requires advanced coupling and supercomputing technologies for earth system models that are discussed in this study, utilizing the example of the implementation of the newly developed Terrestrial Systems Modeling Platform (TerrSysMP v1.0) on JUQUEEN (IBM Blue Gene/Q) of the Jülich Supercomputing Centre, Germany. The applied coupling strategies rely on the Multiple Program Multiple Data (MPMD) paradigm using the OASIS suite of external couplers, and require memory and load balancing considerations in the exchange of the coupling fields between different component models and the allocation of computational resources, respectively. Using the advanced profiling and tracing tool Scalasca to determine an optimum load balancing leads to a 19% speedup. In massively parallel supercomputer environments, the coupler OASIS-MCT is recommended, which resolves memory limitations that may be significant in case of very large computational domains and exchange fields as they occur in these specific test cases and in many applications in terrestrial research. However, model I/O and initialization in the petascale range still require major attention, as they constitute true big data challenges in light of future exascale computing resources. Based on a factor-two speedup due to compiler optimizations, a refactored coupling interface using OASIS-MCT and an optimum load balancing, the problem size in a weak scaling study can be increased by a factor of 64 from 512 to 32 768 processes while maintaining parallel efficiencies above 80% for the component models.

Large electromagnetic simulation by hybrid approach on large‐scale parallel computing systems

SummaryThis paper deals with the parallel computing and computation time prediction applied to computational electromagnetics based on rigorous methods for large and complex structures, such as tunnels and airplane cabins. The main objective of this work is represented by the electromagnetic computation of an airplane cabin in order to characterize the propagation channel for a given frequency inside the structure. A well‐known three‐dimensional numerical method, transmission‐line matrix, is adopted for this work. Transmission‐line matrix modelling method is applied to volumes, using the symmetrical condensed node. The planar structures are modelled with the mode matching approach. In order to analyse the electromagnetic field at high frequencies inside large structures as an airplane cabin, the transmission‐line matrix mesh step is very small compared with the size of the physical structure. Thus, the total amount of data that need to be processed exceeds the RAM memory presented on a traditional computing machine. The implementation of the parallel transmission‐line matrix algorithm is a viable solution to this problem taking full advantage of the parallel and distributed architectures, as clusters, grids and supercomputers. The results prove the benefits of the grid computing and supercomputer environments to solve electrically large structures. The parallel hybrid approach, based on Message Passing Interface, is designed for Single Programme Multiple Data programming model. The study highlights the role of parallelization scheme, grid versus supercomputer, with respect to the size of the problem and its repartition. In order to access the computing resources available on different parallel computing platforms, users need to respect a charter containing time and resource number restrictions when doing reservations. So, a time prediction model based on the profile of the transmission‐line matrix algorithm has been developed to allow safe reservations of the computing nodes on grid environment. To have a good agreement between the measured and the predicted values for the computation time, the problem of the cache misses appearing during the simulation has been considered in the profile‐based prediction model of the transmission‐line matrix/modal hybrid application, according to the processor cache memory size. The maximum error for the estimated values of the computation times is about 6.09%. The relevance of this work is based on the fact that is given a complete tool to deal with the large electromagnetic structures: a parallel transmission‐line matrix implementation to take the advantage of the distributed architectures and a time prediction model in order to optimize the resource reservation and the simulation execution on the computing nodes. Copyright © 2014 John Wiley & Sons, Ltd.

A Parallel EM Algorithm for Model-Based Clustering Applied to the Exploration of Large Spatio-Temporal Data

We develop a parallel expectation–maximization (EM) algorithm for multivariate Gaussian mixture models and use it to perform model-based clustering of a large climate dataset. Three variants of the EM algorithm are reformulated in parallel and a new variant that is faster is presented. All are implemented using the single program, multiple data programming model, which is able to take advantage of the combined collective memory of large distributed computer architectures to process larger datasets. Displays of the estimated mixture model rather than the data allow us to explore multivariate relationships in a way that scales to arbitrary size data. We study the performance of our methodology on simulated data and apply our methodology to a high-resolution climate dataset produced by the community atmosphere model (CAM5). This article has supplementary material online.

Guest Editors' Introduction: Special Section on Many-Task Computing

IT is our honor to serve as guest editors of this special section of the IEEE Transactions on Parallel and Distributed Systems (TPDS) on many-task computing (MTC). This section focuses on the methods required to manage and execute large multiple program multiple data (MPMD) computations on large clusters, grids, clouds, and supercomputers. We are pleased to present 10 high-quality contributions chosen from 42 submissions, on resource management, data-intensive computing, applications, and MTC on supercomputers, grids, and clouds. We introduce the term many-task computing (MTC) [2] for computations that bridge the gap between high-performance computing (HPC) and high-throughput computing (HTC) [1]. MTC differs from HTC in its emphasis on using many computing resources over short periods of time to accomplish many computational tasks (both dependent and independent), for which primary metrics are measured in seconds (e.g., FLOPS, tasks/sec., MB/s I/O rates), as opposed to jobs per month. MTC computations comprise multiple distinct activities, coupled via files, shared memory, or message passing. Tasks may be small or large, uniprocessor or multiprocessor, or compute-intensive or data-intensive. The set of tasks may be static or dynamic, homogeneous or heterogeneous, or loosely coupled or tightly coupled. The number of tasks, quantity of computing, and volumes of data may be large. Today’sHPCsystemsareaviableplatformforMTC[3],but large MTC applications can stress HPC hardware and sotware. Challenges include local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, application scalability, and understanding the limitations of HPC systems in order to identify good candidate MTC applications [4]. MTC applications can also be executed on cloud systems, but face other challenges there, for example, relating to internode communication performance. Three recent MTC workshops (MTAGS, http://dsl.cs. uchicago.edu/MTAGS10/) and this special section attracted 142 abstracts and 110 paper submissions, from which 41 papers were accepted. Papers covered resource management, data-intensive computing, applications, and MTC on supercomputers, grids, and clouds. More than 1,000 people have participated as coauthors, program committee members, reviewers, and attendees in these venues. We are well beyond a critical mass for a new, thriving community, which is quickly expanding.

Mapping of settlements in high-resolution satellite imagery using high performance computing

Classifying urban land cover from high-resolution satellite imagery is challenging, and those challenges are compounded when the imagery databases are very large. Accurate land cover data is a crucial component of the population distribution modeling efforts of the Oak Ridge National Laboratory’s (ORNL) LandScan Program. Currently, LandScan Program imagery analysts manually interpret high-resolution (1–5 m) imagery to augment existing satellite-derived medium (30 m) and coarse (1 km) resolution land cover datasets. For LandScan, the high-resolution image archives that require interpretation are on the order of terabytes. The goal of this research is to automate human settlement mapping by utilizing ORNL’s high performance computing capabilities. Our algorithm employs gray-level and local edge-pattern co-occurrence matrices to generate texture and edge patterns. Areas of urban land cover correlate with statistical features derived from these texture and edge patterns. We have parallelized our algorithms for implementation on a 64-node system using a single instruction multiple data programming model (SIMD) with Message Passing Interface (MPI) as the communication mode. Our parallel-configured classifier performs 30–40 times faster than stand-alone alternatives. We have tested our system on IKONOS imagery. The early results are promising, pointing towards future large-scale classification of human settlements at high-resolution.

An efficient parallel computation of unsteady incompressible viscous flow with elastic moving and compliant boundaries on unstructured grids

This paper presents the development and validation of a parallel unstructured-grid fluid–structure interaction (FSI) solver for the simulation of unsteady incompressible viscous flow with long elastic moving and compliant boundaries. The Navier–Stokes solver on unstructured moving grid using the arbitrary Lagrangian Eulerian formulation is based on the artificial compressibility approach and a high-order characteristics-based finite-volume scheme. Both unsteady flow and FSI are calculated with a matrix-free implicit dual time-stepping scheme. A membrane model has been formulated to study fluid flow in a channel with an elastic membrane wall and their interactions. This model can be employed to calculate arbitrary wall movement and variable tension along the membrane, together with a dynamic mesh method for large deformation of the flow field. The parallelization of the fluid–structure solver is achieved using the single program multiple data programming paradigm and message passing interface for communication of data. The parallel solver is used to simulate fluid flow in a two-dimensional channel with and without moving membrane for validation and performance evaluation purposes. The speedups and parallel efficiencies obtained by this method are excellent, using up to 16 processors on a SGI Origin 2000 parallel computer. A maximum speedup of 23.14 could be achieved on 16 processors taking advantage of an improved handling of the membrane solver. The parallel results obtained are compared with those using serial code and they are found to be identical. Copyright © 2005 John Wiley & Sons, Ltd.

Varying the domain size of the dynamic load-balancing algorithm DASUD for SPMD and MPMD programming scenarios

Dynamic load balancing is a key problem for the efficient use of parallel systems when solving applications with unpredictable load estimates. However, depending on the underlying programming paradigm Single Program Multiple Data (SPMD) or Multiple Program Multiple Data (MPMD) the balancing requirements vary. In SPMD scenarios, a perfect load balance is desired, whereas in MPMD scenarios it might be better to quickly obtain a large reduction in load imbalance in a short period of time. We propose extending the local domain of a given processor in the load-balancing algorithms to find a better scope for each paradigm. For that purpose, we present a generalised version of the Diffusion Algorithm Searching Unbalanced Domains (called ds-DASUD), which extends the local domain of each processor beyond its immediate neighbour. ds-DASUD belongs to the iterative distributed load-balancing (IDLB) class and, in its original formulation, operates in a diffusion scheme where a processor balances its load with all its immediate neighbours (ds=1). We evaluate this algorithm for the two programming paradigms varying the domain size. The evaluation was carried out using two simulators (load-balancing and network simulators) for a large set of load distributions that exhibit different degrees of initial workload unbalancing. These distributions were applied to torus and hypercube topologies, and the number of processors ranged from 8 to 128. From these experiments, we conclude that the 1-DASUD fits well for SPMD scenarios, whereas for MPMD 3-DASUD and ((d/2)+1)-DASUD for hypercube and torus topologies, respectively, obtain the best trade-off between the imbalance reduction (up to 85%) and the cost incurred in reaching it.

PSBLAS

Many computationally intensive problems in engineering and science give rise to the solution of large, sparse, linear systems of equations. Fast and efficient methods for their soltion are very important because these systems usually occur in the innermost loop of the computational scheme. Parallelization is often necessary to achieve an acceptable level of performance. This paper presents the design, implementation, and interface of a library of Basic Linear Algebra Subroutines for sparse matrices (PSBLAS) which is specifically tailored to distributed-memory computers. PSBLAS enables easy, efficient, and portable implementations of parallel iterative solvers for linear systems. The interface keeps in view a Single Program Multiple Data programming model on distributed-memory machines. However, the architecture of the library does not exclude an implementation in different paradigms, such as those based on the shared-memory model.