Hybrid Parallel Programming Model Research Articles

Implementation and evaluation of MPI 4.0 partitioned communication libraries

Partitioned point-to-point communication primitives provide a performance-oriented mechanism to support a hybrid parallel programming model and have been included in the upcoming MPI-4.0 standard. These primitives enable an MPI library to transfer parts of the data buffer while the application provides partial contributions using multiple threads or tasks or simply pipelines the buffers sequentially. The focus of this paper is the design and implementation of a layered library that provides the functionality of these newer APIs and supports application development using these newer APIs. This library provides an opportunity to explore potential optimizations and identify further enhancements to the APIs. Initial experience in designing this library along with preliminary performance results are presented. In addition, the library is compared to initial prototype libraries that have recently become available that have been updated to the standard-compliant interface. These prototype libraries were built on remote-memory-access (RMA) primitives, offering insight into different implementation strategies. In general, we observe an interesting trade-off space, with the RMA-based implementation proving more performant for send-side partitioning, with increases in perceived bandwidth 8.9x on average over a single send, compared to the persistent-based implementation, which shows improvements 4.0x on average. In comparing the two implementations, we find that the persistent-based implementation enables more overlap for receive-side partitioning up to 5.37X the RMA library’s overlap, while the RMA-based implementation provides better send-side performance of up to 70%.

Empirical investigation: performance and power‐consumption based dual‐level model for exascale computing systems

Exascale computing systems (ECS) are anticipated to perform at Exaflop speed (1018 operations per second) using power consumption <20 MW. This ultrascale performance requires the speedup in the system by thousand-fold enhancement in current Petascale. For future high-performance computing (HPC), power consumption is one of the vital challenges faced to achieve Exaflops through the traditional way of increasing clock-speed. One standard way to attain such significant performance is through massive parallelism. In the early stages, it is hard to decide the promising parallel programming approach that can provide massive parallelism to attain ExaFlops. This article commences with a short description and implementation of algorithms of various hybrid parallel programming models (PPMs) for homogeneous and heterogeneous cluster systems. Furthermore, the authors evaluated performance and power consumption in these hybrid models by implementing in two HPC benchmarking applications such as square matrix multiplication and Jacobi iterative solver for two-dimensional Laplace equation. The results demonstrated that the hybrid of heterogeneous (MPI + X) outperformed to homogeneous parallel programming (MPI + OpenMP) model. This empirical investigation of hybrid PPMs is a leading step for researchers and development communities to select a promising model for emerging ECS.

MPI/OpenMP hybrid parallel algorithm for resolution of identity second-order Møller-Plesset perturbation calculation of analytical energy gradient for massively parallel multicore supercomputers.

A massively parallel algorithm of the analytical energy gradient calculations based the resolution of identity Møller-Plesset perturbation (RI-MP2) method from the restricted Hartree-Fock reference is presented for geometry optimization calculations and one-electron property calculations of large molecules. This algorithm is designed for massively parallel computation on multicore supercomputers applying the Message Passing Interface (MPI) and Open Multi-Processing (OpenMP) hybrid parallel programming model. In this algorithm, the two-dimensional hierarchical MP2 parallelization scheme is applied using a huge number of MPI processes (more than 1000 MPI processes) for acceleration of the computationally demanding O(N5 ) step such as calculations of occupied-occupied and virtual-virtual blocks of MP2 one-particle density matrix and MP2 two-particle density matrices. The new parallel algorithm performance is assessed using test calculations of several large molecules such as buckycatcher C60 @C60 H28 (144 atoms, 1820 atomic orbitals (AOs) for def2-SVP basis set, and 3888 AOs for def2-TZVP), nanographene dimer (C96 H24 )2 (240 atoms, 2928 AOs for def2-SVP, and 6432 AOs for cc-pVTZ), and trp-cage protein 1L2Y (304 atoms and 2906 AOs for def2-SVP) using up to 32,768 nodes and 262,144 central processing unit (CPU) cores of the K computer. The results of geometry optimization calculations of trp-cage protein 1L2Y at the RI-MP2/def2-SVP level using the 3072 nodes and 24,576 cores of the K computer are presented and discussed to assess the efficiency of the proposed algorithm. © 2017 Wiley Periodicals, Inc.

Software Framework for Parallel BEM Analyses with H-matrices Using MPI and OpenMP

A software framework has been developed for use in parallel boundary element method (BEM) analyses. The framework program was parallelized in a hybrid parallel programming model, and both multiple processes and threads were used. Additionally, an H-matrix library for a distributed memory parallel computer was also developed to accelerate the analysis. In this paper, we describe the basic design concept for the framework and details of its implementation. The framework program, which was written with MPI functions and OpenMP directives, is mainly intended to reduce the user’s parallel programming costs. We also show the results of a sample analysis performed with approximately 60,000 unknowns. The numerical results verify the effectiveness of both the parallelization and the H-matrix method. In the test analysis, which was performed using a single core, the H-matrix version of the framework is 17-fold faster than the dense matrix version. The parallel framework program with the H-matrix attains an approximately 50-fold acceleration using 128 cores when compared with sequential computation.

DaSH: A benchmark suite for hybrid dataflow and shared memory programming models

Abstract The current trend in development of parallel programming models is to combine different well established models into a single programming model in order to support efficient implementation of a wide range of real world applications. The dataflow model has particularly managed to recapture the interest of the research community due to its ability to express parallelism efficiently. Thus, a number of recently proposed hybrid parallel programming models combine dataflow and traditional shared memory models. Their findings have influenced the introduction of task dependency in the OpenMP 4.0 standard. This article presents DaSH – the first comprehensive benchmark suite for hybrid dataflow and shared memory programming models. DaSH features 11 benchmarks, each representing one of the Berkeley dwarfs that capture patterns of communication and computation common to a wide range of emerging applications. DaSH also includes sequential and shared-memory implementations based on OpenMP and Intel TBB to facilitate easy comparison between hybrid dataflow implementations and traditional shared memory implementations based on work-sharing and/or tasks. Finally, we use DaSH to evaluate three different hybrid dataflow models, identify their advantages and shortcomings, and motivate further research on their characteristics.

MPI/OpenMP Hybrid Parallel Algorithm of Resolution of Identity Second-Order Møller-Plesset Perturbation Calculation for Massively Parallel Multicore Supercomputers.

A new algorithm for massively parallel calculations of electron correlation energy of large molecules based on the resolution of identity second-order Møller-Plesset perturbation (RI-MP2) technique is developed and implemented into the quantum chemistry software NTChem. In this algorithm, a Message Passing Interface (MPI) and Open Multi-Processing (OpenMP) hybrid parallel programming model is applied to attain efficient parallel performance on massively parallel supercomputers. An in-core storage scheme of intermediate data of three-center electron repulsion integrals utilizing the distributed memory is developed to eliminate input/output (I/O) overhead. The parallel performance of the algorithm is tested on massively parallel supercomputers such as the K computer (using up to 45 992 central processing unit (CPU) cores) and a commodity Intel Xeon cluster (using up to 8192 CPU cores). The parallel RI-MP2/cc-pVTZ calculation of two-layer nanographene sheets (C150H30)2 (number of atomic orbitals is 9640) is performed using 8991 node and 71 288 CPU cores of the K computer.

An Out-of-Core Method for CFD Simulation in Heterogeneous Environment

In recent years, the highly parallel graphics processing unit (GPU) is rapidly gaining maturity as a powerful engine for high performance computer. However, in most computational fluid dynamics (CFD) simulations, the computational capacity of CPU was ignored. In this paper, we propose a hybrid parallel programming model to utilize the computational capacity of both CPU and GPU. Considering the memory amount of CPU and GPU, we also propose an out-of-core method to increase the simulation scale on single node. The experiment results show that the programming model can utilize the computational capacity of both CPU and GPU efficiently and the out-of-core method can increase the simulation scale on single node.

The Hybrid Parallel Algorithm of Online Verification Based on P2P

Taking into account the gradually expanded scale of large interconnected power system, especially the UHV synchronous grid successfully puts into operation. Traditional centralized computing will encounter the bottleneck of hardware computing power. A protection setting online checking hybrid parallel algorithm based on P2P is proposed. Peer-to-peer communications using P2P network technology to achieve inter-regional information fast interactive. The design of MPI+OpenMP hybrid parallel programming models and algorithms is emphatically introduced. Online checking two levels parallel of process-level and thread-level is achieved through the parallel analysis of online checking. Finally, the hybrid parallel algorithm was tested and compared based on P2P technology distributed parallel computing platform. The results show that the proposed algorithm is correct and effective.

Large-scale Simulations of 3D Groundwater Flow Using Parallel Geometric Multigrid Method

The multigrid method used with OpenMP/MPI hybrid parallel programming models is expected to play an important role in large-scale scientific computing on post-peta/exa-scale supercomputer systems. In the present work, the effect of sparse matrix storage formats on the performance of parallel geometric multigrid solvers was evaluated, and a new data structure for the Ellpack-Itpack (ELL) format is proposed. The proposed method is implemented for pGW3D-FVM, a parallel code for 3D groundwater flow simulations using the multigrid method, and the robustness and performance of the code was evaluated on up to 4,096 nodes (65,536 cores) of the Fujistu FX10 supercomputer system at the University of Tokyo. The parallel multigrid solver using the ELL format with coarse grid aggregation provided excellent performance improvement in both weak scaling (13%–35%) and strong scaling (40%–70%) compared to the original code using the CRS format.

A Survey of Parallel Programming Models and Tools in the Multi and Many-Core Era

In this work, we present a survey of the different parallel programming models and tools available today with special consideration to their suitability for high-performance computing. Thus, we review the shared and distributed memory approaches, as well as the current heterogeneous parallel programming model. In addition, we analyze how the partitioned global address space (PGAS) and hybrid parallel programming models are used to combine the advantages of shared and distributed memory systems. The work is completed by considering languages with specific parallel support and the distributed programming paradigm. In all cases, we present characteristics, strengths, and weaknesses. The study shows that the availability of multi-core CPUs has given new impulse to the shared memory parallel programming approach. In addition, we find that hybrid parallel programming is the current way of harnessing the capabilities of computer clusters with multi-core nodes. On the other hand, heterogeneous programming is found to be an increasingly popular paradigm, as a consequence of the availability of multi-core CPUs+GPUs systems. The use of open industry standards like OpenMP, MPI, or OpenCL, as opposed to proprietary solutions, seems to be the way to uniformize and extend the use of parallel programming models.

Parallel iterative solvers for finite-element methods using an OpenMP/MPI hybrid programming model on the Earth Simulator

An efficient parallel iterative method for finite-element method has been developed for symmetric multiprocessor (SMP) cluster architectures with vector processors such as the Earth Simulator. The method is based on a three-level hybrid parallel programming model, including message passing for inter-SMP node communication, loop directives by OpenMP for intra-SMP node parallelization and vectorization for each processing element (PE). Simple 3D linear elastic problems with more than 2.2 × 10 9 DOF have been solved using 3 × 3 block ICCG(0) method with additive Schwarz domain decomposition and PDJDS/CM-RCM reordering on 176 nodes of the Earth Simulator, achieving performance of 3.80 TFLOPS. Furthermore, effect of color number in reordering has been evaluated on various types of computers.

Three-level hybrid vs. flat MPI on the Earth Simulator: Parallel iterative solvers for finite-element method

An efficient parallel iterative method for finite element method has been developed for symmetric multiprocessor (SMP) cluster architectures with vector processors such as the Earth Simulator. The method is based on a three-level hybrid parallel programming model, including message passing for inter-SMP node communication, loop directives by OpenMP for intra-SMP node parallelization and vectorization for each processing element (PE). Simple 3D linear elastic problems with more than 2.2 × 10 9 DOF have been solved using 3 × 3 block ICCG(0) method with additive Schwarz domain decomposition and PDJDS/CM-RCM reordering on 176 nodes of the Earth Simulator, achieving performance of 3.80 TFLOPS. The PDJDS/CM-RCM reordering method provides excellent vector and parallel performance in SMP nodes. A three-level hybrid parallel programming model outperforms flat MPI in the problems involving large numbers of SMP nodes.

MPI/OpenMP Hybrid Parallel Molecular Dynamics Simulation on SMP Cluster Architecture

The authors applied MPI/OpenMP hybrid parallel programming model to molecular dynamics (MD) for simulating a protein structure. In a symmetric multiprocessor (SMP) cluster architecture, it can be expected that the hybrid parallel programming model, which uses the message passing library such as MPI for inter-SMP node communication and the loop directives such as OpenMP for intra-SMP node parallelization, is the most effective one. In this study, the parallel performance of the hybrid style was compared with that of conventional flat parallel programming style, which uses only MPI. The computer environments used here are Hitachi SR8000/MPP placed at the University of Tokyo. When using 16 SMP nodes (128 PEs), it was observed that the parallel efficiency with the hybrid style was 90%, which was 15% higher than that of the flat-MPI for MD simulation of a protein with 33 402 atoms.

Parallel iterative solvers for unstructured grids using a directive/MPI hybrid programming model for the GeoFEM platform on SMP cluster architectures

AbstractIn this paper, an efficient parallel iterative method for unstructured grids developed by the authors for shared memory symmetric multiprocessor (SMP) cluster architectures on the GeoFEM platform is presented. The method is based on a three‐level hybrid parallel programming model, including message passing for inter‐SMP node communication, loop directives for intra‐SMP node parallelization and vectorization for each processing element (PE). Simple 3D elastic linear problems with more than $10^8$ degrees of freedom have been solved by $3\times3$ block ICCG(0) with additive Schwarz domain decomposition and PDJDS/CM‐RCM reordering on 16 SMP nodes of a Hitachi SR8000 parallel computer, achieving a performance of 20 Gflops. The PDJDS/CM‐RCM reordering method provides excellent vector and parallel performance in SMP nodes, and is essential for parallelization of forward/backward substitution in IC/ILU factorization with global data dependency. The method developed was also tested on an NEC SX‐4 and attained 969 Mflops (48.5% of peak performance) using a single processor. The additive Schwarz domain decomposition method provides robustness for the GeoFEM parallel iterative solvers with localized preconditioning. Copyright © 2002 John Wiley &amp; Sons, Ltd.