Hybrid OpenMP Research Articles

Heterogeneous programming using OpenMP and CUDA/HIP for hybrid CPU-GPU scientific applications

Hybrid computer systems combine compute units (CUs) of different nature like CPUs, GPUs and FPGAs. Simultaneously exploiting the computing power of these CUs requires a careful decomposition of the applications into balanced parallel tasks according to both the performance of each CU type and the communication costs among them. This paper describes the design and implementation of runtime support for OpenMP hybrid GPU-CPU applications, when mixed with GPU-oriented programming models (e.g. CUDA/HIP). The paper describes the case for a hybrid multi-level parallelization of the NPB-MZ benchmark suite. The implementation exploits both coarse-grain and fine-grain parallelism, mapped to compute units of different nature (GPUs and CPUs). The paper describes the implementation of runtime support to bridge OpenMP and HIP, introducing the abstractions of Computing Unit and Data Placement. We compare hybrid and non-hybrid executions under state-of-the-art schedulers for OpenMP: static and dynamic task schedulings. Then, we improve the set of schedulers with two additional variants: a memorizing-dynamic task scheduling and a profile-based static task scheduling. On a computing node composed of one AMD EPYC 7742 @ 2.250 GHz (64 cores and 2 threads/core, totalling 128 threads per node) and 2 × GPU AMD Radeon Instinct MI50 with 32 GB, hybrid executions present speedups from 1.10× up to 3.5× with respect to a non-hybrid GPU implementation, depending on the number of activated CUs.

Parallelization of Array Method with Hybrid Programming: OpenMP and MPI

For parallelization of applications with high processing times and large amounts of storage in High Performance Computing (HPC) systems, shared memory programming and distributed memory programming have been used; a parallel application is represented by Parallel Task Graphs (PTGs) using Directed Acyclic Graphs (DAGs). For the execution of PTGs in HPC systems, a scheduler is executed in two phases: scheduling and allocation; the execution of the scheduler is considered an NP-complete combinatorial problem and requires large amounts of storage and long processing times. Array Method (AM) is a scheduler to execute the task schedule in a set of clusters; this method was programmed sequentially, analyzed and tested using real and synthetic application workloads in previous work. Analyzing the proposed designs of this method in this research work, the parallelization of the method is extended using hybrid OpenMP and MPI programming in a server farm and using a set of geographically distributed clusters; at the same time, a novel method for searching free resources in clusters using Lévy random walks is proposed. Synthetic and real workloads have been experimented with to evaluate the performance of the new parallel schedule and compare it to the sequential schedule. The metrics of makespan, waiting time, quality of assignments and search for free resources were evaluated; the results obtained and described in the experiments section show a better performance with the new version of the parallel algorithm compared to the sequential version. By using the parallel approach with hybrid programming applied to the extraction of characteristics of the PTGs, applied to the search for geographically distributed resources with Lévy random walks and applied to the metaheuristic used, the results of the metrics are improved. The makespan is decreased even when the loads increase, the times of the tasks in the waiting queue are decreased, the quality of assignments in the clusters is improved by causing the tasks with their subtasks to be assigned in the same clusters or in cluster neighbors and, finally, the searches for free resources are executed in different geographically distributed clusters, not sequentially.

Multiscale modeling of coupled thermo-mechanical behavior of granular media in large deformation and flow

Heat generation and transfer in a granular material can be intricately coupled with their mechanical responses, playing a key role in causing excessive large deformation, flow and failure of the material. The coupling may manifest in various forms, including thermal induced stress, mechanically induced heat and thermally induced melting in granular media. We propose a novel hierarchical multiscale modeling framework, TM-DEMPM, to model the coupled thermo-mechanical behavior in granular media which may undergo large deformation and flow. Material Point Method (MPM) is hierarchically coupled with Discrete Element Method (DEM) to offer physics-based, natural scale-crossing simulations of thermo-mechanical granular responses without assuming complicated phenomenological constitutive models. To offer speedup for the numerical solution, hybrid OpenMP and GPU-based parallelization is proposed to take advantage of the hierarchical computing structure of the framework. The proposed framework may provide an effective and efficient pathway to next-generation simulation of engineering-scale large-deformation problems that involve complicated thermo-mechanical coupling in granular media.

Fast and Accurate Solution of Integral Formulations of Large MQS Problems Based on Hybrid OpenMP–MPI Parallelization

This paper proposes an optimal strategy to parallelize the solution of large 3D magneto-quasi-static (MQS) problems, by combining the MPI and OpenMP approaches. The studied numerical problem comes from a weak-form integral formulation of a MQS problem and is finally cast in terms of a large linear system to be solved by means of a direct method. For this purpose, two main tasks are identified: the assembly and the inversion of the matrices. The paper focuses on the optimization of the resources required for assembling the matrices, by exploiting the feature of a hybrid OpenMP–MPI approach. Specifically, the job is shared between clusters of nodes in parallel by adopting an OpenMP paradigm at the node level and a MPI one at the process level between nodes. Compared with other solutions, such as pure MPI, this hybrid parallelization optimizes the available resources, with respect to the speed, allocated memory, and the communication between nodes. These advantages are clearly observed in the case studies analyzed in this paper, coming from the study of large plasma fusion machines, such as the fusion reactor ITER. Indeed, the MQS problems associated with such applications are characterized by a huge computational cost that requires parallel computing approaches.

Runtime Support for Openmp Hybrid Cpu-Gpu Applications

Runtime Support for Openmp Hybrid Cpu-Gpu Applications

Computational Efficiency Examination of a Regional Numerical Weather Prediction Model using KISTI Supercomputer NURION

For well-resolving extreme weather events, running numerical weather prediction model with high resolution in time and space is essential. We explore how efficiently such modeling could be, using NURION. We have examined one of community numerical weather prediction models, WRF, and KISTI’s 5th supercomputer NURION of national HPC. Scalability of the model has been tested at first, and we have compared the computational efficiency of hybrid openMP + MPI runs with pure MPI runs. In addition to those parallel computing experiments, we have tested a new storage layer called burst buffer to see whether it can accelerate frequent I/O. We found that there are significant differences between the computational environments for running WRF model. First of all, we have tested a sensitivity of computational efficiency to the number of cores per node. The sensitivity experiments certainly tell us that using all cores per node does not guarantee the best results, rather leaving several cores per node could give more stable and efficient computation. For the current experimental configuration of WRF, moreover, pure MPI runs gives much better computational performance than any hybrid openMP + MPI runs. Lastly, we have tested burst buffer storage layer that is expected to accelerate frequent I/O. However, our experiments show that its impact is not consistently positive. We clearly confirm the positive impact with relatively smaller problem size experiments while the impact was not seen with bigger problem experiments. Significant sensitivity to the different computational configurations shown this paper strongly suggests that HPC users should find out the best computing environment before massive use of their applications

High-Performance Computing Implementations of Agent-Based Economic Models for Realizing 1:1 Scale Simulations of Large Economies

We present a scalable high-performance computing implementation of an agent-based economic model using distributed + shared-memory hybrid parallelization paradigms, capable of simulating 1:1 scale models of large economies like the eurozone. Agent-based economic models consist of millions of agents interacting over several graphs, which are either centralized or scale-free in nature. While most of the interactions are bi-directional, the interaction graphs are dense and random and keep evolving as the simulation progresses. These characteristics cause a very large and unknown number of random communications among MPI processes, posing challenges to developing scalable parallel extensions. Further, random access to large volume of data makes the algorithms highly memory-bound, severely degrading computational performance. Adopting various strategies inspired by the real-world functioning of economies, we reduce the large unknown number of communications to a known handful number. Memory-intensive algorithms are improved to make these cache-efficient, and advanced MPI functions are used to minimize communication overhead, thereby attaining higher performance and scalability. Further, an MPI + OpenMP hybrid model is developed to best utilize modern many-core computing nodes with low per-core memory capacity. It is demonstrated that our implementation can simulate a full fledged economic model with 331 million agents within 108 seconds using 128 CPU cores attaining 70 percent strong scalability.

Lattice Boltzmann simulations of magnetic particles in a three-dimensional microchannel

A lattice Boltzmann model for magnetic particles in a fluid is established, in which both hydrodynamic and magnetic interactions are fully coupled. The interpolated bounce back scheme is adopted to describe the fluid-particle interaction. We combine the implicit particle velocity updated scheme and the modified momentum-exchange method to make the algorithm stable and also maintain the Galilean invariance. Moreover, MPI + OpenMP hybrid programming mode is used for parallel computing. The model is validated by numerical simulation of the sedimentation of a particle in a tank. We investigate the motion of magnetic particles in a microchannel with carrier fluid under the uniform magnetic field and hybrid magnetic field, and then the magnetic particles flow through a variable-section microchannel is studied. The results indicate that the structures formed by magnetic particles can be regulated by controlling the external magnetic field and flow field; besides, the more ordered structures have more influence on the fluid flow.

Spatiotemporal parallelization of an analytical heat conduction model for additive manufacturing via a hybrid OpenMP + MPI approach

The ability to do thermal simulations for entire additive manufacturing builds is a key computational problem facing the additive manufacturing community; however, complex numerical models considering multiple physical phenomena currently do not have the capacity for simulations at this scale. To this end, conduction only analytic models offer a viable approach due to the massive drop in computational expense. Here, we extend an existing implementation which uses a governing equation which can be evaluated at any point in space and time. This implementation already utilizes OpenMP with a spatial decompositions scheme stemming from a melt pool tracking algorithm. We then combine this with a parallel in time (PinT) approach to make the problem highly parallelizable. The new scheme, which uses MPI for internode communication and OpenMP for intranode communication, is shown to scale very well across multiple computational nodes. This approach results in the ability to simulate the 3D solidification conditions for entire layers of additively manufactured parts in minutes making part scale thermal simulations more practical.

CFD-DEM simulation of fluidization of multisphere- modelled cylindrical particles

A hybrid Open-MP (Shared memory) – MPI (Distributed memory) parallel Computational Fluid Dynamics (CFD)-Discrete Element Method (DEM) model has been developed for simulating fluidization of cylindrical particles in a fluidized bed. In the DEM model, five, seven, nineteen and sixty overlapping spherical particles in different formation methods including single-row, multi-row and hybrid formations have been used to represent one cylindrical particle. The simulation results of the hybrid representation for mimicking a cylindrical particle are more accurate than other representations; particularly in terms of, particle number density along the bed height and particle orientation distribution in the bed, in comparison with previously reported experimental data. The simulation results show that the preferential orientation of the particles is horizontal in the fluidize bed, but the number of particles with vertical orientation increases when they approach to walls. Only multi-row and hybrid formations could capture this increase near the walls.

Exploiting OpenMP and OpenACC to accelerate a geometric approach to molecular docking in heterogeneous HPC nodes

In drug discovery, molecular docking is the task in charge of estimating the position of a molecule when interacting with the docking site. This task is usually used to perform screening of a large library of molecules, in the early phase of the process. Given the amount of candidate molecules and the complexity of the application, this task is usually performed using high-performance computing (HPC) platforms. In modern HPC systems, heterogeneous platforms provide a better throughput with respect to homogeneous platforms. In this work, we ported and optimized a molecular docking application to a heterogeneous system, with one or more GPU accelerators, leveraging a hybrid OpenMP and OpenACC approach. The target application focuses on the virtual screening phases in the drug discovery process, and it is based on geometric transformations of the target ligands. We prove that our approach has a better exploitation of the node compared to pure CPU/GPU data splitting approaches, reaching a throughput improvement up to 25% while considering the same computing node.

A parallel unidirectional coupled DEM-PBM model for the efficient simulation of computationally intensive particulate process systems

The accurate modeling of the physics underlying particulate processes is complicated and requires significant computational capabilities to solve using particle-based models. In this work, a unidirectional multi-scale approach was used to model the high shear wet granulation process. A multi-dimensional population balance model (PBM) was developed with a mechanistic kernel, which in turn obtained collision data from the discrete element modeling (DEM) simulation. The PBM was parallelized using a hybrid OpenMP+MPI approach. The DEM simulations were performed using LIGGGHTS, which was parallelized using MPI. Speedups of approximately 14 were obtained for the PBM simulations and approximately 12 for the DEM simulations. The uni-directional coupling of DEM to PBM was performed using middle-ware components (RADICAL-Pilot) that did not require modifications of the DEM or PBM codes, yet supported flexible execution on high-performance platforms. Results demonstrate scaling from 1 to 128 cores for the PBM and up to 256 cores for the DEM. The proposed method, implementations and middle-ware enable the modeling of high shear wet granulation process faster than existing approaches in literature.

Improvement of workload balancing using parallel loop self-scheduling on Intel Xeon Phi

In recent years, Intel promotes its new product Xeon Phi coprocessor, which is similar to the x86 architecture coprocessor. It has about 60 cores and can be regarded as a single computing node, with the computing power that cannot be ignored. This work aims to improve the workload balance by parallel loop self-scheduling scheme performed on Xeon Phi-based computer cluster. The proposed concept is implemented by hybrid MPI and OpenMP parallel programming in C language. Since parallel loop self-scheduling composes of static and dynamic allocation, weighting algorithm is adopted in the static part, while the well-known loop self-scheduling is adopted in dynamic part. The loop block is partitioned according to the weighting of MIC and HOST nodes. Accordingly, Xeon Phi with many-core is adopted to implement parallel loop self-scheduling. Finally, we test the performance in the experiments by four applicable problems: matrix multiplication, sparse matrix multiplication, Mandelbrot set and circuit meet. The experimental results indicate how to do the weight allocation and which scheduling method can achieve the best performance.

A Fixed-Mesh Approach for Gas-Liquid-Rigid Interaction Problems

A fixed-mesh approach has been developed to facilitate predicting a certain class of gas-liquid-rigid interaction problems. All the basic equations are discretized on a Cartesian grid in a finite-difference manner. The Volume-Of-Fluid and Boundary Data Immersion methods are employed to treat the gas-liquid and fluid-rigid interfaces, respectively. A hybrid OpenMP- MPI approach is adopted for parallel computing. The developed code is validated through comparisons with experiments of an oil-air flow driven by a rotating disk with holes. The simulated results demonstrate the capability in capturing the velocity distributions.

Performance optimizations for scalable implicit RANS calculations with SU2

In this paper, we present single- and multi-node optimizations of SU2, a widely-used, open-source Computational Fluid Dynamics application, aimed at improving performance and scalability for implicit Reynolds-averaged Navier–Stokes calculations on unstructured grids. Typical industry-standard implementations are currently limited by unstructured accesses, variable degrees of parallelism, as well as the global synchronizations inherent in traditionally used Krylov linear solvers. Therefore, we rely on aggressive single-node optimizations, such as hierarchical parallelism, dynamic threading, compacted memory layout, and vectorization, along with a communication-friendly agglomeration (geometric) linear multigrid solver. Based on results with the well-known ONERA M6 geometry, our single core and shared memory optimizations result in a speedup of 2.6X on the latest 14-core Intel® Xeon™ 11Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance E5-2697v3 processor when compared to the baseline SU2 implementation with 14 MPI ranks. In multi-node settings, the hybrid OpenMP+MPI multigrid implementation achieves 2X higher parallel efficiency on 256 nodes over conventional Krylov-based (GMRES) methods.

Megadock 4.0. An Ultra-High-Performance Protein-Protein Docking Software for Heterogeneous Supercomputers

Protein-protein interaction (PPI) plays a core role in cellular functions. In recent years, PPI prediction methods based on protein docking have been developed and have been applied for large-scale PPI network prediction based on tertiary structures. However, such network prediction requires much computing resources, and a faster PPI prediction method is eagerly demanded. We have developed an ultra-high-throughput PPI prediction system based on rigid-body protein docking, “MEGADOCK 4.0”. MEGADOCK 4.0 can perform faster docking based on its original scoring function and implementation for heterogeneous supercomputers. MEGADOCK 4.0 was implemented by two parallelization techniques for massively parallel supercomputing environment; the MPI and OpenMP hybrid parallelization and a CUDA-based implementation on GPUs. MEGADOCK 4.0 achieved an excellent scalability on supercomputing environments, such as K computer and TSUBAME 2.5. We also present newly implemented MEGADOCK for many integrated core architecture like Intel Xeon Phi co-processor. In addition, we have already applied MEGADOCK system to a number of interactome analyses such as (a) bacterial chemotaxis pathway consisted of 101 PDB structures and required 10,201 pairs of protein dockings, (b) human apoptosis pathway consisted of 158 PDB structures and required 24,964 pairs of protein dockings, and (c) human epidermal growth factor receptor related pathway consisted of 1,921 PDB structures and required 3,690,241 pairs of protein dockings. We present a new protein-protein docking engine aimed at exhaustive docking of millions of protein pairs. The system was shown to be scalable when running on thousands of nodes and multiple accelerators.

Real-time and real-space program tuned in K-computer

We have transported the code from Earth Simulator (ES) to K-Computer (KC) keeping a scalability. The base code achieved remarkable linear scalability and significant parallelization efficiency in EC before. The reason of success is that, taking into account the architectural differences between ES and KC, we improve the program step by step. Actually, the program showed a 37% improvement in the total performance with the implementations such as hybrid OpenMP and MPI parallelization, and the optimal rank mapping, and the MPI communication concealment, including removing the unnecessary processes such as zero clear and data copy.

Developing a scalable hybrid MPI/OpenMP unstructured finite element model

The trend of all modern computer architectures, and the path to exascale, is towards increasing numbers of lower power cores, with a decreasing memory to core ratio. This imposes a strong evolutionary pressure on algorithms and software to efficiently utilise all levels of parallelism available on a given platform while minimising data movement. Unstructured finite elements codes have long been effectively parallelised using domain decomposition methods, implemented using libraries such as the Message Passing Interface (MPI). However, there are many optimisation opportunities when threading is used for intra-node parallelisation for the latest multi-core/many-core platforms. The benefits include increased algorithmic freedom, reduced memory requirements, cache sharing, reduced number of partitions, less MPI communication and I/O overhead.In this paper, we report progress in implementing a hybrid OpenMP–MPI version of the unstructured finite element code Fluidity. For matrix assembly kernels, the OpenMP parallel algorithm uses graph colouring to identify independent sets of elements that can be assembled concurrently with no race conditions. In this phase there are no MPI overheads as each MPI process only assembles its own local part of the global matrix. We use an OpenMP threaded fork of PETSc to solve the resulting sparse linear systems of equations. We experiment with a range of preconditioners, including HYPRE which provides the algebraic multigrid preconditioner BoomerAMG where the smoother is also threaded. Since unstructured finite element codes are well known to be memory latency bound, particular attention is paid to ccNUMA architectures where data locality is particularly important to achieve good intra-node scaling characteristics. We also demonstrate that utilising non-blocking algorithms and libraries are critical to mixed-mode application so that it can achieve better parallel performance than the pure MPI version.

Parallelization of 2D MPDATA EULAG algorithm on hybrid architectures with GPU accelerators

• Parallelization of stencil-based 2D MPDATA algorithm across both GPUs and CPUs. • Minimization of communication between CPU and GPU at the cost of extra computations. • Adaptation of MPDATA to CPUs using space and temporal blocking techniques. • Adaptation of MPDATA to GPUs based on hierarchical decomposition . • Approach to optimization of MPDATA on GPUs using autotuning technique. EULAG (Eulerian/semi-Lagrangian fluid solver) is an established computational model developed for simulating thermo-fluid flows across a wide range of scales and physical scenarios. The dynamic core of EULAG includes the multidimensional positive definite advection transport algorithm (MPDATA) and elliptic solver. In this work we investigate aspects of an optimal parallel version of the 2D MPDATA algorithm on modern hybrid architectures with GPU accelerators, where computations are distributed across both GPU and CPU components. Using the hybrid OpenMP–OpenCL model of parallel programming opens the way to harness the power of CPU–GPU platforms in a portable way. In order to better utilize features of such computing platforms, comprehensive adaptations of MPDATA computations to hybrid architectures are proposed. These adaptations are based on efficient strategies for memory and computing resource management, which allow us to ease memory and communication bounds, and better exploit the theoretical floating point efficiency of CPU–GPU platforms. The main contributions of the paper are: • method for the decomposition of the 2D MPDATA algorithm as a tool to adapt MPDATA computations to hybrid architectures with GPU accelerators by minimizing communication and synchronization between CPU and GPU components at the cost of additional computations; • method for the adaptation of 2D MPDATA computations to multicore CPU platforms, based on space and temporal blocking techniques; • method for the adaptation of the 2D MPDATA algorithm to GPU architectures, based on a hierarchical decomposition strategy across data and computation domains, with support provided by the developed GPU task scheduler allowing for the flexible management of available resources; • approach to the parametric optimization of 2D MPDATA computations on GPUs using the autotuning technique, which allows us to provide a portable implementation methodology across a variety of GPUs. Hybrid platforms tested in this study contain different numbers of CPUs and GPUs – from solutions consisting of a single CPU and a single GPU to the most elaborate configuration containing two CPUs and two GPUs. Processors of different vendors are employed in these systems – both Intel and AMD CPUs, as well as GPUs from NVIDIA and AMD. For all the grid sizes and for all the tested platforms, the hybrid version with computations spread across CPU and GPU components allows us to achieve the highest performance. In particular, for the largest MPDATA grids used in our experiments, the speedups of the hybrid versions over GPU and CPU versions vary from 1.30 to 1.69, and from 1.95 to 2.25, respectively.

Photon-plasma: A modern high-order particle-in-cell code

We present the photon-plasma code, a modern high order charge conserving particle-in-cell code for simulating relativistic plasmas. The code is using a high order implicit field solver and a novel high order charge conserving interpolation scheme for particle-to-cell interpolation and charge deposition. It includes powerful diagnostics tools with on-the-fly particle tracking, synthetic spectra integration, 2D volume slicing, and a new method to correctly account for radiative cooling in the simulations. A robust technique for imposing (time-dependent) particle and field fluxes on the boundaries is also presented. Using a hybrid OpenMP and MPI approach, the code scales efficiently from 8 to more than 250.000 cores with almost linear weak scaling on a range of architectures. The code is tested with the classical benchmarks particle heating, cold beam instability, and two-stream instability. We also present particle-in-cell simulations of the Kelvin-Helmholtz instability, and new results on radiative collisionless shocks.