Non-blocking Communication Research Articles

Two-level parallel load balancing strategy for accelerating DSMC simulations in near-continuum gases

The Direct Simulation Monte Carlo (DSMC) algorithm is widely employed for simulating rarefied gas flows and is increasingly applied in near-continuum regimes for research and engineering purposes. However, its computational demands, notably load imbalance and extended simulation time, hinder widespread adoption. Addressing these challenges, this paper introduces the Two-Level parallel load balancing strategy. This novel approach combines thread-level and multi-process parallelism to enhance load balancing and reduce simulation time. Key features include a thread-level load-decoupling strategy implemented via OpenMP and a multi-process load balancing mechanism employing distributed memory via MPI. Building upon our previous [Formula: see text] [L. Li, W. Ren and B. Zhang, J. Aeronaut. Astronaut. Aviat. Ser. A 46, 88 (2014)] approach, the load balancing mechanism utilizes Stop At Risk (SAR) criteria for repartitioning with METIS. Additionally, a specialized data transmission mechanism utilizing MPI nonblocking communication minimizes global communication between processes. Validation and evaluation are performed using four hypersonic flow cases around a cylinder and sphere, demonstrating significant improvements. Notably, the proposed strategy achieves [Formula: see text] enhancement over the [Formula: see text] strategy under 512 CPU cores compared to 16 CPU cores, and reduces between-process communication time with [Formula: see text]. These advancements contribute to enhancing the effectiveness of the DSMC algorithm in near-continuum aerodynamic simulations.

Exploring energy saving opportunities in fault tolerant HPC systems

Nowadays, improving the energy efficiency of high-performance computing (HPC) systems is one of the main drivers in scientific and technological research. As large-scale HPC systems require some fault-tolerant method, the opportunities to reduce energy consumption should be explored. In particular, rollback-recovery methods using uncoordinated checkpoints prevent all processes from re-executing when a failure occurs. In this context, it is possible to take actions to reduce the energy consumption of the nodes whose processes do not re-execute. This work is an extension of a previous one, in which we proposed a series of strategies to manage energy consumption at failure-time. In this work, we have enriched our simulator and the experimentation by including non-blocking communications (with and without system buffering) and a largest number of candidate processes to be analyzed. We have called the latter as cascade analysis, because it includes processes that gets blocked by communication indirectly with the failed process. The simulations show that the savings were negligible in the worst case, but in some scenarios, it was possible to achieve significant ones; the maximum saving achieved was 90% in a time interval of 16 minutes. As a result, we show the feasibility of improving energy efficiency in HPC systems in the presence of a failure.

Parallel Immersed Boundary Method for Two-Phase Flows on DCU Clusters

The immersed boundary method (IBM) is widely used in simulating multiphase flows. DCU (Deep Computing Unit) acceleration device is a kind of GPU-like device, which is developed by GPU technology authorised by AMD. To achieve an efficient numerical simulation of two-phase flows, we develop a parallel DCU-based immersed boundary method using the MPI+HIP programming model. To solve the slow convergence problem caused by domain partitioning in multigrid methods, a coarse grid aggregation (CGA) technique is effectively employed. We adopt a nonblocking communication approach with HIP streams to overlap computation and communication. To validate the proposed method running on DCU clusters, we conducted two numerical experiments of droplet deformation in shear flow and the rising bubble. Performance tests show that a speedup of 181 times more than the CPU version can be achieved with 32 DCUs in a 1024 × 512 × 512 mesh grid.

An Alternate GPU-Accelerated Algorithm for Very Large Sparse LU Factorization

The LU factorization of very large sparse matrices requires a significant amount of computing resources, including memory and broadband communication. A hybrid MPI + OpenMP + CUDA algorithm named SuperLU3D can efficiently compute the LU factorization with GPU acceleration. However, this algorithm faces difficulties when dealing with very large sparse matrices with limited GPU resources. Factorizing very large matrices involves a vast amount of nonblocking communication between processes, often leading to a break in SuperLU3D calculation due to the overflow of cluster communication buffers. In this paper, we present an improved GPU-accelerated algorithm named SuperLU3D_Alternate for the LU factorization of very large sparse matrices with fewer GPU resources. The basic idea is “divide and conquer”, which means dividing a very large matrix into multiple submatrices, performing LU factorization on each submatrix, and then assembling the factorized results of all submatrices into two complete matrices L and U. In detail, according to the number of available GPUs, a very large matrix is first divided into multiple submatrices using the elimination tree. Then, the LU factorization of each submatrix is alternately computed with limited GPU resources, and its intermediate LU factors from GPUs are saved to the host memory or hard disk. Finally, after finishing the LU factorization of all submatrices, these factorized submatrices are assembled into a complete lower triangular matrix L and a complete upper triangular matrix U, respectively. The SuperLU3D_Alternate algorithm is suitable for hybrid CPU/GPU cluster systems, especially for a subset of nodes without GPUs. To accommodate different hardware resources in various clusters, we designed the algorithm to run in the following three cases: sufficient memory for GPU nodes, insufficient memory for GPU nodes, and insufficient memory for the entire cluster. The results from LU factorization test on different matrices in various cases show that the larger the matrix is, the more efficient this algorithm is under the same GPU memory consumption. In our numerical experiments, SuperLU3D_Alternate achieves speeds of up to 8× that of SuperLU3D (CPU only) and 2.5× that of SuperLU3D (CPU + GPU) on the hybrid cluster with six Tesla V100S GPUs. Furthermore, when the matrix is too big to be handled by SuperLU3D, SuperLU3D_Alternate can still utilize the cluster’s host memory or hard disk to solve it. By reducing the amount of data exchange to prevent exceeding the buffer’s limit of the cluster MPI nonblocking communication, our algorithm enhances the stability of the program.

A parallel particle swarm optimization framework based on a fork-join thread pool using a work-stealing mechanism

Particle Swarm Optimization (PSO) is one of the most popular optimization algorithms that has been adopted in various fields, including design, scheduling, and biochemistry. However, the algorithm is time-consuming when facing high-dimensional or multi-objective optimizing problems. Parallel PSO is thus proposed to improve its computing efficiency, and many studies have been conducted. However, the low-level system design is seldom considered in parallel programming, which may have a nonnegligible impact on computing efficiency, creating a gap between low-level optimization and high-level algorithm design. Therefore, this paper proposes a Parallel Asynchronous PSO (PAPSO) framework based on thread pools utilizing multicore processors and adopts a cross-level approach to bridge the gap. A series of experiments are designed and conducted to examine how the aforementioned method can improve the parallel execution efficiency of PSO compared with the OpenMP framework and nonparallel PSO. Results indicate that PAPSO can significantly improve PSO computing efficiency by reducing the elapsed time, approaching approximately 20% optimization compared with OpenMP. Additionally, it achieves an approximately linear speedup of up to 4.5 threads. In addition, the particle communication experiment shows that the nonblocking communication protocol is effective for maintaining the computing elapsed time in the same level facing different neighborhood sizes. Finally, the work-stealing mechanism achieves an average of 16% improvement for general computing scenarios and maintain up to 16% improvement for imbalanced workload scenarios. Generally, the major contribution of this paper is that we innovate the thread pooling management concept with the fork-join model in parallelizing PSO to make sufficient use of multiple core CPUs for parallel computing through multithread programming.

A Minimal Round-Trip Strategy Based on Graph Matching for Parallel DGTD Method With Local Time-Stepping

The load distribution in the local time stepping (LTS) method significantly impacts its computing efficiency. This letter proposes a minimal round-trip (MRT) strategy of the LTS method to balance the communication load of the discontinuous Galerkin time-domain (DGTD) method. By discovering the matching of the connected graph of computing nodes, independent communication with a similar load can be done in the same round trip to minimize waiting between nodes in nonblocking communication, thereby decreasing the communication time of the DGTD-LTS technique. The numerical results indicate that the MRT strategy reduces the communication time between processors by 50% and improves the parallel performance when the LTS method is implemented in the DGTD method. The parallel scale of the MRT approach may be increased to 16 000 nodes (1 040 000 cores) on the supercomputer, and the parallel efficiency is greater than 73.8%.

Deneb: An open-source high-performance multi-physical flow solver based on high-order DRM-DG method

High-order methods are being recognized as powerful tools for handling scale-resolving simulations over complex geometry. However, several obstacles still block their complete applications to practical engineering problems: a compromise between accuracy and efficiency on mixed-curved meshes, inherent vulnerability to numerical oscillations, and lack of open-source high-performance solvers for researchers. To address these issues, we present Deneb, an open-source high-order accurate numerical solver that enables high-performance scale-resolving simulations on PDE-based flow systems. Deneb uses the physical domain-based modal discontinuous Galerkin (DG) method; thus, it can provide an arbitrary high-order accurate solution on mixed-curved meshes and has the potential for handling polyhedral meshes as well. The direct reconstruction method (DRM) efficiently executes the numerical integration of DG volume and surface integrals without accuracy loss on non-affine elements where mapping functions are high-degree. The resulting DRM-DG method eliminates the severe cost of a quadrature-based approach on mixed-curved meshes. Deneb offers explicit and implicit Runge–Kutta methods as well to achieve high-order accuracy in time. In addition, Krylov subspace methods and preconditioners are available for high-performance linear system solving in parallel. Deneb possesses a strong capability to resolve multi-physical shocks without numerical instabilities with the aid of multi-dimensional limiting and artificial viscosity methods. In particular, the hierarchical multi-dimensional limiting process enables efficient computations of supersonic flows without time-step restriction. The current release of Deneb covers the simulations of hypersonic equilibrium and magneto-hydrodynamic flows as well as compressible Navier–Stokes equations, but it has the potential to solve any PDE-based multi-physical flow systems. Several benchmark problems are presented to highlight Deneb's capability to perform scale-resolving and multi-physical flow simulations. A scalability test is also presented to verify the scaling characteristics of Deneb for high-performance computing. Program summaryProgram title: DenebCPC library link to program files:https://doi.org/10.17632/723n5r797n.1Developer's repository link:https://github.com/HojunYouKr/DenebLicensing provisions: BSD-3-ClauseProgramming language: C++17Nature of problem: The physical domain-based modal DG method can achieve the expected order of accuracy with the optimal number of polynomial bases even on non-affine elements. However, the DG method becomes significantly expensive on high-order curved elements when using quadrature rules, blocking its applicability to practical engineering problems. The numerical integration should be much more efficient without compromising accuracy. In addition, the less diffusive nature of high-order methods makes them susceptible to producing spurious numerical oscillations near flow discontinuities, potentially leading to numerical instabilities. Thus, an accurate and robust shock-capturing method is essential to simulate multi-physical flows under compressible regimes. Finally, the solver needs high scalability to perform large-scale computations in parallel.Solution method: DRM is applied to the DG volume and surface integrals to perform efficient numerical integration on non-affine elements without accuracy loss. The resulting method, DRM-DG, provides arbitrary high-order accurate solutions to various PDE-based flow problems on mixed-curved meshes. The solution is also high-order accurate in time due to high-order explicit and implicit Runge–Kutta methods implemented. The external library enables high-performance linear system solving with various preconditioners in parallel. Both multi-dimensional limiting and artificial viscosity methods suppress unwanted subcell oscillations across physical discontinuities. In particular, the limiting methods simulate complex supersonic flows efficiently without time-step restriction. The solver is highly scalable on parallel computing with the aid of non-blocking communications and latency hiding.

The inherent overlapping in the parallel calculation of the Laplacian

A new approach for the parallel computation of the Laplacian in the Fourier domain is presented. This numerical problem inherits the intrinsic sequencing involved in the calculation of any multidimensional Fast Fourier Transform (FFT) where blocking communications assure that its computation is strictly carried out dimension by dimension. Such data dependency vanishes when one considers the Laplacian as the sum of n independent one-dimensional kernels, so that computation and communication can be naturally overlapped with nonblocking communications. Overlapping is demonstrated to be responsible for the speedup figures we obtain when our approach is compared to state-of-the-art parallel multidimensional FFTs.

MPI detach — Towards automatic asynchronous local completion

When aiming for large-scale parallel computing, waiting time due to network latency, synchronization, and load imbalance are the primary opponents of high parallel efficiency. A common approach to hide latency with computation is the use of non-blocking communication. In the presence of a consistent load imbalance, synchronization cost is just the visible symptom of the load imbalance. Tasking approaches as in OpenMP, TBB, OmpSs, or C++20 coroutines promise to expose a higher degree of concurrency, which can be distributed on available execution units and significantly increase load balance. Available MPI non-blocking functionality does not integrate seamlessly into such tasking parallelization. In this work, we present a slim extension of the MPI interface to allow seamless integration of non-blocking communication with available concepts of asynchronous execution in OpenMP and C++. Using our concept allows to span task dependency graphs for asynchronous execution over the full distributed memory application. We furthermore investigate compile-time analysis necessary to transform an application using blocking MPI communication into an application integrating OpenMP tasks with our proposed MPI interface extension.

ITERATIVE AND PARALLEL PERFORMANCE ANALYSIS OF NON-BLOCKING COMMUNICATION ALGORITHMS IN THE MASSIVELY PARALLEL NEUTRON TRANSPORT CODE PIDOTS

The PIDOTS neutral particle transport code utilizes a red/black implementation of the Parallel Gauss-Seidel algorithm to solve the SN approximation of the neutron transport equation on 3D Cartesian meshes. PIDOTS is designed for execution on massively parallel platforms and is capable of using the full resources of modern, leadership class high performance computers. Initial testing revealed that some configurations of PIDOTS’s Integral Transport Matrix Method solver demonstrated unexpectedly poor parallel scaling. Work at Idaho and Los Alamos National Laboratories then revealed that this inefficiency was a result of the accumulation of high-cost latency events in the complex blocking communication networks employed during each PIDOTS iteration. That work explored the possibility of minimizing those inefficiencies while maintaining a blocking communications model. While significant speedups were obtained, it was shown that fully mitigating the problem on general-purpose platforms was highly unlikely for a blocking code. This work continues that analysis by implementing a deeply interleaved non-blocking communication model into PIDOTS. This new model benefits from the optimization work performed on the blocking model while also providing significant opportunities to overlap the remaining un-mitigated communication costs with computation. Additionally, our new approach is easily transferable to other similarly spatially decomposed codes. The resulting algorithm was tested on LANL’s Trinity system at up to 32,768 processors and was found at that processor count to effectively hide 100% of MPI communication cost – equivalently 20% of the red/black phase time. It is expected that the implemented interleaving algorithm can fully support far higher processor counts and completely hide communication costs up ~50% of total iteration time.

Hybrid MPI and CUDA Parallelization for CFD Applications on Multi-GPU HPC Clusters

Graphics processing units (GPUs) have a strong floating-point capability and a high memory bandwidth in data parallelism and have been widely used in high-performance computing (HPC). Compute unified device architecture (CUDA) is used as a parallel computing platform and programming model for the GPU to reduce the complexity of programming. The programmable GPUs are becoming popular in computational fluid dynamics (CFD) applications. In this work, we propose a hybrid parallel algorithm of the message passing interface and CUDA for CFD applications on multi-GPU HPC clusters. The AUSM + UP upwind scheme and the three-step Runge–Kutta method are used for spatial discretization and time discretization, respectively. The turbulent solution is solved by the K−ω SST two-equation model. The CPU only manages the execution of the GPU and communication, and the GPU is responsible for data processing. Parallel execution and memory access optimizations are used to optimize the GPU-based CFD codes. We propose a nonblocking communication method to fully overlap GPU computing, CPU_CPU communication, and CPU_GPU data transfer by creating two CUDA streams. Furthermore, the one-dimensional domain decomposition method is used to balance the workload among GPUs. Finally, we evaluate the hybrid parallel algorithm with the compressible turbulent flow over a flat plate. The performance of a single GPU implementation and the scalability of multi-GPU clusters are discussed. Performance measurements show that multi-GPU parallelization can achieve a speedup of more than 36 times with respect to CPU-based parallel computing, and the parallel algorithm has good scalability.

Adjoint computations by algorithmic differentiation of a parallel solver for time-dependent PDEs

A computational fluid dynamics code is differentiated using algorithmic differentiation (AD) in both tangent and adjoint modes. The two novelties of the present approach are (1) the adjoint code is obtained by letting the AD tool Tapenade invert the complete layer of message passing interface (MPI) communications, and (2) the adjoint code integrates time-dependent, non-linear and dissipative (hence physically irreversible) PDEs with an explicit time integration loop running for ca. 106 time steps. The approach relies on using the Adjoinable MPI library to reverse the non-blocking communication patterns in the original code, and by controlling the memory overhead induced by the time-stepping loop with binomial checkpointing. A description of the necessary code modifications is provided along with the validation of the computed derivatives and a performance comparison of the tangent and adjoint codes.

Two-Stage Column Block Parallel LU Factorization Algorithm

Parallel computing is increasingly important in computer architectures, parallel architecture has become ubiquitous in our everyday lives. Novel architectures and programming models pose new challenges to algorithm design and system software development. This paper presents a two-stage column block parallel LU factorization algorithm for multiple-processor architectures. Any given matrix is first partitioned into large blocks, and then, every large block is partitioned into a number of small blocks according to the number of processors. Finally, the small column blocks are allocated to processors in an orderly “serpentine arrangement.” Each iteration of the column block parallel LU factorization is separated into two stages of operation. In the first stage, the first-step factorization operation is processed in advance and nonblocking communication is used to reduce the processor idle and waiting time and improve parallelism. In the second stage, the large blocks are used to satisfy more powerful processors, such as GPUs, which require more data to exploit their computing capabilities. Experiments are conducted on a multicore system and multi-GPU system with different configurations to test the algorithm's performance. Compared with other related column block parallel LU factorizations, the two-stage algorithm exhibits better load balancing and parallel execution time performance.

Asynchronous vs. synchronous interfacing to time-triggered communication systems

Abstract Time-triggered communication facilitates the construction of multi-component real-time systems whose components are in control of their temporal behaviour. However, the interface of a time-triggered communication system has to be accessed with care, to avoid that the temporal independence of components gets lost. This paper shows two interfacing strategies, one for asynchronous interface access (in two variants, one being the new Rate-bounded Non-Blocking Communication protocol) and one for time-aware, synchronized interface access, that allow components to maintain temporal independence. The paper describes and compares these interfacing strategies.

Wide-sense nonblocking multicast in WRMD WDM optical linear array and ring

Multicast communication is the simultaneous transmission of information from a single source to various destinations. It occurs extensively in telecommunication and advanced computing systems including enterprises operating datacenters. WDM optical networks act as the backbone for telecommunication and high performance networks. Wavelengths are scarce resources in WDM optical networks. Wavelengths usage in a network must be limited to decrease its cost and complexity. Optical linear array and ring networks find widespread application in interconnection networks including LAN and WAN. In this paper, wide-sense nonblocking multicast communication is analyzed in wavelength reusable multi-carrier distributed (WRMD) wavelength division multiplexing (WDM) linear array, unidirectional ring and bidirectional ring networks. The sufficient and necessary condition on the minimum wavelength number required for the network to be wide-sense nonblocking is derived for each of the above network topologies in the presence of a single multi-carrier light source (MCLS). An explicit wavelength allotment technique is also proposed for the above network topologies.

A SCATTERING SIMULATION MODEL FOR NONSPHERICAL AEROSOL PARTICLES BASED ON PARALLEL FDTD SCHEME

Abstract. In order to simulate the scattering properties of nonspherical aerosol particles in visible and near infrared band precisely and efficiently, a scattering computation model for aerosol particles based on parallel FDTD (Finite Difference Time Domain) is developed. The basic principle of FDTD is introduced, and a new parallel computation scheme for FDTD is proposed, and is realized by MPI repeated non-blocking communication technique. The FDTD scattering model is validated against Lorenz-Mie theory and T Matrix method. Simulation results show that, the scattering properties obtained parallel FDTD scattering model are qualitatively in good agreement with the T matrix method and Lorenz-Mie theory, validating the accuracy of our model. The relative simulation error of Mueller is slightly larger in forward scattering directions than that in backward directions for particles with small size parameter, while for large particles, the result is opposite.

Asynchronous multi-splitting method for linear and pseudo-linear problems

This paper deals with the solution of fluid-structure interaction problems using an algorithm consisting in the coupling between two solvers, each one related to the fluid and the structure respectively. Due to the difficult decomposition of the domain into sub-domains, we consider for each environment a parallel multi-splitting algorithm which corresponds to an unified presentation of sub-domain methods with or without overlapping. Such method combines several contractant fixed point mappings and, we show that, under appropriate assumptions, each fixed point mapping are contractive in finite dimensional spaces normed by Hilbertian and non-Hilbertian norms. Moreover, we show in a novel way that such study is valid for the parallel synchronous and more generally asynchronous large scale linear systems arising in the solution of fluid-structure interaction problems and can be extended to the case where the displacement of the structure is submitted to some constraints. We perform parallel simulations for a fluid-structure test case on different clusters, considering blocking and non-blocking communications. The performances of the parallel simulations are presented and analyzed.

Wavelength-Reused Hierarchical Optical Network on Chip Architecture for Manycore Processors

Manycore processor is becoming the mainstream platform for cloud computing applications. However, the design of high-performance and sustainable inter-core communication network is still a challenging problem. Optical Network on Chip (ONoC) is an emerging chip-scale optical communication technology with high bandwidth capacity and energy efficiency. In this paper, we present a Wavelength Reused Hierarchical ONoC architecture, WRH-ONoC. It leverages the nonblocking wavelength-routed $\lambda$λ-router and hierarchical networking to reuse the limited number of wavelengths. In WRH-ONoC, all the cores are grouped into multiple subsystems, and the cores in the same subsystem are directly interconnected using a $\lambda$λ-router for nonblocking communication. For inter-subsystem communication, all subsystems are further connected through multiple $\lambda$λ-routers and gateways in a hierarchical manner. Thus, the available wavelengths can be reused in different $\lambda$λ-routers. Furthermore, WRHm-ONoC, an efficient extension with multicast ability is also proposed. Given the numbers of cores and available wavelengths, we derive the minimum hardware requirement, the expected end-to-end delay, and the maximum data rate. Theoretical analysis and simulation results indicate WRH-ONoC achieves prominent improvement on the communication performance and sustainability, e.g., 46.0 percent of reduction on zero-load delay and 72.7 percent of improvement on throughput for 400 cores with the modest hardware/energy costs.

An Enhanced File Transfer Mechanism Using an Additional Blocking Communication Channel and Thread for IoT Environments.

In this paper, we propose an enhanced file transfer mechanism for a communication framework (CM) for Internet of Things (IoT) applications. Our previous file transfer method uses a basic non-blocking communication channel and thread for the CM (non-blocking method), but this method has a cost of adding additional bytes to each original file block. Therefore, it is not suitable for the transfer of large-sized files. Other existing file transfer methods use a separate channel to transfer large-sized files. However, the creation of a separate channel increases the total transmission delay as the transfer frequency increases. The proposed method uses a dedicated blocking communication channel in a separate thread (blocking method). The blocking method uses a separate channel and thread which are dedicated to transferring file blocks. As it creates the separate channel in advance before the file transfer task, the proposed method does not have an additional channel creation cost at the moment of the file transfer. Through file transfer experiments, the blocking method showed a shorter file transfer time than the non-blocking method, and the transmission delay was increased as the file size grew. By supporting both non-blocking and blocking methods, an application can flexibly select the desirable method according to its requirement. If the application requires the transfer of small-sized files infrequently, it can use the non-blocking method. If the application needs to transfer small-sized or large-sized files frequently, a good alternative is to use the blocking method.

In-Hardware Moving Compute to Data Model to Accelerate Thread Synchronization on Large Multicores

In this article, the moving computation to data model (MC2D) is proposed to accelerate thread synchronization by pinning shared data to dedicated cores, and utilize in-hardware core-to-core messaging to communicate critical code execution. The MC2D model optimizes shared data locality by eliminating unnecessary data movement, and alleviates contended synchronization using nonblocking communication between threads. This article evaluates task-parallel algorithms under their synchronization-centric classification to demonstrate that the effectiveness of the MC2D model to exploit performance correlates with the number and frequency of synchronizations. The evaluation on Tilera TILE-Gx72 multicore shows that the MC2D model delivers highest performance scaling gains for ordered and unordered algorithms that expose significant synchronizations due to task and data level dependencies. The MC2D model is also shown to deliver at par performance with the traditional atomic operations based model for highly data parallel algorithms from the unordered category.