Distributed Memory Platforms Research Articles

Runtime support for CPU-GPU high-performance computing on distributed memory platforms

IntroductionHardware heterogeneity is here to stay for high-performance computing. Large-scale systems are currently equipped with multiple GPU accelerators per compute node and are expected to incorporate more specialized hardware. This shift in the computing ecosystem offers many opportunities for performance improvement; however, it also increases the complexity of programming for such architectures.MethodsThis work introduces a runtime framework that enables effortless programming for heterogeneous systems while efficiently utilizing hardware resources. The framework is integrated within a distributed and scalable runtime system to facilitate performance portability across heterogeneous nodes. Along with the design, this paper describes the implementation and optimizations performed, achieving up to 300% improvement on a single device and linear scalability on a node equipped with four GPUs.ResultsThe framework in a distributed memory environment offers portable abstractions that enable efficient inter-node communication among devices with varying capabilities. It delivers superior performance compared to MPI+CUDA by up to 20% for large messages while keeping the overheads for small messages within 10%. Furthermore, the results of our performance evaluation in a distributed Jacobi proxy application demonstrate that our software imposes minimal overhead and achieves a performance improvement of up to 40%.DiscussionThis is accomplished by the optimizations at the library level and by creating opportunities to leverage application-specific optimizations like over-decomposition.

Balancing Tracking Granularity and Parallelism in Many-Task Systems: The Horizons Approach

Reducing the need for users to manually manage the details of work and data distribution is an important goal of high-level many-task runtime systems. For distributed memory platforms this means that the runtime system has to keep track of both fine-grained task dependencies and data residency meta-information. The amount of such meta-information is proportional to the granularity of parallelism which needs to be managed, introducing a trade-off. More precise tracking of data state allows leveraging more opportunities for compute and transfer parallelism, while also introducing more overhead. As such, the fidelity of the information being tracked needs to be managed carefully, ideally without introducing additional latency, communication or substantial compute overhead. We present the “Horizons” approach, designed to fulfill these goals. Specifically, horizons allow for the effective and efficient management of parallelism and the coalescing of previous fine-grained tracking information while maintaining an easily configurable scheduling window with full information precision. As an additional benefit, they provide consistent cluster-wide decision points without requiring any inter-node communication, and effectively cap the size of state tracking data structures even in the presence of problematic access patterns. Experimental evaluation on microbenchmarks and dry runs demonstrates that horizons are effective in keeping the scheduling complexity constant, while their own overhead is negligible—below 10μs\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$10\\, \\upmu {\\rm s}$$\\end{document} per horizon when building a command graph for 512 GPUs. We additionally demonstrate the performance impact of horizons—as well as their low overhead—on a real-world application.

Topology-free immersed boundary method for incompressible turbulence flows: An aerodynamic simulation for “dirty” CAD geometry

To design a method to solve the issues of handling “dirty” and highly complex geometries, the topology-free method combined with the immersed boundary method is presented for viscous and incompressible flows at a high Reynolds number. The method simultaneously employs a ghost-cell technique and distributed forcing technique to impose the boundary conditions. An axis-projected interpolation scheme is used to avoid searching failures during fluid and solid identification. This method yields a topology-free immersed boundary, which particularly suits flow simulations of highly complex geometries. Difficulties generally arise when generating the calculation grid for these scenarios. This method allows dirty data to be handled without any preparatory treatment work to simplify or clean-up the geometry. This method is also applicable to the coherent structural turbulence model employed in this study. The verification cases, used in conjunction with the second-order central-difference scheme, resulted in first-order accuracy at finer resolution, although the coarser resolution retained second-order accuracy. This method is fully parallelized for distributed memory platforms. In this study, the accuracy and fidelity of this method were examined by simulating the flow around the bluff body, past a flat plate, and past dirty spheres. These simulations were compared with experimental data and other established results. Finally, results from the simulation of practical applications demonstrate the ability of the method to model highly complex, non-canonical three-dimensional flows. The countermeasure based on the accurate classification of geometric features has provided a robust and reasonable solution.

Efficient and Accurate Electromagnetic Angular Sweeping of Rough Surfaces by MPI Parallel Randomized Low-Rank Decomposition

A message passing interface (MPI) parallel scheme on distributed memory platforms is developed for the low-rank decomposition to accommodate the memory requirement during angular sweeping of a rough surface in terms of tapered wave incidence. Numerical examples, including that conducted on a $160\times 160$ square wavelength rough surface, are carried out to demonstrate the performance of the proposed MPI angular sweeping with respect to accuracy, efficiency, scalability, and the peak memory requirement.

A tutorial-driven introduction to the parallel finite element library FEMPAR v1.0.0

This work is a user guide to the FEMPAR scientific software library. FEMPAR is an open-source object-oriented framework for the simulation of partial differential equations (PDEs) using finite element methods on distributed-memory platforms. It provides a rich set of tools for numerical discretization and built-in scalable solvers for the resulting linear systems of equations. An application expert that wants to simulate a PDE-governed problem has to extend the framework with a description of the weak form of the PDE at hand (and additional perturbation terms for non-conforming approximations). We show how to use the library by going through three different tutorials. The first tutorial simulates a linear PDE (Poisson equation) in a serial environment for a structured mesh using both continuous and discontinuous Galerkin finite element methods. The second tutorial extends it with adaptive mesh refinement on octree meshes. The third tutorial is a distributed-memory version of the previous one that combines a scalable octree handler and a scalable domain decomposition solver. The exposition is restricted to linear PDEs and simple geometries to keep it concise. The interested user can dive into more tutorials available in the FEMPAR public repository to learn about further capabilities of the library, e.g., nonlinear PDEs and nonlinear solvers, time integration, multi-field PDEs, block preconditioning, or unstructured mesh handling. Program summaryProgram Title:FEMPARProgram Files doi:http://dx.doi.org/10.17632/dtx487wp57.1Licensing provisions: GNU General Public License 3Programming language: MPI, Fortran2003/2008 (Object-Oriented Programming features)Nature of problem: Computational simulation of a broad range of large-scale application problems governed by Partial Differential EquationsSolution method: Arbitrary-order grad-, curl-, and div-conforming finite elements on n-cube and n-simplex meshes. Continuous and Discontinuous Galerkin FEM. Adaptive Mesh Refinement and Coarsening via forests-of-octrees. Diagonally Implicit Runge–Kutta time integrators. Newton–Raphson linearization. Block preconditioning for multiphysics applications. Multilevel Balancing Domain Decomposition by Constraints preconditioning. Krylov subspace iterative solvers. Sparse direct solvers.Additional comments: Program Github repository https://github.com/fempar/fempar Program website http://www.fempar.org

Distributed-memory parallelization of the aggregated unfitted finite element method

The aggregated unfitted finite element method (AgFEM) is a methodology recently introduced in order to address conditioning and stability problems associated with embedded, unfitted, or extended finite element methods. The method is based on removal of basis functions associated with badly cut cells by introducing carefully designed constraints, which results in well-posed systems of linear algebraic equations, while preserving the optimal approximation order of the underlying finite element spaces. The specific goal of this work is to present the implementation and performance of the method on distributed-memory platforms aiming at the efficient solution of large-scale problems. In particular, we show that, by considering AgFEM, the resulting systems of linear algebraic equations can be effectively solved using standard algebraic multigrid preconditioners. This is in contrast with previous works that consider highly customized preconditioners in order to allow one the usage of iterative solvers in combination with unfitted techniques. Another novelty with respect to the methods available in the literature is the problem sizes that can be handled with the proposed approach. While most of previous references discussing linear solvers for unfitted methods are based on serial non-scalable algorithms, we propose a parallel distributed-memory method able to efficiently solve problems at large scales. This is demonstrated by means of a weak scaling test defined on complex 3D domains up to 300M degrees of freedom and one billion cells on 16K CPU cores in the Marenostrum-IV platform. The parallel implementation of the AgFEM method is available in the large-scale finite element package FEMPAR.

Managing Pending Events in Sequential and Parallel Simulations Using Three-tier Heap and Two-tier Ladder Queue

Performance of sequential and parallel Discrete Event Simulations (DES) is strongly influenced by the data structure used for managing and processing pending events. Accordingly, we propose and evaluate the effectiveness of our multi-tiered (two- and three-tier) data structures and our Two-tier Ladder Queue, for both sequential and optimistic parallel simulations on distributed memory platforms. Our experiments compare the performance of our data structures against a performance-tuned version of the Ladder Queue, which has been shown to outperform many other data structures for DES. The core simulation-based empirical assessments are in C++ and are based on 2,500 configurations of well-established PHOLD and PCS benchmarks. In addition, we use an Avian Influenza Epidemic Model (AIM) for experimental analyses. We have conducted experiments on two computing clusters with different hardware to ensure our results are reproducible. Moreover, to fully establish the robustness of our analysis and data structures, we have also implemented pertinent queues in Java and verified consistent, reproducible performance characteristics. Collectively, our analyses show that our three-tier heap and two-tier ladder queue outperform the Ladder Queue by 60× in some simulations, particularly those with higher concurrency per Logical Process (LP), in both sequential and Time Warp synchronized parallel simulations.

Efficient Subpopulation Based Parallel TLBO Optimization Algorithms

A numerous group of optimization algorithms based on heuristic techniques have been proposed in recent years. Most of them are based on phenomena in nature and require the correct tuning of some parameters, which are specific to the algorithm. Heuristic algorithms allow problems to be solved more quickly than deterministic methods. The computational time required to obtain the optimum (or near optimum) value of a cost function is a critical aspect of scientific applications in countless fields of knowledge. Therefore, we proposed efficient algorithms parallel to Teaching-learning-based optimization algorithms. TLBO is efficient and free from specific parameters to be tuned. The parallel proposals were designed with two levels of parallelization, one for shared memory platforms and the other for distributed memory platforms, obtaining good parallel performance in both types of parallel architectures and on heterogeneous memory parallel platforms.

A highly scalable parallel encoder version of the emergent JEM video encoder

In 2016, 73% of total Internet traffic came from video transmission and this percentage is expected to reach 82% by 2021. These figures show the importance of using video compression standards that maximize video quality while minimizing the necessary bandwidth. In 2013, the HEVC standard was released accounting for an approximate 50% bit rate saving compared to H.264/AVC while maintaining the same reconstruction quality. To address increases in video IP traffic, a new generation of video coding techniques is required that achieve higher compression rates. Compression improvements are being implemented in a software package known as the Joint Exploration Test Model. In this work, we present two parallel JEM model solutions specifically designed for distributed memory platforms for both All Intra and Random Access coding modes. The proposed parallel algorithms achieved high levels of efficiency, in particular for the All Intra mode. They also showed great scalability.

Efficient distributed memory management with RDMA and caching

Recent advancements in high-performance networking interconnect significantly narrow the performance gap between intra-node and inter-node communications, and open up opportunities for distributed memory platforms to enforce cache coherency among distributed nodes. To this end, we propose GAM, an efficient distributed in-memory platform that provides a directory-based cache coherence protocol over remote direct memory access (RDMA). GAM manages the free memory distributed among multiple nodes to provide a unified memory model, and supports a set of user-friendly APIs for memory operations. To remove writes from critical execution paths, GAM allows a write to be reordered with the following reads and writes, and hence enforces partial store order (PSO) memory consistency. A light-weight logging scheme is designed to provide fault tolerance in GAM. We further build a transaction engine and a distributed hash table (DHT) atop GAM to show the ease-of-use and applicability of the provided APIs. Finally, we conduct an extensive micro benchmark to evaluate the read/write/lock performance of GAM under various workloads, and a macro benchmark against the transaction engine and DHT. The results show the superior performance of GAM over existing distributed memory platforms.

A randomized least squares solver for terabyte-sized dense overdetermined systems

We present a fast randomized least-squares solver for distributed-memory platforms. Our solver is based on the Blendenpik algorithm, but employs multiple random projection schemes to construct a sketch of the input matrix. These random projection sketching schemes, and in particular the use of the randomized Discrete Cosine Transform, enable our algorithm to scale the distributed memory vanilla implementation of Blendenpik to terabyte-sized matrices and provide up to ×7.5 speedup over a state-of-the-art scalable least-squares solver based on the classic QR algorithm. Experimental evaluations on terabyte scale matrices demonstrate excellent speedups on up to 16,384 cores on a Blue Gene/Q supercomputer.

A matrix-algebraic formulation of distributed-memory maximal cardinality matching algorithms in bipartite graphs

We describe parallel algorithms for computing maximal cardinality matching in a bipartite graph on distributed-memory systems. Unlike traditional algorithms that match one vertex at a time, our algorithms process many unmatched vertices simultaneously using a matrix-algebraic formulation of maximal matching. This generic matrix-algebraic framework is used to develop three efficient maximal matching algorithms with minimal changes. The newly developed algorithms have two benefits over existing graph-based algorithms. First, unlike existing parallel algorithms, cardinality of matching obtained by the new algorithms stays constant with increasing processor counts, which is important for predictable and reproducible performance. Second, relying on bulk-synchronous matrix operations, these algorithms expose a higher degree of parallelism on distributed-memory platforms than existing graph-based algorithms.We report high-performance implementations of three maximal matching algorithms using hybrid OpenMP-MPI and evaluate the performance of these algorithm using more than 35 real and randomly generated graphs. On real instances, our algorithms achieve up to 200 ? speedup on 2048 cores of a Cray XC30 supercomputer. Even higher speedups are obtained on larger synthetically generated graphs where our algorithms show good scaling on up to 16,384 cores.

Optimization of the Processing of Data Streams on Roughly Characterized Distributed Resources

The AS4DR (Adaptive Scheduling for Distributed Resources) scheduling method presented in this paper aims at maximizing throughput, when processing several data streams by divisible load applications on star-shaped distributed memory platforms, with available speeds for communicating and computing which may be poorly estimated, or varying over time. The total workload is supposed to be unknown. According to the computation cost model, AS4DR can either maximize throughput, or CPU utilization by avoiding data-starvation of the computing units. An experimental assessment of the adaptation of the workload distribution to the variation of the communicating and computing speeds has been performed that shows that the use of AS4DR can significantly improve the throughput. This paper also experimentally assesses a resource selection method to set up star-shaped clusters of distributed resources, so as to process efficiently a set of data streams with AS4DR.

Distributed memory parallel approaches for HEVC encoder

The HEVC video coding standard launched on 2013, is able to reduce to the half, on average, the bit stream size produced by H.264/AVC encoder at the same video quality, but it requires nearly 70 % more time than H.264/AVC to encode a video sequence. In this paper we propose several parallelization approaches to the HEVC encoder. Our proposals, for distributed memory platforms, work at a coarse grain level parallelization, being one group of pictures (GOP) the basic structure. These approaches encode simultaneously several GOPs. To obtain good parallel performance, a right GOP conformation and distribution should be applied.

A COMPLEX MIX-SHIFTED PARALLEL QR ALGORITHM FOR THE C-METHOD

The C-method is an exact method for analyzing gratings and rough surfaces. This method leads to large-size dense complex non-Hermitian eigenvalue. In this paper, we introduce a parallel QR algorithm that is specifically designed for the C-method. We define the early shift for the matrix according to the observed properties. We propose a combination of the early shift , Wilkinson's shift and exceptional shift together to accelerate convergence. First, we use the early shift in order to have quick deflation of some eigenvalues. The multi-window bulge chain chasing and parallel aggressive early deflation are used. This approach ensures that most computations are performed in level 3 BLAS operations. The aggressive early deflation approach can detect deflation much quicker and accelerate convergence. Mixed MPI-Open MP techniques are used for performing the codes to hybrid shared and distributed memory platforms. We validate our approach by comparison with experimental data for scattering patterns of two-dimensional rough surfaces.

Complex Network Partitioning Using Label Propagation

We present PuLP (partitioning using label propagation), a parallel and memory-efficient graph partitioning method specifically designed to partition low-diameter networks with skewed degree distributions on shared-memory multicore platforms. Graph partitioning is an important problem in scientific computing because it impacts the execution time and energy efficiency of computations on distributed-memory platforms. Partitioning determines the in-memory layout of a graph, which affects locality, intertask load balance, communication time, and overall memory utilization. A novel feature of our PuLP method is that it optimizes for multiple objective metrics simultaneously, while satisfying multiple partitioning constraints. Using our method, we are able to partition a web crawl with billions of edges on a single compute server in under a minute. For a collection of test graphs, we show that PuLP uses up to $7.8\times$ less memory than state-of-the-art partitioners and is $5.0\times$ faster, on average, than a...

Topology-oblivious optimization of MPI broadcast algorithms on extreme-scale platforms

Significant research has been conducted in collective communication operations, in particular in MPI broadcast, on distributed memory platforms. Most of the research efforts aim to optimize the collective operations for particular architectures by taking into account either their topology or platform parameters. In this work we propose a simple but general approach to optimization of the legacy MPI broadcast algorithms, which are widely used in MPICH and Open MPI. The proposed optimization technique is designed to address the challenge of extreme scale of future HPC platforms. It is based on hierarchical transformation of the traditionally flat logical arrangement of communicating processors. Theoretical analysis and experimental results on IBM BlueGene/P and a cluster of the Grid’5000 platform are presented.

Session details: Special Issue on Big Spatial Data (Part 2)

This issue is Part 2 of SIGSPATIAL Special issue on Spatial Big Data. The issue includes five interesting articles. The first two articles present two approaches for MapReduce-based systems built to support Big Spatial Data. In particular, the first article by Eldawy and Mokbel presents the ecosystem of SpatialHadoop, which includes the SpatialHadoop engine with a built-in support for spatial data, the Pigeon spatial language, and a visualization layer. The second article by Wang et al. presents the Hadoop-GIS system; a scalable and high performance spatial data warehousing system for large scale spatial queries, equipped with GPU-based geometric algorithms integrated into the MapReduce pipeline. The next two articles focus on using GPUs as a means of accelerating large-scale spatial data processing. In particular, in the third article, Prasad et al. lay out a vision for accelerating geo-spatial computations and analytics using a combination of shared and distributed memory platforms,with GPUs and hundreds to thousands processing cores. The fourth article by Zhang et al. presents data parallel designs on GPU-accelerated clusters for spatial indexing, spatial joins, and various other spatial operations. The issue is concluded with an article by Bhaduri et al., discussing various trending applications of big data at Oak Ridge National Lab (ORNL). This concludes the second part of the special issued on Big Spatial Data; all composed of ten interesting and thought-provoking articles. I would like to sincerely thank all the authors for their invited contributions.

Experimenting with low-overhead OpenMP runtime on IBM Blue Gene/Q

As newer supercomputers continue to increase the number of threads, there is growing pressure on applications to exploit more of the available parallelism in their codes, including coarse-, medium-, and fine-grain parallelism. OpenMP™ is one of the dominant shared-memory programming models and is well suited for exploiting medium- and fine-grain parallelism. OpenMP research has focused on application tuning, compiler optimizations, programming-model extensions, and porting to distributed-memory platforms; however, we have found that current algorithms used to implement basic OpenMP constructs have significant overheads and scale poorly. In this paper, we explore low-overhead, scalable algorithms for creating parallel regions and demonstrate reductions in overhead of up to a factor of 5 on an IBM Blue Gene®/Q node.

Custom Microcoded Dynamic Memory Management for Distributed On-Chip Memory Organizations

Multiprocessor system-on-chip (MPSoCs) have attracted significant attention since they are recognized as a scalable paradigm to interconnect and organize a high number of cores. Current multicore embedded systems exhibit increased levels of dynamic behavior, leading to unexpected memory footprint variations unknown at design time. Dynamic memory management (DMM) is a promising solution for such types of dynamic systems. Although some efficient dynamic memory managers have been proposed for conventional bus-based MPSoC platforms, there are no DMM solutions regarding the constraints and the opportunities delivered by the physical distribution of multiple memory nodes of the platform. In this work, we address the problem of providing customized microcoded DMM on MPSoC platforms with distributed memory organization. Customization is enabled at application- and platform-level. Results show that customized microcoded DMM can serve approximately 7× more allocation requests compared to pure distributed memory platforms and perform 25% faster than the corresponding high-level implementation in C language.