Multi-core Clusters Research Articles

Extending [formula omitted]-Lop to model concurrent MPI communications in multicore clusters

Achieving optimal performance of MPI applications on current multi-core architectures, composed of multiple shared communication channels and deep memory hierarchies, is not trivial. Formal analysis using parallel performance models allows one to depict the underlying behavior of the algorithms and their communication complexities, with the aims of estimating their cost and improving their performance.LogGP model was initially conceived to predict the cost of algorithms in mono-processor clusters based on point-to-point transmissions with network latency and bandwidth based parameters. It remains as the representative model, with multiple extensions for handling high performance networks, covering particular contention cases, channels hierarchies or protocol costs. These very specific branches lead LogGP to partially lose its initial abstract modeling purpose.More recent lognP represents a point-to-point transmission as a sequence of implicit transfers or data movements. Nevertheless, similar to LogGP, it models an algorithm in a parallel architecture as a sequence of message transmissions, an approach inefficient to model algorithms more advanced than simple tree-based one, as we will show in this work.In this paper, τ–Lop model is extended to multi-core clusters and compared to previous models. It demonstrates the ability to predict the cost of advanced algorithms and mechanisms used by mainstream MPI implementations, such as MPICH or Open MPI, with high accuracy. τ–Lop is based on the concept of concurrent transfers, and applies it to meaningfully represent the behavior of parallel algorithms in complex platforms with hierarchical shared communication channels, taking into account the effects of contention and deployment of processes on the processors. In addition, an exhaustive and reproducible methodology for measuring the parameters of the model is described.

Understanding the Impact of the Interconnection Network Performance of Multi-core Cluster Architectures

Understanding the Impact of the Interconnection Network Performance of Multi-core Cluster Architectures

Parallel and Scalable Short-Read Alignment on Multi-Core Clusters Using UPC++

The growth of next-generation sequencing (NGS) datasets poses a challenge to the alignment of reads to reference genomes in terms of alignment quality and execution speed. Some available aligners have been shown to obtain high quality mappings at the expense of long execution times. Finding fast yet accurate software solutions is of high importance to research, since availability and size of NGS datasets continue to increase. In this work we present an efficient parallelization approach for NGS short-read alignment on multi-core clusters. Our approach takes advantage of a distributed shared memory programming model based on the new UPC++ language. Experimental results using the CUSHAW3 aligner show that our implementation based on dynamic scheduling obtains good scalability on multi-core clusters. Through our evaluation, we are able to complete the single-end and paired-end alignments of 246 million reads of length 150 base-pairs in 11.54 and 16.64 minutes, respectively, using 32 nodes with four AMD Opteron 6272 16-core CPUs per node. In contrast, the multi-threaded original tool needs 2.77 and 5.54 hours to perform the same alignments on the 64 cores of one node. The source code of our parallel implementation is publicly available at the CUSHAW3 homepage (http://cushaw3.sourceforge.net).

Portable Application-level Checkpointing for Hybrid MPI-OpenMP Applications

As parallel machines increase their number of processors, so does the failure rate of the global system, thus, long-running applications will need to make use of fault tolerance techniques to ensure the successful execution completion. Most of current HPC systems are built as clusters of multicores. The hybrid MPI-OpenMP paradigm provides numerous benefits on these systems. This paper presents a checkpointing solution for hybrid MPI-OpenMP applications, in which checkpoint consistency is guaranteed by using a coordination protocol intra-node, while no inter-node coordination is needed. The proposal reduces network utilization and storage resources in order to optimize the I/O cost of fault tolerance, while minimizing the checkpointing overhead. Besides, the portability of the solution and the dynamic parallelism provided by OpenMP enable the restart of the applications using machines with different architectures, operating systems and/or number of cores, adapting the number of running OpenMP threads for the best exploitation of the available resources. Extensive evaluation using hybrid MPI-OpenMP applications from the ASC Sequoia Benchmark Codes and NERSC-8/Trinity benchmarks is presented, showing the effectiveness and efficiency of the approach.

Assessing the cost of redistribution followed by a computational kernel: Complexity and performance results

Algorithms for finding the optimal distribution compatible with a given data partition.Analysis of the algorithms for different cost metrics.NP-completeness proof for the redistribution problem followed by a computational kernel.Experimental results for the 1D-stencil kernel and the QR factorization algorithm. The classical redistribution problem aims at optimally scheduling communications when reshuffling from an initial data distribution to a target data distribution. This target data distribution is usually chosen to optimize some objective for the algorithmic kernel under study (good computational balance or low communication volume or cost), and therefore to provide high efficiency for that kernel. However, the choice of a distribution minimizing the target objective is not unique. This leads to generalizing the redistribution problem as follows: find a re-mapping of data items onto processors such that the data redistribution cost is minimal, and the operation remains as efficient. This paper studies the complexity of this generalized problem. We compute optimal solutions and evaluate, through simulations, their gain over classical redistribution. We also show the NP-hardness of the problem to find the optimal data partition and processor permutation (defined by new subsets) that minimize the cost of redistribution followed by a simple computational kernel. Finally, experimental validation of the new redistribution algorithms are conducted on a multicore cluster, for both a 1D-stencil kernel and a more compute-intensive dense linear algebra routine.

Architecture Support for Tightly-Coupled Multi-Core Clusters with Shared-Memory HW Accelerators

Coupling processors with acceleration hardware is an effective manner to improve energy efficiency of embedded systems. Many-core is nowadays a dominating design paradigm for SoCs, which opens new challenges and opportunities for designing HW blocks. Exploring acceleration solutions that naturally fit into well-established parallel programming models and that can be incrementally added on top of existing parallel applications is thus extremely important. In this paper we focus on tightly-coupled multi-core cluster architectures, representative of the basic building block of the most recent many-cores, and we enhance it with dedicated HW processing units (HWPU). We propose an architecture where the HWPUs share the same L1 data memory through which processors also communicate, implementing a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> zero-copy</i> communication model. High-level synthesis (HLS) tools are used to generate HW blocks, then a custom wrapper interfaces the latter to the tightly coupled cluster. We validate our proposal on RTL models, running both synthetic workload and real applications. Experimental results demonstrate that on average our solution provides nearly identical performance to traditional private-memory coarse-grained accelerators, but it achieves up to 32 percent better performance/area/watt and it requires only minimal modifications to legacy parallel codes.

Design of a Multithreaded Barnes-Hut Algorithm for Multicore Clusters

We describe in this paper an implementation of the Barnes-Hut algorithm on multicore clusters. Based on a partitioned global address space (PGAS) library, the design integrates intranode multithreading and internode one-sided communication, exemplifying a PGAS $+$ X programming style. Within a node, the computation is decomposed into tasks (subtasks) and multitasking is used to hide network latency. We study the tradeoffs between locality in private caches and locality in shared caches and bring the insights into the design. As a result, our implementation consumes less memory per core, invokes less internode communication, and enjoys better load-balancing strategies. The final code achieves up to 41 percent performance improvement over a non-multithreaded counterpart. Through detailed comparison, we also show its advantages over other well-known Barnes-Hut implementations, both in programming complexity and in performance.

Computing the Smallest Eigenvalue of Large Ill-Conditioned Hankel Matrices

AbstractThis paper presents a parallel algorithm for finding the smallest eigenvalue of a family of Hankel matrices that are ill-conditioned. Such matrices arise in random matrix theory and require the use of extremely high precision arithmetic. Surprisingly, we find that a group of commonly-used approaches that are designed for high efficiency are actually less efficient than a direct approach for this class of matrices. We then develop a parallel implementation of the algorithm that takes into account the unusually high cost of individual arithmetic operations. Our approach combines message passing and shared memory, achieving near-perfect scalability and high tolerance for network latency. We are thus able to find solutions for much larger matrices than previously possible, with the potential for extending this work to systems with greater levels of parallelism. The contributions of this work are in three areas: determination that a direct algorithm based on the secant method is more effective when extreme fixed-point precision is required than are the algorithms more typically used in parallel floating-point computations; the particular mix of optimizations required for extreme precision large matrix operations on a modern multi-core cluster, and the numerical results themselves.

Automatic tuning of sparse matrix-vector multiplication on multicore clusters

To have good performance and scalability, parallel applications should be sophisticatedly optimized to exploit intra-node parallelism and reduce inter-node communication on multicore clusters. This paper investigates the automatic tuning of the sparse matrix-vector (SpMV) multiplication kernel implemented in a partitioned global address space language, which supports a hybrid thread- and process-based communication layer for multicore systems. One-sided communication is used for inter-node data exchange, while intra-node communication uses a mix of process shared memory and multithreading. We develop performance models to facilitate selecting the best configuration of threads and processes hybridization as well as the best communication pattern for SpMV. As a result, our tuned SpMV in the hybrid runtime environment consumes less memory and reduces inter-node communication volume, without damaging the data locality. Experiments are conducted on 12 real sparse matrices. On 16-node Xeon and 8-node Opteron clusters, our tuned SpMV kernel gets on average 1.4X and 1.5X improvement in performance over the well-optimized process-based message-passing implementation, respectively.

Regularization parameter selection in the 3D gravity inversion of the basement relief using GCV: A parallel approach

In this study, we present a computationally efficient automatic method for the optimal selection of the regularization parameter in the 3D inversion of gravity data. We use a linearized inversion method to estimate the depth of a 3D sedimentary basin in which the density contrast varies parabolically with the depth. The sedimentary basin is discretized into rectangular juxtaposed prisms with thicknesses that represent the depths from the surface to the interface and are the parameters to be estimated in the inversion process from the gravity anomaly.To deal with the singularity of the normal equation matrix in the linearized least squares method, standard-form Tikhonov regularization is incorporated in each step. The Generalized cross-validation (GCV) technique, which is capable of finding the optimal regularization parameter in an automatic manner, is adopted in this study. We present the simulation results with a synthetic model of a complex sedimentary basin, taking advantage of a parallel inversion algorithm implemented by adopting the message passing interface (MPI) standard on a multi-core cluster. This makes it possible to reduce the computer time by more than one order of magnitude. The applicability and efficacy of the GCV technique for the selection of the optimum regularization parameter are demonstrated.

Matrix multiplication with a hypercube algorithm on multi-core processor cluster

The algorithm of multiplication of matrices of Dekel, Nassimi and Sahani or Hypercube is analysed, modified and implemented on multicore processor cluster, where the number of processors used is less than that required by the algorithm n3&lt;sub&gt;3&lt;/sub&gt;. 2&lt;sub&gt;3&lt;/sub&gt;, 4&lt;sub&gt;3&lt;/sub&gt; and 8&lt;sub&gt;3&lt;/sub&gt; processing units are used to multiply matrices of the order of 10x10, 10&lt;sub&gt;2&lt;/sub&gt;x10&lt;sub&gt;2&lt;/sub&gt; and 10&lt;sub&gt;3&lt;/sub&gt;X10&lt;sub&gt;3&lt;/sub&gt;. The results of the mathematical model of the modified algorithm and those obtained from the computational experiments show that it is possible to reach acceptable speedup and parallel efficiencies, based on the number of used processor units. It also shows that the influence of the external communication link among the nodes is reduced if a combination of the available communication channels among the cores in a multi-core cluster is used.

Parallel Multiobjective Metaheuristics for Inferring Phylogenies on Multicore Clusters

The development of efficient parallel algorithms based on mixed mode programming represents one of the most popular lines of research in current bioinformatics. By exploiting hardware resources at inter-node/intra-node level, we can address grand computational challenges which involve the optimization of multiple objective functions simultaneously. In this sense, the inference of evolutionary trees represents one of the most difficult NP-hard problems in the field. Tackling such a problem requires efficient parallel designs to take advantage of the characteristics of modern multicore clusters. In this paper, we aim to solve the phylogenetic inference problem by applying MPI/OpenMP schemes to two multiobjective metaheuristics: fast non-dominated sorting genetic algorithm and multiobjective firefly algorithm. In order to assess the performance achieved by these proposals under different system and problem sizes, we have conducted experiments on six real nucleotide data sets according to a statistical methodology. Our parallel and multiobjective metrics point out the relevance of combining hybrid programming and novel bioinspired designs with regard to other parallel and biological approaches from the literature.

Improving the parallel efficiency of large-scale structural dynamic analysis using a hierarchical approach

In order to improve the parallel efficiency of large-scale structural dynamic analysis, a hierarchical approach adapted to the hardware topology of multi-core clusters is proposed. The hierarchical approach is constructed based on the strategies of two-level partitioning and two-level condensation. The data for parallel computing is first prepared through two-level partitioning to guarantee the load balancing within and across nodes. Then during the analysis of each time step, the convergence rate of interface problem is significantly improved by further reducing its size with two-level condensation. Furthermore, the communication overheads are considerably reduced by separating the intra-node and inter-node communications and minimizing the inter-node communication. Numerical experiments conducted on Dawning-5000A supercomputer indicate that the hierarchical approach was superior in performance compared with the conventional Newmark algorithm based on the domain decomposition method.

BSP-Why: A Tool for Deductive Verification of BSP Algorithms with Subgroup Synchronisation

We present bsp-why, a tool for deductive verification of bsp algorithms with subgroup synchronisation. From bsp programs, bsp-why generates sequential codes for the back-end condition generator why and thus benefits from its large range of existing provers. By enabling subgroups, the user can prove the correctness of programs that run on hierarchical machines--e.g. clusters of multi-cores. In general, bsp-why is able to generate proof obligations of mpi programs that only use collective operations. Our case studies are distributed state-space construction algorithms, the basis of model-checking.

THC-MP: High performance numerical simulation of reactive transport and multiphase flow in porous media

The numerical simulation of multiphase flow and reactive transport in the porous media on complex subsurface problem is a computationally intensive application. To meet the increasingly computational requirements, this paper presents a parallel computing method and architecture. Derived from TOUGHREACT that is a well-established code for simulating subsurface multi-phase flow and reactive transport problems, we developed a high performance computing THC-MP based on massive parallel computer, which extends greatly on the computational capability for the original code. The domain decomposition method was applied to the coupled numerical computing procedure in the THC-MP. We designed the distributed data structure, implemented the data initialization and exchange between the computing nodes and the core solving module using the hybrid parallel iterative and direct solver. Numerical accuracy of the THC-MP was verified through a CO2 injection-induced reactive transport problem by comparing the results obtained from the parallel computing and sequential computing (original code). Execution efficiency and code scalability were examined through field scale carbon sequestration applications on the multicore cluster. The results demonstrate successfully the enhanced performance using the THC-MP on parallel computing facilities.

Efficient and Scalable Algorithms for Smoothed Particle Hydrodynamics on Hybrid Shared/Distributed-Memory Architectures

This paper describes a new fast and implicitly parallel approach to neighbor-finding in multiresolution smoothed particle hydrodynamics (SPH) simulations. This new approach is based on hierarchical cell decompositions and sorted interactions, within a task-based formulation. It is shown to be faster than traditional tree-based codes and to scale better than domain decomposition--based approaches on hybrid shared/distributed-memory parallel architectures, e.g., clusters of multicores, achieving a $40\times$ speedup over the Gadget-2 simulation code.

Parallel Numerical Simulations of Three-Dimensional Electromagnetic Radiation with MPI-CUDA Paradigms

Using parallel computation can enhance the performance of numerical simulation of electromagnetic radiation and get great runtime reduction. We simulate the electromagnetic radiation calculation based on the multicore CPU and GPU Parallel Architecture Clusters by using MPI-OpenMP and MPI-CUDA hybrid parallel algorithm. This is an effective solution comparing to the traditional finite-difference time-domain method which has a shortage in the calculation of the electromagnetic radiation on the problem of inadequate large data space and time. What is more, we use regional segmentation, subregional data communications, consolidation, and other methods to improve procedures nested parallelism and finally verify the correctness of the calculation results. Studying these two hybrid models of parallel algorithms run on the high-performance cluster computer, we draw the conclusion that both models are suitable for large-scale numerical calculations, and MPI-CUDA hybrid model can achieve higher speedup.

An adaptive and hierarchical task scheduling scheme for multi-core clusters

• An adaptive and hierarchical task scheduling scheme (AHS) is proposed. • Work-sharing is used in conjunction with work-stealing. • An initial partitioning is performed with respect to the pattern of task parallelism . • A practical implementation of AHS is described. • The theoretical, simulation and experimental studies of AHS are presented. Work-stealing and work-sharing are two basic paradigms for dynamic task scheduling. This paper introduces an adaptive and hierarchical task scheduling scheme (AHS) for multi-core clusters, in which work-stealing and work-sharing are adaptively used to achieve load balancing. Work-stealing has been widely used in task-based parallel programing languages and models, especially on shared memory systems. However, high inter-node communication costs hinder work-stealing from being directly performed on distributed memory systems. AHS addresses this issue with the following techniques: (1) initial partitioning, which reduces the inter-node task migrations; (2) hierarchical scheduling scheme, which performs work-stealing inside a node before going across the node boundary and adopts work-sharing to overlap computation and communication at the inter-node level; and (3) hierarchical and centralized control for inter-node task migration, which improves the efficiency of victim selection and termination detection. We evaluated AHS and existing work-stealing schemes on a 16-nodes multi-core cluster. Experimental results show that AHS outperforms existing schemes by 11–21.4%, for the benchmarks studied in this paper.

Scaling up MapReduce-based Big Data Processing on Multi-GPU systems

MapReduce is a popular data-parallel processing model encompassed with recent advances in computing technology and has been widely exploited for large-scale data analysis. The high demand on MapReduce has stimulated the investigation of MapReduce implementations with different architectural models and computing paradigms, such as multi-core clusters, Clouds, Cubieboards and GPUs. Particularly, current GPU-based MapReduce approaches mainly focus on single-GPU algorithms and cannot handle large data sets, due to the limited GPU memory capacity. Based on the previous multi-GPU MapReduce version MGMR, this paper proposes an upgrade version MGMR++ to eliminate GPU memory limitation and a pipelined version, PMGMR, to handle the Big Data challenge through both CPU memory and hard disks. MGMR++ is extended from MGMR with flexible C++ templates and CPU memory utilization, while PMGMR fine-tuned the performance through the latest GPU features such as streams and Hyper-Q as well as hard disk utilization. Compared to MGMR (Jiang et al., Cluster Computing 2013), the proposed schemes achieve about 2.5-fold performance improvement, increase system scalability, and allow programmers to write straightforward MapReduce code for Big Data.

Alya: Computational Solid Mechanics for Supercomputers

While solid mechanics codes are now conventional tools both in industry and research, the increasingly more exigent requirements of both sectors are fuelling the need for more computational power and more advanced algorithms. For obvious reasons, commercial codes are lagging behind academic codes often dedicated either to the implementation of one new technique, or the upscaling of current conventional codes to tackle massively large scale computational problems. Only in a few cases, both approaches have been followed simultaneously. In this article, a solid mechanics simulation strategy for parallel supercomputers based on a hybrid approach is presented. Hybrid parallelization exploits the thread-level parallelism of multicore architectures, combining MPI tasks with OpenMP threads. This paper describes the proposed strategy, programmed in Alya, a parallel multi-physics code. Hybrid parallelization is specially well suited for the current trend of supercomputers, namely large clusters of multicores. The strategy is assessed through transient non-linear solid mechanics problems, both for explicit and implicit schemes, running on thousands of cores. In order to demonstrate the flexibility of the proposed strategy under advance algorithmic evolution of computational mechanics, a non-local parallel overset meshes method (Chimera-like) is implemented and the conservation of the scalability is demonstrated.