MPI Collective Communications Research Articles

Optimal low-latency network topologies for cluster performance enhancement

We propose that clusters interconnected with network topologies having minimal mean path length will increase their overall performance for a variety of applications. We approach our heuristic by constructing clusters of up to 36 nodes having Dragonfly, torus, ring, Chvatal, Wagner, Bidiakis and several other topologies with minimal mean path lengths and by simulating the performance of 256-node clusters with the same network topologies. The optimal (or sub-optimal) low-latency network topologies are found by minimizing the mean path length of regular graphs. The selected topologies are benchmarked using ping-pong messaging, the MPI collective communications, and the standard parallel applications including effective bandwidth, FFTE, Graph 500 and NAS parallel benchmarks. We established strong correlations between the clusters' performances and the network topologies, especially the mean path lengths, for a wide range of applications. In communication-intensive benchmarks, clusters with optimal network topologies out-perform those with mainstream topologies by several folds. It is striking that a mere adjustment of the network topology suffices to reclaim performance from the same computing hardware.

Static/Dynamic validation of MPI collective communications in multi-threaded context

Scientific applications mainly rely on the MPI parallel programming model to reach high performance on supercomputers. The advent of manycore architectures (larger number of cores and lower amount of memory per core) leads to mix MPI with a thread-based model like OpenMP. But integrating two different programming models inside the same application can be tricky and generate complex bugs. Thus, the correctness of hybrid programs requires a special care regarding MPI calls location. For example, identical MPI collective operations cannot be performed by multiple non-synchronized threads. To tackle this issue, this paper proposes a static analysis and a reduced dynamic instrumentation to detect bugs related to misuse of MPI collective operations inside or outside threaded regions. This work extends PARCOACH designed for MPI-only applications and keeps the compatibility with these algorithms. We validated our method on multiple hybrid benchmarks and applications with a low overhead.

Recent advances in the Message Passing Interface

ion for MPI programs with pointer-based data structures. The main features of DRASync are: it amortizes communication among MPI processes to allow efficient parallel allocation in a global address space; it takes advantage of bulk deallocation and good locality with pointer-based data structures. Finally, DRASync supports ownership semantics of regions by MPI processes akin to reader–writer locks, which makes for a high-level, intuitive synchronization tool in MPI programs, without sacrificing message-passing performance. In ‘‘An Evaluation of MPI Message Rate on Hybrid-Core Processors’’, Barrett et al. analyze the ability of simple and more complex cores to perform MPI matching operations for various scenarios in order to gain insight into how MPI implementations for future hybrid-core processors should be designed. The authors compare throughput-oriented cores to University Carlos III of Madrid, Spain single-thread optimized cores in terms of the ability to perform MPI match processing. The intent of this study is to gain insight into the ability of throughputoriented cores to adequately perform MPI matching and to better understand how MPI implementations on future hybrid-core processors should allocate computing resources to try to optimize performance-critical MPI operations. Collective MPI communications have to be executed in the same order by all processes in their communicator and the same number of times, otherwise it is not conforming to the standard and a deadlock can occur. As soon as the control-flow involving these collective operations becomes more complex, in particular including conditionals on process ranks, ensuring the correction of such code is error-prone. The paper ‘‘PARCOACH: Combining Static and Dynamic Validation of MPI Collective Communications’’ by Saillard et al. proposes a static analysis to detect when such situation occurs, combined with a code transformation that prevents deadlocking. They show with several benchmarks the small impact on performance and the ease of integration of their techniques in the development process. In ‘‘Extreme-scale Computing Services Over MPI: Experiences, Observations and Features Proposal for Next Generation Message Passing Interface’’ by Zounmevo et al., the authors present their experiences in using MPI as a network transport for a large-scale distributed storage system. The authors discuss the features of MPI that facilitate adoption as well as aspects which require various workarounds. Based on use cases, the authors derive a wish-list for both MPI implementations and the MPI forum to facilitate the adoption of MPI large-scale persistent services. Finally in the paper ‘‘Optimization of MPI collective operations on the IBM Blue Gene/Q Supercomputer’’, Kumar et al. present scalable algorithms to optimize MPI collective operations by taking advantage of the various features of the Blue Gene/Q torus and collective networks. The authors accelerate summing of network packets with local buffers by the use of the Quad Processing SIMD unit in the Blue Gene/Q cores and executing the sums on multiple communication threads supported by the optimized communication libraries.

PARCOACH: Combining static and dynamic validation of MPI collective communications

Nowadays most scientific applications are parallelized based on MPI communications. Collective MPI communications have to be executed in the same order by all processes in their communicator and the same number of times, otherwise they do not conform to the standard and a deadlock or other undefined behaviour can occur. As soon as the control flow involving these collective operations becomes more complex, in particular including conditionals on process ranks, ensuring the correction of such code is error-prone. We propose in this paper a static analysis to detect when such a situation occurs, combined with a code transformation that prevents deadlocking. We focus on blocking MPI collective operations in single program multiple data applications, assuming MPI calls are not nested in multithreaded regions. We show on several benchmarks the small impact on performance and the ease of integration of our techniques in the development process.

Kernel-assisted and topology-aware MPI collective communications on multicore/many-core platforms

Multicore Clusters, which have become the most prominent form of High Performance Computing (HPC) systems, challenge the performance of MPI applications with non-uniform memory accesses and shared cache hierarchies. Recent advances in MPI collective communications have alleviated the performance issue exposed by deep memory hierarchies by carefully considering the mapping between the collective topology and the hardware topologies, as well as the use of single-copy kernel assisted mechanisms. However, on distributed environments, a single level approach cannot encompass the extreme variations not only in bandwidth and latency capabilities, but also in the capability to support duplex communications or operate multiple concurrent copies. This calls for a collaborative approach between multiple layers of collective algorithms, dedicated to extracting the maximum degree of parallelism from the collective algorithm by consolidating the intra- and inter-node communications.In this work, we present HierKNEM, a kernel-assisted topology-aware collective framework, and the mechanisms deployed by this framework to orchestrate the collaboration between multiple layers of collective algorithms. The resulting scheme maximizes the overlap of intra- and inter-node communications. We demonstrate experimentally, by considering three of the most used collective operations (Broadcast, Allgather and Reduction), that (1) this approach is immune to modifications of the underlying process-core binding; (2) it outperforms state-of-art MPI libraries (OpenMPI, MPICH2 and MVAPICH2) demonstrating up to a 30x speedup for synthetic benchmarks, and up to a 3x acceleration for a parallel graph application (ASP); (3) it furthermore demonstrates a linear speedup with the increase of the number of cores per compute node, a paramount requirement for scalability on future many-core hardware.

Algorithm optimization of MPI collective communications in KD60

Large clusters have been developed to multi-core era,and multi-core architecture makes new demands on parallel computation.Message Passing Interface(MPI) is the most commonly used parallel programming model,and collective communications is an important part of the MPI standard.Efficient collective communications algorithm plays a vital role in improving the performance of parallel computation.This paper first analyzed the architecture features of KD60 and communication hierarchy characteristics under multi-core architecture,and then introduced the implementation of collective communications algorithm in MPICH2 and pointed out its' deficiencies.At last,this article took broadcasting as an example,using an improved algorithm based on CMP architecture,which changes the communication mode of the original algorithm.At the same time,this paper optimized the algorithm according to the architecture characteristics of KD60.The experimental results show that the improved algorithm improves the performance of broadcast in MPI.

Performance Models for MPI Collective Communications with Network Contention

The paper presents a novel approach to estimate the performance of MPI collective communications. Our objective is to help researchers to make appropriate decisions on their message-passing applications. For each collective communication, we attempt to apply LogGP and P-LogP standard point-to-point models. The resulted models are compared with the empirical data in order to identify the most suitable for performance characterization of collective operations. For the communications on large clusters with large size messages, the network contention problem can significantly affect the performance. Hence, to reduce the relative gap between the prediction and the measured runtime, the contention issue is also modeled, by a queuing theory analysis method, and taken in account with the total performance estimation. The experiments performed on a cluster which consists of 64 processors interconnected by Gigabit Ethernet network show encouraging results. For any collective operation, given a number of processors and a range of message sizes, there is at least one model that predicts the performance precisely. We could achieve a gap between the predicted and the measured run-time around 15%. Thus, by handling the contention problem, we could reduce around 80% of the relative gap.

A framework for adaptive collective communications for heterogeneous hierarchical computing systems

Collective communication operations are widely used in MPI applications and play an important role in their performance. However, the network heterogeneity inherent to grid environments represent a great challenge to develop efficient high performance computing applications. In this work we propose a generic framework based on communication models and adaptive techniques for dealing with collective communication patterns on grid platforms. Toward this goal, we address the hierarchical organization of the grid, selecting the most efficient communication algorithms at each network level. Our framework is also adaptive to grid load dynamics since it considers transient network characteristics for dividing the nodes into clusters. Our experiments with the broadcast operation on a real-grid setup indicate that an adaptive framework allows significant performance improvements on MPI collective communications.

Towards an Accurate Model for Collective Communications

The performance of the MPI's collective communications is critical in most MPI-based applications. A general algorithm for a given collective communication operation may not give good performance on all systems due to the differences in architectures, network parameters and the storage capacity of the underlying MPI implementation. Hence, collective communications have to be tuned for the system on which they will be executed. In order to determine the optimum parameters of collective communications on a given system in a time-efficient manner, the collective communications need to be modeled efficiently. In this paper, we discuss various techniques for modeling collective communications.