High Performance Conjugate Gradient Research Articles

Development of benchmark automation suite and evaluation of various high-performance computing systems

This study aimed to develop a dynamic benchmark automation suite to measure a range of benchmark performances and evaluate the various high-performance computing (HPC) systems. Our suite supports an automated scaling test and profiling data based on hardware performance counters to analyze the system characteristics. We selected four HPC benchmarks—STREAM, High-Performance Linpack, High-Performance Conjugate Gradient, and NAS Parallel Benchmark-for experiments and configured testbeds based on five different systems and an Intel Knights Landing (KNL) cluster with 16 nodes. The Intel KNL system showed both unstable memory and high benchmark performances for a specific input range. Modern Intel systems also exhibited proper characteristics on compute-intensive workloads, whereas the up-to-date AMD system showed high efficiency and proper characteristics on memory-intensive and real application workloads. We also verified that each system has an optimal environment and characteristic for various combinations of experimental variables and profiling data.

Understanding Communication Patterns in HPCG

Conjugate Gradient (CG) algorithms form a large part of many HPC applications, examples include bioinformatics and weather applications. These algorithms allow numerical solutions to complex linear systems. Understanding how distributed implementations of these algorithms use a network interconnect will allow system designers to gain a deeper insight into their exacting requirements for existing and future applications.This short paper documents our initial investigation into the communication patterns present in the High Performance Conjugate Gradient (HPCG) benchmark. Through our analysis, we identify patterns and features which may warrant further investigation to improve the performance of CG algorithms and applications which make extensive use of them. In this paper, we capture communication traces from runs of the HPCG benchmark at a variety of different processor counts and then examine this data to identify potential performance bottlenecks. Initial results show that there is a fall in the throughput of the network when more processes are communicating with each other, due to network contention.

HPCG and HPGMG benchmark tests on multiple program, multiple data (MPMD) mode on Blue Waters—A Cray XE6/XK7 hybrid system

SummaryThe high‐performance conjugate gradients (HPCG) and high‐performance geometric multi‐grid (HPGMG) benchmarks are alternatives to the traditional LINPACK benchmark (HPL) in measuring the performance of modern HPC platforms. We performed HPCG and HPGMG benchmark tests on a Cray XE6/XK7 hybrid supercomputer, Blue Waters at National Center for Supercomputing Applications (NCSA). The benchmarks were tested on CPU‐based and GPU‐enabled nodes separately, and then we analyzed characteristic parameters that affect their performance. Based on our analyses, we performed HPCG and HPGMG runs in multiple program, multiple data (MPMD) mode in Cray Linux Environment in order to measure their hybrid performance on both CPU‐based and GPU‐enabled nodes. We observed and analyzed several performance issues during those tests. Based on lessons learned from this study, we provide recommendations about how to optimize science applications on modern hybrid HPC platforms.

Core Module Optimizing PDE Sparse Matrix Models With HPCG Example

This paper introduces a fundamentally new computer architecture for supercomputers. The core module is application compatible with an existing superscalar microprocessor, with minimized energy use, and is optimized for local sparse matrix operations. Optimized sparse matrix manip- ulation is discussed by analyzing the High Performance Conjugate Gradient (HPCG) benchmark speci...cation. This analysis shows how the DRAM memory wall is removed for this benchmark, and for sparse matrix models of partial di¤erential equations (PDEs) for a wide cross section of applications. By giving the programmer improved control over the con...guration of the super- computer, the potential for communication problems is minimized. Application compatibility is achieved while removing the superscalar instruction interpreter and multi-thread controller from the existing microprocessor’s hardware. These are transformed into compile-time utilities. The instruction cache is removed through an innovation in VLIW instruction processing. The data caches are unnecessary and are turned o¤ in order to optimally implement sparse matrix models.

Main Memory in HPC

An important aspect of High-Performance Computing (HPC) system design is the choice of main memory capacity. This choice becomes increasingly important now that 3D-stacked memories are entering the market. Compared with conventional Dual In-line Memory Modules (DIMMs), 3D memory chiplets provide better performance and energy efficiency but lower memory capacities. Therefore, the adoption of 3D-stacked memories in the HPC domain depends on whether we can find use cases that require much less memory than is available now. This study analyzes the memory capacity requirements of important HPC benchmarks and applications. We find that the High-Performance Conjugate Gradients (HPCG) benchmark could be an important success story for 3D-stacked memories in HPC, but High-Performance Linpack (HPL) is likely to be constrained by 3D memory capacity. The study also emphasizes that the analysis of memory footprints of production HPC applications is complex and that it requires an understanding of application scalability and target category, i.e., whether the users target capability or capacity computing. The results show that most of the HPC applications under study have per-core memory footprints in the range of hundreds of megabytes, but we also detect applications and use cases that require gigabytes per core. Overall, the study identifies the HPC applications and use cases with memory footprints that could be provided by 3D-stacked memory chiplets, making a first step toward adoption of this novel technology in the HPC domain.

Hazel Hen – leading HPC technology and its impact on science in Germany and Europe

The impact of Europe's fastest supercomputer on research and industry.Progress in CFD and aeroacoustics by high-end computing.Progress in climate research by high-end computing.Deployment and operation of a European high-end computer system. The new HLRS flagship system Hazel Hen (a Cray XC40) is Europe's fastest system in the High Performance Conjugate Gradients (HPCG) benchmarking list. Designed to provide maximum sustainable performance to the users of HLRS - ignoring both peak and Linpack performance numbers - it has even climbed to number 8 in the TOP500 list and is one of the fastest systems in the world used both by research and industry. This paper will present insight into the strategy that was at the heart of the HLRS decision. It will further show how Hazel Hen can and has already impacted research and industry by providing high sustained performance for real world applications. In this paper we will show two applications and discuss the industrial uptake of the system.

Exploring performance and energy tradeoffs for irregular applications: A case study on the Tilera many-core architecture

High performance, parallel applications with irregular data accesses are becoming a critical workload class for modern systems. In particular, the execution of such workloads on emerging many-core systems is expected to be a significant component of applications in data mining, machine learning, scientific computing and graph analytics. However, power and energy constraints limit the capabilities of individual cores, memory hierarchy and on-chip interconnect of such systems, thus leading to architectural and software trade-offs that must be understood in the context of the intended application’s behavior. Irregular applications are notoriously hard to optimize given their data-dependent access patterns, lack of structured locality and complex data structures and code patterns.We have ported two irregular applications, graph community detection using the Louvain method (Grappolo) and high-performance conjugate gradient (HPCCG), to the Tilera many-core system and have conducted a detailed study of platform-independent and platform-specific optimizations that improve their performance as well as reduce their overall energy consumption. To conduct this study, we employ an auto-tuning based approach that explores the optimization design space along three dimensions—memory layout schemes, GCC compiler flag choices and OpenMP loop scheduling options. We leverage MIT’s OpenTuner auto-tuning framework to explore and recommend energy optimal choices for different combinations of parameters. We then conduct an in-depth architectural characterization to understand the memory behavior of the selected workloads. Finally, we perform a correlation study to demonstrate the interplay between the hardware behavior and application characteristics.Using auto-tuning, we demonstrate whole-node energy savings and performance improvements of up to 49.6% and 60% relative to a baseline instantiation, and up to 31% and 45.4% relative to manually optimized variants.

623 Tflop/s HPCG run on Tianhe-2: Leveraging millions of hybrid cores

In this article, we present a new hybrid algorithm to enable and scale the high-performance conjugate gradients (HPCG) benchmark on large-scale heterogeneous systems such as the Tianhe-2. Based on an inner–outer subdomain partitioning strategy, the data distribution between host and device can be balanced adaptively. The overhead of data movement from both the MPI communication and the PCI-E transfer can be significantly reduced by carefully rearranging and fusing operations. A variety of parallelization and optimization techniques for performance-critical kernels are exploited and analyzed to maximize the performance gain on both host and device. We carry out experiments on both a small heterogeneous computer and the world’s largest one, the Tianhe-2. On the small system, a thorough comparison and analysis has been presented to select from different optimization choices. On Tianhe-2, the optimized implementation scales to the full-system level of 3.12 million heterogeneous cores, with an aggregated performance of 623 Tflop/s and a parallel efficiency of 81.2%.

High-performance conjugate gradient performance improvement on the K computer

The high-performance conjugate gradient (HPCG) is new benchmark software for supercomputers that provides a more realistic performance metric than existing benchmarks, such as the LINPACK benchmark. The HPCG measures the speed of solving symmetric sparse linear system equations using the conjugate gradient method preconditioned by a multigrid symmetric Gauss–Seidel smoother. The combination of a sparse linear system and a preconditioned conjugate gradient method is widely used in many scientific and engineering computer applications. This study introduces a tuning method for the K computer. According to weak-scaling measurements on the K computer, it has good parallel scalability. Therefore, our tuning strategy focuses on single CPU performance rather than parallel performance. Single CPU performance strongly depends on memory throughput and multicore utilization. Therefore, we attempt to improve memory/cache access performance and multithreading efficiency. As a result, a HPCG score obtained with the K computer achieved second place at SC’14.

Characterizing the efficiency of multicore and manycore processors for the solution of sparse linear systems

We analyze the efficiency of servers equipped with state-of-the-art general-purpose multicore processors as well as platforms based on accelerators such as graphics processing units (GPUs) and the Intel Xeon Phi. Following the proposal recently advocated in the High Performance Conjugate Gradient (HPCG) benchmark, we leverage for this purpose efficient implementations of ILUPACK, a preconditioned solver for sparse linear systems that comprises numerical kernels and data access patterns analogous to those of HPCG. Our study analyzes the (computational) performance and energy efficiency, with two different metrics for each: time/floating-point throughput for the former; and energy/floating-point throughput-per-Watt for the latter.

High-performance conjugate-gradient benchmark: A new metric for ranking high-performance computing systems

We describe a new high-performance conjugate-gradient (HPCG) benchmark. HPCG is composed of computations and data-access patterns commonly found in scientific applications. HPCG strives for a better correlation to existing codes from the computational science domain and to be representative of their performance. HPCG is meant to help drive the computer system design and implementation in directions that will better impact future performance improvement.

Performance analysis of the high-performance conjugate gradient benchmark on GPUs

Graphics processing unit accelerated supercomputers have proved to be very effective, especially with regard to power efficiency, for accelerating compute intensive applications like the high-performance Linpack used in the TOP500 list. This paper presents the details of a CUDA implementation of the high-performance conjugate gradient, a new proposed benchmark that better represents modern application workloads which rely more heavily on memory system and network performance than high-performance Linpack. The results obtained at full scale on the largest graphics processing unit supercomputers in the world, Titan, the Cray XK7 at ORNL and Piz-Daint, the Cray XC30 at CSCS, indicate that graphics processing unit accelerated supercomputers are also very effective for this type of workload. A comparison with other architectures is also presented, showing that graphics processing units, with their high memory bandwidth, are the highest performing devices for this new benchmark.

Optimizations in a high-performance conjugate gradient benchmark for IA-based multi- and many-core processors

This paper presents optimizations in a high-performance conjugate gradient benchmark (HPCG) for multi-core Intel® Xeon® processors and many-core Xeon Phi™ coprocessors. Without careful optimization, the HPCG benchmark under-utilizes the compute resources available in modern processors due to its low arithmetic intensity and challenges in parallelizing the Gauss–Seidel smoother (GS). Our optimized implementation fuses GS with sparse matrix vector multiplication (SpMV) to address the low arithmetic intensity, overcoming the performance otherwise bound by memory bandwidth. This fusion optimization is progressively more effective in newer generation Xeon processors, demonstrating the usefulness of their larger caches for sparse matrix operations: Sandy Bridge, Ivy Bridge, and Haswell processors achieve 93%, 99%, and 103%, respectively, of the ideal performance with a constraint that matrices are streamed from memory. Our implementation also parallelizes GS using fine-grain level-scheduling, a method that has been believed not to scale with many cores. Our GS implementation scales with 60 cores in Xeon Phi coprocessors, for the finest level of the multi-grid pre-conditioner. At the coarser levels, we address the limited parallelism using block multi-color re-ordering, achieving 21 GFLOPS with one Xeon Phi coprocessor. These optimizations distinguish our HPCG implementation from the others that stream most of the data from main memory and rely on multi-color re-ordering for parallelism. Our optimized implementation has been evaluated in clusters with various configurations, and we find that low-diameter high-radix network topologies such as Dragonfly realize high parallelization efficiencies because of fast all-reduce collectives. In addition, we demonstrate that our optimizations not only benefit the HPCG dataset, which is based on a structured 3D grid, but also a wide range of unstructured matrices.

High performance conjugate gradient solver on multi-GPU clusters using hypergraph partitioning

Motivated by high computation power and low price per performance ratio of GPUs, GPU accelerated clusters are being built for high performance scientific computing. In this work, we propose a scalable implementation of a Conjugate Gradient (CG) solver for unstructured matrices on a GPU-extended cluster, where each cluster node has multiple GPUs. Basic computations of the solver are held on GPUs and communications are managed by the CPU. For sparse matrix-vector multiplication, which is the most time-consuming operation, solver selects the fastest between several high performance kernels running on GPUs. In a GPU-extended cluster, it is more difficult than traditional CPU clusters to obtain scalability, since GPUs are very fast compared to CPUs. Since computation on GPUs is faster, GPU-extended clusters demand faster communication between compute units. To achieve scalability, we adopt hypergraph-partitioning models, which are state-of-the-art models for communication reduction and load balancing for parallel sparse iterative solvers. We implement a hierarchical partitioning model which better optimizes underlying heterogeneous system. In our experiments, we obtain up to 94 Gflops double-precision CG performance using 64 NVIDIA Tesla GPUs on 32 nodes.