Xeon Phi Research Articles

HeteroPP: A directive‐based heterogeneous cooperative parallel programming framework

AbstractHeterogeneous platforms composed of multiple different types of computing devices (such as CPUs, GPUs, and Intel MICs) have been widely used recently. However, most of parallel applications developed in such a heterogeneous platform usually only utilize a certain kind of computing device due to the lack of easy‐to‐use heterogeneous cooperative parallel programming models. To reduce the difficulty of heterogeneous cooperative parallel programming, a directive‐based heterogeneous cooperative parallel programming framework called HeteroPP is proposed. HeteroPP provides an easier way for programmers to fully exploit multiple different types of computing devices to concurrently and cooperatively perform data‐parallel applications on heterogeneous platforms. An extension to OpenMP directives and clauses is proposed to make it possible for programmers to easily offload a data‐parallel compute kernel to multiple different types of computing devices. A source‐to‐source compiler is designed to help programmers to automatically generate multiple device‐specific compute kernels that can be concurrently and cooperatively performed on heterogeneous platforms. Many experiments are conducted with 12 typical data‐parallel applications implemented with HeteroPP on a heterogeneous CPU‐GPU‐MIC platform. The results show that HeteroPP not only greatly simplifies the heterogeneous cooperative parallel programming, but also can fully utilize the CPUs, GPU, and MIC to efficiently perform these applications.

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

Implicit propagation of directly addressed grids in lattice Boltzmann methods

SummaryLattice Boltzmann methods (LBM) are well suited to highly parallel computational fluid dynamics simulations due to their separability into a perfectly parallel collision step and a propagation step that only communicates within a local neighborhood. The implementation of the propagation step provides constraints for the maximum possible bandwidth‐limited performance, memory layout and usage of vector instructions. This article revisits and extends the work on implicit propagation on directly addressed grids started by A‐A and its shift‐swap‐streaming (SSS) formulation by reconsidering them as transformations of the underlying space filling curve. In this work, a new periodic shift (PS) pattern is proposed that imposes minimal restrictions on the implementation of collision operators and utilizes virtual memory mapping to provide consistent performance across a range of targets. Various implementation approaches as well as time dependency and performance anisotropy are discussed. Benchmark results for SSS and PS on SIMD CPUs including Intel Xeon Phi as well as Nvidia GPUs are provided. Finally, the application of PS as the propagation pattern of the open source LBM framework OpenLB is summarized.

A Numerical Code for a Wide Range of Compressible Flows on Hybrid Computational Architectures

The major points in the development of the parallel multiplatform multipurpose numerical code solving the full unsteady Navier–Stokes equations are presented. The developed code is primarily designed for running on multi-GPU computational devices but can also be used on traditional multicore CPUs and even on manycore processors such as Intel Xeon Phi. Physical models include calorically perfect inert gas, single- and multi-temperature approaches for chemically reactive flows and an Euler–Euler model for gas-particle suspensions. Main details of the implementation are described. Shock capturing TVD and WENO schemes in general curvilinear coordinates are used for spatial approximation. Explicit, semi-implicit and fully implicit schemes are employed for advancing solution in time. The code is written in C++ with CUDA API and opensource libraries, such as MPI, zlib and VTK. A few examples of numerical simulations are briefly described to provide general idea of the numerical code capabilities. They include a supersonic flow past a wedge, a jet exhausting from a square nozzle, a heavy gas bubble descending in a lighter medium and a heterogeneous detonation in gas-particle suspension.

Energy Efficiency of a New Parallel PIC Code for Numerical Simulation of Plasma Dynamics in Open Trap

The generation of energy-efficient parallel scientific codes became very important in the time of carbon footprint reduction. In this paper, we briefly present our latest particle-in-cell code with the results of a numerical simulation of plasma dynamics in an open trap. This code can be auto-vectorized by the Fortran compiler for Intel Xeon processors with AVX-512 instructions such as Intel Xeon Phi and the highest series of all generations of Intel Xeon Scalable processors. Efficient use of processor architecture is the main feature of an energy-efficient solution. We present a step-by-step methodology of energy consumption calculation using Intel hardware features and Intel VTune software. We also give an estimated value of carbon footprint with the impact of high-performance water cooled hardware. The Power Usage Effectiveness (PUE) in the case of high-performance water cooled hardware is equal to 1.03–1.05, and is up to 1.3 in the case of air-cooled systems. This means that power consumption of liquid cooled systems is lower than that air-cooled ones by up to 25%. All these factors play an important role in the carbon footprint reduction problem.

Efficient exact algorithms for continuous bi‐objective performance‐energy optimization of applications with linear energy and monotonically increasing performance profiles on heterogeneous high performance computing platforms

AbstractPerformance and energy are the two most important objectives for optimization on heterogeneous high performance computing platforms. This work studies a mathematical problem motivated by the bi‐objective optimization of data‐parallel applications on such platforms for performance and energy. First, we formulate the problem and present an exact algorithm of polynomial complexity solving the problem where all the application profiles of objective type one are continuous and strictly increasing, and all the application profiles of objective type two are linear increasing. We then apply the algorithm to develop solutions for two related optimization problems of parallel applications on heterogeneous hybrid platforms, one for performance and dynamic energy and the other for performance and total energy. Our proposed solution methods are then employed to solve the two bi‐objective optimization problems for two data‐parallel applications, matrix multiplication and gene sequencing, on a hybrid platform employing five heterogeneous processors, namely, two different Intel multicore CPUs, an Nvidia K40c GPU, an Nvidia P100 PCIe GPU, and an Intel Xeon Phi.

Resource-Efficient Use of Modern Processor Architectures For Numerically Solving Cardiac Ionic Cell Models.

A central component in simulating cardiac electrophysiology is the numerical solution of nonlinear ordinary differential equations, also called cardiac ionic cell models, that describe cross-cell-membrane ion transport. Biophysically detailed cell models often require a considerable amount of computation, including calls to special mathematical functions. This paper systematically studies how to efficiently use modern multicore CPUs for this costly computational task. We start by investigating the code restructurings needed to effectively enable compiler-supported SIMD vectorisation, which is the most important performance booster in this context. It is found that suitable OpenMP directives are sufficient for achieving both vectorisation and parallelisation. We then continue with an evaluation of the performance optimisation technique of using lookup tables. Due to increased challenges for automated vectorisation, the obtainable benefits of lookup tables are dependent on the hardware platforms chosen. Throughout the study, we report detailed time measurements obtained on Intel Xeon, Xeon Phi, AMD Epyc and two ARM processors including Fujitsu A64FX, while attention is also paid to the impact of SIMD vectorisation and lookup tables on the computational accuracy. As a realistic example, the benefits of performance enhancement are demonstrated by a 109-run ensemble on the Oakforest-PACS system, where code restructurings and SIMD vectorisation yield an 84% reduction in computing time, corresponding to 63,270 node hours.

A DSMC program that supports different heterogeneous systems to simulate near-continuum gases

It has become a trend for Direct Simulation Monte Carlo (DSMC) to simulate near-continuum gas, which makes the problem of a large amount of computation more prominent in turn. Moreover, heterogeneous architecture is becoming mainstream for modern supercomputers. Speeding DSMC on heterogeneous systems has been a must consideration for DSMC researchers. However, little research on the DSMC program that supports different heterogeneous systems is available. In this paper, we developed a DSMC program supporting heterogeneous systems through OpenMP4.5, and our codes can execute on many accelerators in theory such as Nvidia GPUs, AMD GPUs and Intel Many Integrated Core (MIC) products. The implementation of our heterogeneous DSMC is elaborated first, and then we explain the algorithm proposed to generate random numbers on accelerators. In addition, the optimization methods, including load-balance and granularity refinement strategy, branch elimination, and particles’ removal and entering, are described in detail, respectively. A shock-bubble interaction (SBI) case under Knudsen number of [Formula: see text] is utilized to validate the accuracy of our DSMC and do the performance analysis. An Nvidia RTX 2080 serves as a device, and the results show that our program can not only gain accurate flowfield information, but also perform well under different computing loads. More specifically, the whole speedup ratio of [Formula: see text] can be obtained relative to a single CPU core, and for the subroutine Collision, the ratio can even reach approximately 90.

Performance Evaluation of Massively Parallel Systems Using SPEC OMP Suite

Performance analysis plays an essential role in achieving a scalable performance of applications on massively parallel supercomputers equipped with thousands of processors. This paper is an empirical investigation to study, in depth, the performance of two of the most common High-Performance Computing architectures in the world. IBM has developed three generations of Blue Gene supercomputers—Blue Gene/L, P, and Q—that use, at a large scale, low-power processors to achieve high performance. Better CPU core efficiency has been empowered by a higher level of integration to gain more parallelism per processing element. On the other hand, the Intel Xeon Phi coprocessor armed with 61 on-chip x86 cores, provides high theoretical peak performance, as well as software development flexibility with existing high-level programming tools. We present an extensive evaluation study of the performance peaks and scalability of these two modern architectures using SPEC OMP benchmarks.

General purpose lattice QCD code set Bridge++ 2.0 for high performance computing

Bridge++ is a general-purpose code set for a numerical simulation of lattice QCD aiming at a readable, extensible, and portable code while keeping practically high performance. The previous version of Bridge++ is implemented in double precision with a fixed data layout. To exploit the high arithmetic capability of new processor architecture, we extend the Bridge++ code so that optimized code is available as a new branch, i.e., an alternative to the original code. This paper explains our strategy of implementation and displays application examples to the following architectures and systems: Intel AVX-512 on Xeon Phi Knights Landing, Arm A64FX-SVE on Fujitsu A64FX (Fugaku), NEC SX-Aurora TSUBASA, and GPU cluster with NVIDIA V100.

Heterogeneous CPU+GPU parallelization for high-accuracy scale-resolving simulations of compressible turbulent flows on hybrid supercomputers

A heterogeneous parallel algorithm for simulation of compressible turbulent flows and its portable software implementation are presented. The underlying numerical method is based on a family of higher accuracy edge-based reconstruction schemes on unstructured mixed-element meshes. The proposed parallel solution can engage a large number of computing devices of most of the existing computing architectures used in modern supercomputers, including manycore CPUs and GPUs. It is capable of co-execution on both CPUs and accelerators simultaneously. The multilevel parallel algorithm combines: MPI for distributing workload among hybrid cluster nodes and between devices inside nodes; OpenMP for manycore CPUs and other supporting devices, such as Intel Xeon Phi; OpenCL for massively-parallel accelerators, such as GPUs of various vendors, including NVIDIA, AMD, Intel. The main focus is on the adaptation of the numerical method and its computational algorithm to the stream processing parallel paradigm. The very limited device memory inherent in GPU computing is also taken into account. A detailed description of the parallel algorithm is presented, as well as the techniques used for its efficient parallel implementation. Special attention is paid to implicit time integration with its linear solver and calculation of convective fluxes and viscous terms. The use of mixed floating-point precision and overlapping communications and computations is also discussed. Parallel performance is demonstrated in practical applications on different kinds of supercomputers using up to 10 thousand cores and multiple GPUs of comparable overall performance.

JURECA: Data Centric and Booster Modules implementing the Modular Supercomputing Architecture at Jülich Supercomputing Centre

JURECA is a Pre-Exascale Modular Supercomputer operated by Jülich Supercomputing Centre at Forschungszentrum Jülich. The system combines a flexible Data Centric (DC) module, based on the Atos BullSequana XH2000 with a selection of best-of-its-kind components, and a scalability-focused Booster module, delivered by Intel and Dell Technologies based on the Xeon Phi many-core processor. With its novel architecture, it supports a wide variety of high-performance computing and data analytics workloads.

Electronic Structure Theory Calculations Using Modern Architectures: KNL vs Haswell

The time to solution and parallel efficiency of several commonly used electronic structure methods (Hartree-Fock, density functional theory, second order perturbation theory, resolution of the identity second order perturbation theory, coupled cluster) are evaluated on both the Intel Xeon Haswell and the Intel Xeon Phi Knights Landing (KNL) architectures. The Haswell completes the benchmark calculations with a faster time to solution than the KNL for all molecules and methods tested. While the Haswell exhibits an average speedup of at least 3.5 relative to the KNL for all nonthreaded computations, the KNL has a better parallel efficiency than the Haswell with increasing core counts. The architectures are further tested using a more computationally costly coupled cluster method on a transition state reaction. The Haswell appears to be the best choice to minimize the time to solution, though for very large systems and high levels of theory that require memory intensive processes the superior memory hierarchy and larger on node memory of the KNL can make it a better choice. These results are used to showcase aspects of novel architectures that will increase efficiency for quantum chemistry applications.

Optimization of heterogeneous systems with AI planning heuristics and machine learning: a performance and energy aware approach

Heterogeneous computing systems provide high performance and energy efficiency. However, to optimally utilize such systems, solutions that distribute the work across host CPUs and accelerating devices are needed. In this paper, we present a performance and energy aware approach that combines AI planning heuristics for parameter space exploration with a machine learning model for performance and energy evaluation to determine a near-optimal system configuration. For data-parallel applications our approach determines a near-optimal host-device distribution of work, number of processing units required and the corresponding scheduling strategy. We evaluate our approach for various heterogeneous systems accelerated with GPU or the Intel Xeon Phi. The experimental results demonstrate that our approach finds a near-optimal system configuration by evaluating only about 7% of reasonable configurations. Furthermore, the performance per Joule estimation of system configurations using our machine learning model is more than 1000 times faster compared to the system evaluation by program execution.

How to Transfer Strided Data into Accelerator

Data transfer from a host central processing unit (CPU) into an accelerator is a performance bottleneck for applications accelerated by accelerators (such as general purpose digital signal processing (GPDSP), many integrated core (MIC), and general purpose graphics processing unit (GPGPU)). It is complicated and inefficient to transfer non-contiguous data with special respect to strided data. In this work, we present three approaches to transfer strided data for different scenarios: Redundant copy (RC), selective copy (SC), and transfer after transformed (TaT). We propose a space and time efficient method named TaT, in which strided data are transformed on the CPU first and then transferred into the accelerator. We simulated regions-of-interest (ROI) coding and validate proposed techniques. TaT was superior to RC on space efficiency and close to SC on saving space, but better than SC on time waste respectively.

FPGA‐based HPC accelerators: An evaluation on performance and energy efficiency

AbstractHardware specialization is a promising direction for the future of digital computing. Reconfigurable technologies enable hardware specialization with modest non‐recurring engineering cost, but their performance and energy efficiency compared to state‐of‐the‐art processor architectures remain an open question. In this article, we use FPGAs to evaluate the benefits of building specialized hardware for numerical kernels found in scientific applications. In order to properly evaluate performance, we not only compare Intel Arria 10 and Xilinx U280 performance against Intel Xeon, Intel Xeon Phi, and NVIDIA V100 GPUs, but we also extend the Empirical Roofline Toolkit (ERT) to FPGAs in order to assess our results in terms of the Roofline model. We show design optimization and tuning techniques for peak FPGA performance at reasonable hardware usage and power consumption. As FPGA peak performance is known to be far less than that of a GPU, we also benchmark the energy efficiency of each platform for the scientific kernels comparing against microbenchmark and technological limits. Results show that while FPGAs struggle to compete in absolute terms with GPUs on memory‐ and compute‐intensive kernels, they require far less power and can deliver nearly the same energy efficiency.

Parallel Execution on HeterogeneousMultiprocessors from Algorithm Models Based on Petri Nets

Current supercomputers are composed by nodes containing a combination of general purpose computing units (CPUs) and specific mathematical coprocessors. In this way, GPGPUs or Xeon Phi cards are attached to the nodes to improve its performance. Both types of processors, CPUs and coprocessors, have many differences, like their architecture, the clock rate of the processors and the operation of the related memory. These are the main factors that conform an heterogeneous multiprocessor. A parallel program that wants to achieve the sum of the performance of both types of processors, must consider not only the complexity of the parallel algorithm, but also the differences in the architecture of processors, increasing its complexity. As a contribution on this problem, this paper presents a model of parallel execution based on Petri Nets, called PN-PEM, that is used not only to model a parallel algorithm, but also to execute it directly on a computer with heterogeneous multiprocessors. An asynchronous execution of tasks and a dynamic scheduler are the main characteristics that allow execute a parallel program on this type of parallel computer. Tests done on a multicore computer with two Xeon Phi cards reach the aggregate performance of both type of processors, confirming the quality of the model used.

Cost-efficient simulations of large-scale electronic structures in the standalone manycore architecture

The tight-binding (TB) model is complementary to the ab-initio methods that can be represented by the well-known Density Functional Theory since the empirical nature of a TB model enables the scope of electronic structure simulations to include realistically sized nanoscale structures that are normally made up of several million atoms. As the major computational hotspot of TB simulations comes from diagonalization of a sparse system matrix whose size is proportional to the number of atoms belonging to a simulation domain, efficient utilization of high performance computers is strongly encouraged. Here we study the feasibility of cost-efficient TB simulations for large-scale electronic structures on the Intel Xeon Phi Knights Landing (KNL) platform, which is equipped with the processor of many-integrated cores and the onboard high-speed memory. Using our in-house code whose scientific fidelity has been validated through various modeling researches, we present detailed discussion on how KNL systems can be exploited to accelerate simulations, and conduct rigorous benchmark tests in competing platforms to justify benefits of the standalone manycore architecture in terms of the time and the energy consumption that must be paid. The capability to handle extremely huge atomic structures in KNL systems is also demonstrated by securing a strong scalability up to 2,500 nodes in the NURION supercomputer for a model problem that has 400 million atoms.

Preliminary study on the automatic parallelism optimization model for image enhancement algorithms based on Intel's® Xeon Phi

AbstractIn unmanned aerial vehicle (UAV) image‐processing applications, one needs to implement different parallel image‐enhancement algorithms on several high‐performance computing platforms utilizing various programming models. To speed up the parallelization procedure and improve its efficiency, the automatic parallel software package, Par4All, is applied in this work. We find that the performance of the original automatic parallelization algorithm produced with Par4All is inefficient. To resolve this problem, we propose different optimization approaches for Par4All based on Intel®'s Xeon Phi high‐performance computing platform that are based on the structural features of the image‐enhancement algorithms, which can further optimize the original parallel algorithm. These approaches mainly include: (1) Par4All automatic parallel search module optimization, (2) dynamic thread setting optimization, and (3) the collaborative parallelization of both CPU and many integrated core (MIC) processors. According to the results of the comparison experiments involving different algorithms, it is shown that the proposed optimization approaches for these kinds of algorithms can significantly improve the performance of automatic parallel algorithms. The acceleration ratio increases approximately by 30%, 70%, and 80% for the multiscale Retinex, Gaussian‐filtering and median‐filtering algorithms, respectively. As continuation and deepening of our previous research work, this research has the potential to be beneficial for other researchers in image‐processing applications with image‐enhancement algorithms.

Bi-Objective Optimization of Data-Parallel Applications on Heterogeneous HPC Platforms for Performance and Energy Through Workload Distribution

Performance and energy are the two most important objectives for optimization on modern parallel platforms. In this article, we show that moving from single-objective optimization for performance or energy to their bi-objective optimization on heterogeneous processors results in a tremendous increase in the number of optimal solutions (workload distributions) even for the simple case of linear performance and energy profiles. We then study full performance and energy profiles of two real-life data-parallel applications and find that they exhibit shapes that are non-linear and complex enough to prevent good approximation of them as analytical functions for input to exact algorithms or optimization software for determining the Pareto front. We, therefore, propose a solution method solving the bi-objective optimization problem on heterogeneous processors. The method's novel component is an efficient and exact global optimization algorithm that takes as an input performance and energy profiles as arbitrary discrete functions of workload size, which accurately and realistically take into account resource contention and NUMA inherent in modern parallel platforms, and returns the Pareto-optimal solutions (generally speaking, load imbalanced). To construct the input discrete energy functions, the method employs a methodology that accurately models the energy consumption by a hybrid data-parallel application executing on a heterogeneous HPC platform containing different computing devices using system-level power measurements provided by power meters. We experimentally analyse the proposed solution method using three data-parallel applications, matrix multiplication, 2D fast Fourier transform (2D-FFT), and gene sequencing, on two connected heterogeneous servers consisting of multicore CPUs, GPUs, and Intel Xeon Phi. We show that it determines a superior Pareto front containing the best load balanced solutions and all the load imbalanced solutions that are ignored by load balancing methods.