Intel Xeon Phi Knights Landing Research Articles

Electronic Structure Theory Calculations Using Modern Architectures: KNL vs Haswell

The time to solution and parallel efficiency of several commonly used electronic structure methods (Hartree-Fock, density functional theory, second order perturbation theory, resolution of the identity second order perturbation theory, coupled cluster) are evaluated on both the Intel Xeon Haswell and the Intel Xeon Phi Knights Landing (KNL) architectures. The Haswell completes the benchmark calculations with a faster time to solution than the KNL for all molecules and methods tested. While the Haswell exhibits an average speedup of at least 3.5 relative to the KNL for all nonthreaded computations, the KNL has a better parallel efficiency than the Haswell with increasing core counts. The architectures are further tested using a more computationally costly coupled cluster method on a transition state reaction. The Haswell appears to be the best choice to minimize the time to solution, though for very large systems and high levels of theory that require memory intensive processes the superior memory hierarchy and larger on node memory of the KNL can make it a better choice. These results are used to showcase aspects of novel architectures that will increase efficiency for quantum chemistry applications.

Cost-efficient simulations of large-scale electronic structures in the standalone manycore architecture

The tight-binding (TB) model is complementary to the ab-initio methods that can be represented by the well-known Density Functional Theory since the empirical nature of a TB model enables the scope of electronic structure simulations to include realistically sized nanoscale structures that are normally made up of several million atoms. As the major computational hotspot of TB simulations comes from diagonalization of a sparse system matrix whose size is proportional to the number of atoms belonging to a simulation domain, efficient utilization of high performance computers is strongly encouraged. Here we study the feasibility of cost-efficient TB simulations for large-scale electronic structures on the Intel Xeon Phi Knights Landing (KNL) platform, which is equipped with the processor of many-integrated cores and the onboard high-speed memory. Using our in-house code whose scientific fidelity has been validated through various modeling researches, we present detailed discussion on how KNL systems can be exploited to accelerate simulations, and conduct rigorous benchmark tests in competing platforms to justify benefits of the standalone manycore architecture in terms of the time and the energy consumption that must be paid. The capability to handle extremely huge atomic structures in KNL systems is also demonstrated by securing a strong scalability up to 2,500 nodes in the NURION supercomputer for a model problem that has 400 million atoms.

Enabling Large-Scale Simulations of Quantum Transport with Manycore Computing

The non-equilibrium Green’s function (NEGF) is being utilized in the field of nanoscience to predict transport behaviors of electronic devices. This work explores how much performance improvement can be driven for quantum transport simulations with the aid of manycore computing, where the core numerical operation involves a recursive process of matrix multiplication. Major techniques adopted for performance enhancement are data restructuring, matrix tiling, thread scheduling, and offload computing, and we present technical details on how they are applied to optimize the performance of simulations in computing hardware, including Intel Xeon Phi Knights Landing (KNL) systems and NVIDIA general purpose graphic processing unit (GPU) devices. With a target structure of a silicon nanowire that consists of 100,000 atoms and is described with an atomistic tight-binding model, the effects of optimization techniques on the performance of simulations are rigorously tested in a KNL node equipped with two Quadro GV100 GPU devices, and we observe that computation is accelerated by a factor of up to ∼20 against the unoptimized case. The feasibility of handling large-scale workloads in a huge computing environment is also examined with nanowire simulations in a wide energy range, where good scalability is procured up to 2048 KNL nodes.

Empirical Performance Analysis of Collective Communication for Distributed Deep Learning in a Many-Core CPU Environment

To accommodate lots of training data and complex training models, “distributed” deep learning training has become employed more and more frequently. However, communication bottlenecks between distributed systems lead to poor performance of distributed deep learning training. In this study, we proposed a new collective communication method in a Python environment by utilizing Multi-Channel Dynamic Random Access Memory (MCDRAM) in Intel Xeon Phi Knights Landing processors. Major deep learning software platforms, such as TensorFlow and PyTorch, offer Python as a main development language, so we developed an efficient communication library by adapting Memkind library, which is a C-based library to utilize high-performance memory MCDRAM. For performance evaluation, we tested the popular collective communication methods in distributed deep learning, such as Broadcast, Gather, and AllReduce. We conducted experiments to analyze the effect of high-performance memory and processor location on communication performance. In addition, we analyze performance in a Docker environment for further relevance given the recent major trend of Cloud computing. By extensive experiments in our testbed, we confirmed that the communication in our proposed method showed performance improvement by up to 487%.

AMR-based molecular dynamics for non-uniform, highly dynamic particle simulations

Accurate simulations of metal under heavy shocks, leading to fragmentation and ejection of particles, cannot be achieved by simply hydrodynamic models and require to be performed at atomic scale using molecular dynamics methods. In order to cope with billions of particles exposed to short range interactions, such molecular dynamics methods need to be highly optimized over massively parallel supercomputers. In this paper, we propose to leverage Adaptive Mesh Refinement techniques to improve efficiency of molecular dynamics code on highly heterogeneous particle configurations. We introduce a series of techniques that optimize the force computation loop using multi-threading and vectorization-friendly data structures. Our design is guided by the need for load balancing and adaptivity raised by highly dynamic particle sets. We analyze performance results on several simulation scenarios, such as the production of an ejecta cloud from shock-loaded metallic surfaces, using a large number of nodes equipped by Intel Xeon Phi Knights Landing processors. Performance obtained with our new Molecular Dynamics code achieves speedups greater than 1.38 against the state-of-the-art LAMMPS implementation.

Genome Sequence Alignment - Design Space Exploration for Optimal Performance and Energy Architectures

Next generation workloads, such as genome sequencing, have an astounding impact in the healthcare sector. Sequence alignment, the first step in genome sequencing, has experienced recent breakthroughs, which resulted in next generation sequencing (NGS). As NGS applications are memory bounded with random memory access patterns, we propose the use of high bandwidth memories like 3D stacked HBM2, instead of traditional DRAMs like DDR4, along with energy efficient compute cores to improve both performance and energy efficiency. Three state-of-the-art NGS applications, Bowtie2, BWA-MEM, and HISAT2 are used as case studies to explore and optimize NGS computing architectures. Then, using the gem5-X architectural simulator, we obtain an overall 68 percent performance improvement and 71 percent energy savings using HBM2 instead of DDR4. Furthermore, we propose an architecture based on ARMv8 cores and demonstrate that 16 ARMv8 64-bit OoO cores with HBM2 outperforms 32-cores of Intel Xeon Phi Knights Landing (KNL) processor with 3D stacked memory. Moreover, we show that by using frequency scaling we can achieve up to 59 percent and 61 percent energy savings for ARM in-order and OoO cores, respectively. Lastly, we show that many ARMv8 in-order cores at 1.5GHz match the performance of fewer OoO cores at 2GHz, while attaining 4.5x energy savings.

On the code modernization of shared sampling alpha matting with OpenMP

Image matting separates the foreground object from a given image by estimating the alpha values. It has been one crucial step in computer vision, as well as an essential technique in visual effects in the modern filmmaking industry. Due to its time complexity being proportional to the size of the unknown region, the computational speed for processing image matting on high-resolution images has been traditionally slow. In this paper, we present the code modernization of Shared Sampling Alpha Matting (SSAM) algorithm with OpenMP to speed up the computation on multicore server platforms. Significant efforts are required due to the sequential nature of the code, with data dependencies and complex data structures in the original code. The implementation of migrating the SSAM code to multicore platforms and the experiments performed on several server platforms of different categories/specs are presented. Our experimental results demonstrate that the Alpha Matting using OpenMP increases performance significantly on Intel Xeon Phi Knights Landing (KNL) processors when processing high-resolution images. The code improvements we achieved can support a parallel application developer with high-performance goals.

Challenges and opportunities for the simulation of calcium waves on modern multi-core and many-core parallel computing platforms.

State-of-the-art distributed-memory computer clusters contain multicore CPUs with 16 and more cores. The second generation of the Intel Xeon Phi many-core processor has more than 60cores with 16GB of high-performance on-chip memory. We contrast the performance of the second-generation Intel Xeon Phi, code-named Knights Landing (KNL), with 68computational cores to the latest multicore CPU Intel Skylake with 18cores. A special-purpose code solving a system of nonlinear reaction-diffusion partial differential equations with several thousands of point sources modeled mathematically by Dirac delta distributions serves as realistic test bed. The system is discretized in space by the finite volume method and advanced by fully implicit time-stepping, with a matrix-free implementation that allows the complex model to have an extremely small memory footprint. The sample application is a seven variable model of calcium-induced calcium release (CICR) that models the interplay between electrical excitation, calcium signaling, and mechanical contraction in a heart cell. The results demonstrate that excellent parallel scalability is possible on both hardware platforms, but that modern multicore CPUs outperform the specialized many-core Intel Xeon Phi KNL architecture for a large class of problems such as systems of parabolic partial differential equations.

Accelerating time series motif discovery in the Intel Xeon Phi KNL processor

Time series analysis is an important research topic of great interest in many fields. Recently, the Matrix Profile method, and particularly one of its implementations—the SCRIMP algorithm—has become a state-of-the-art approach in this field. This is a technique that brings the possibility of obtaining exact motifs from a time series, which can be used to infer events, predict outcomes, detect anomalies and more. However, the memory-bound nature of the SCRIMP algorithm limits the execution performance in some processor architectures. In this paper, we analyze the SCRIMP algorithm from the performance viewpoint in the context of the Intel Xeon Phi Knights Landing architecture (KNL), which integrates high-bandwidth memory (HBM) modules, and we combine several techniques aimed at exploiting the potential of this architecture. On the one hand, we exploit the multi-threading and vector capabilities of the architecture. On the other hand, we explore how to allocate data in order to take advantage of the available hybrid memory architecture that conjugates both the high-bandwidth 3D-stacked HBM and the DDR4 memory modules. The experimental evaluation shows a performance improvement up to $$190\,\times $$ with respect to the sequential execution and that the use of the HBM memory improves performance in a factor up to $$5\,\times $$ with respect to the DDR4 memory.

Optimizing the HOMME dynamical core for multicore platforms

The approach of the next-generation computing platforms offers a tremendous opportunity to advance the state-of-the-art in global atmospheric dynamical models. We detail our incremental approach to utilize this emerging technology by enhancing concurrency within the High-Order Method Modeling Environment (HOMME) atmospheric dynamical model developed at the National Center for Atmospheric Research (NCAR). The study focused on improvements to the performance of HOMME which is a Fortran 90 code with a hybrid (MPIOpenMP) programming model. The article describes the changes made to the use of message passing interface (MPI) and OpenMP as well as single-core optimizations to achieve significant improvements in concurrency and overall code performance. For our optimization studies, we utilize the “Cori” system with an Intel Xeon Phi Knights Landing processor deployed at the National Energy Research Supercomputing Center and the “`Cheyenne” system with an Intel Xeon Broadwell processor installed at the NCAR. The results from the studies, using “workhorse” configurations performed at NCAR, show that these changes have a transformative impact on the computational performance of HOMME. Our improvements have shown that we can effectively increase potential concurrency by efficiently threading the vertical dimension. Further, we have seen a factor of two overall improvement in the computational performance of the code resulting from the single-core optimizations. Most notably from the work is that our incremental approach allows for high-impact changes without disrupting existing scientific productivity in the HOMME community.

Vectorization of High-performance Scientific Calculations Using AVX-512 Intruction Set

Modern calculation codes used in supercomputing are very demanding of computing resources. For their effective appliance requires the use of parallelization at all levels, starting with the use of multiprocess and multi-threaded programming, and ending with vectorization. The AVX-512 instruction set, first introduced in Intel Xeon Phi Knights Landing and Intel Xeon Skylake microprocessors, opens up broad possibilities for vectorizing code and allows to speed up the execution of applications in several times. This article discusses some aspects of the application of vectorization in the program code of some kinds, which is found in high-performance scientific computing.

Re-engineering the ant colony optimization for CMP architectures

The ant colony optimization (ACO) is inspired by the behavior of real ants, and as a bioinspired method, its underlying computation is massively parallel by definition. This paper shows re-engineering strategies to migrate the ACO algorithm applied to the Traveling Salesman Problem to modern Intel-based multi- and many-core architectures in a step-by-step methodology. The paper provides detailed guidelines on how to optimize the algorithm for the intra-node (thread and vector) parallelization, showing the performance scalability along with the number of cores on different Intel architectures, reporting up to 5.5x speedup factor between the Intel Xeon Phi Knights Landing and Intel Xeon v2. Moreover, parallel efficiency is provided for all targeted architectures, finding that core load imbalance, memory bandwidth limitations, and NUMA effects on data placement are some of the key factors limiting performance. Finally, a distributed implementation is also presented, reaching up to 2.96x speedup factor when running the code on 3 nodes over the single-node counterpart version. In the latter case, the parallel efficiency is affected by the synchronization frequency, which also affects the quality of the solution found by the distributed implementation.

HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model

Abstract. We present an architecture-portable and performant implementation of the atmospheric dynamical core (High-Order Methods Modeling Environment, HOMME) of the Energy Exascale Earth System Model (E3SM). The original Fortran implementation is highly performant and scalable on conventional architectures using the Message Passing Interface (MPI) and Open MultiProcessor (OpenMP) programming models. We rewrite the model in C++ and use the Kokkos library to express on-node parallelism in a largely architecture-independent implementation. Kokkos provides an abstraction of a compute node or device, layout-polymorphic multidimensional arrays, and parallel execution constructs. The new implementation achieves the same or better performance on conventional multicore computers and is portable to GPUs. We present performance data for the original and new implementations on multiple platforms, on up to 5400 compute nodes, and study several aspects of the single- and multi-node performance characteristics of the new implementation on conventional CPU (e.g., Intel Xeon), many core CPU (e.g., Intel Xeon Phi Knights Landing), and Nvidia V100 GPU.

Performance of the AMS Offline Software at National Energy Research Scientific Computing Centre and Argonne Leadership Computing Facility

The Alpha Magnetic Spectrometer [1] (AMS) is a high energy physics experiment installed and operating on board of the International Space Station (ISS) from May 2011 and expected to last through year 2024 and beyond. More than 50 million of CPU hours has been delivered for AMS Monte Carlo simulations using NERSC and ALCF facilities in 2017. The details of porting of the AMS software to the 2nd Generation Intel Xeon Phi Knights Landing architecture are discussed, including the MPI emulation module to allow the AMS offline software to be run as multiple-node batch jobs. The performance of the AMS simulation software at NERSC Cori (KNL 7250), ALCF Theta (KNL 7230), and Mira (IBM BG/Q) farms is also discussed.

Resource aggregation for task-based Cholesky Factorization on top of modern architectures

Hybrid computing platforms are now commonplace, featuring a large number of CPU cores and accelerators. This trend makes balancing computations between these heterogeneous resources performance critical. In this paper we propose ag-gregating several CPU cores in order to execute larger parallel tasks and improve load balancing between CPUs and accelerators. Additionally, we present our approach to exploit internal parallelism within tasks, by combining two runtime system schedulers: a global runtime system to schedule the main task graph and a local one one to cope with internal task parallelism. We demonstrate the relevance of our approach in the context of the dense Cholesky factorization kernel implemented on top of the StarPU task-based runtime system. We present experimental results showing that our solution outperforms state of the art implementations on two architectures: a modern heterogeneous machine and the Intel Xeon Phi Knights Landing.

Multinode implementation of an extended Hodgkin–Huxley simulator

Mathematical models with varying degrees of complexity have been proposed and simulated in an attempt to represent the intricate mechanisms of the human neuron. One of the most biochemically realistic and analytical models, based on the Hodgkin–Huxley (HH) model, has been selected for study in this paper. In order to satisfy the model’s computational demands, we present a simulator implemented on Intel Xeon Phi Knights Landing manycore processors. This high-performance platform features an x86-based architecture, allowing our implementation to be portable to other common manycore processing machines. This is reinforced by the fact that Phi adopts the popular OpenMP and MPI programming models. The simulator performance is evaluated when calculating neuronal networks of varying sizes, density and network connectivity maps. The evaluation leads to an analysis of the neuronal synaptic patterns and their impact on performance when tackling this type of workload on a multinode system. It will be shown that the simulator can calculate 100 ms of simulated brain activity for up to 2 millions of biophysically-accurate neurons and 2 billion neuronal synapses within one minute of execution time. This level of performance renders the application an efficient solution for large-scale detailed model simulation.

Параллельный алгоритм вычисления матрицы Евклидовых расстояний для многоядерного процессора Intel Xeon Phi поколения Knights Landing

Computation of a Euclidean distance matrix (EDM) is a typical task in a wide spectrum of problems connected with data mining. Currently, many parallel algorithms for this task have been developed for graphical processors. These developments, however, cannot be directly applied to the Intel Many Integrated Core systems. In this paper, we suggest a parallel algorithm for EDM computation on Intel Xeon Phi Knights Landing processor in the case when the input data fit into the main memory. The algorithm exploits block-oriented scheme of computations that allows for the efficient utilization of Intel Xeon Phi vectorization abilities. In the algorithm, we also apply apply a sophisticated data layout to store data points in main memory so as to reduce the number of processor cache misses during EDM computations. Experimental evaluation of the algorithm on real-world and synthetic datasets shows that t is highly scalable and outruns analogues in the case of rectangular matrices with low-dimensional data points.

Optimization of elastodynamic finite integration technique on Intel Xeon Phi Knights Landing processors

This work describes the development and optimization of an implementation of an isotropic elastodynamic finite integration technique (EFIT) code for parallelized computation on Intel Knights Landing (KNL) hardware. EFIT is a numerical approach resulting in standard staggered-grid finite difference equations for the elastodynamic equations of motion to simulate bulk waves in solids. The computationally efficient simulation of elastodynamic wave propagation and interactions in aerospace materials is of high-interest in the fields of nondestructive evaluation (NDE) and structural health monitoring (SHM). Ultrasonic inspection uses an ultrasonic signal, generated at the surface of the material/structure via use of a piezoelectric transducer, to propagate sound waves into the material where it interacts with any existing defects, as well as with structural boundaries and any material inhomogeneity. Reflections from defects and boundaries are then measured by a transducer. Realistic ultrasound simulation tools can significantly aid the development and optimization of inspection techniques and can assist in the interpretation of experimental data.The optimization of an elastodynamics simulation code for the KNL Many Integrated Core processor was performed. The optimization focused on data locality and vectorization. Results show that tiling of the data to exploit the cache behavior and allow for significant utilization of the KNL hardware. The MPI implementation allows for a scalable implementation enabling large problems to be simulated. The model results were validated against theoretical dispersion curves to within 2% of the group velocity, and within 0.5% of the phase velocity of the A0 mode. Aggressive use of tiling, threading, and vectorization techniques allowed for dramatically improved time to solution.

Some useful optimisations for unstructured computational fluid dynamics codes on multicore and manycore architectures

This paper presents a number of optimisations for improving the performance of unstructured computational fluid dynamics codes on multicore and manycore architectures such as the Intel Sandy Bridge, Broadwell and Skylake CPUs and the Intel Xeon Phi Knights Corner and Knights Landing manycore processors. We discuss and demonstrate their implementation in two distinct classes of computational kernels: face-based loops represented by the computation of fluxes and cell-based loops representing updates to state vectors. We present the importance of making efficient use of the underlying vector units in both classes of computational kernels with special emphasis on the changes required for vectorising face-based loops and their intrinsic indirect and irregular access patterns. We demonstrate the advantage of different data layouts for cell-centred as well as face data structures and architectural specific optimisations for improving the performance of gather and scatter operations which are prevalent in unstructured mesh applications. The implementation of a software prefetching strategy based on auto-tuning is also shown along with an empirical evaluation on the importance of multithreading for in-order architectures such as Knights Corner. We explore the various memory modes available on the Intel Xeon Phi Knights Landing architecture and present an approach whereby both traditional DRAM as well as MCDRAM interfaces are exploited for maximum performance. We obtain significant full application speed-ups between 2.8 and 3X across the multicore CPUs in two-socket node configurations, 8.6X on the Intel Xeon Phi Knights Corner coprocessor and 5.6X on the Intel Xeon Phi Knights Landing processor in an unstructured finite volume CFD code representative in size and complexity to an industrial application. Program summaryProgram Title: some_opt_for_unstructured_cfdProgram Files doi:http://dx.doi.org/10.17632/zyh2zkf3jw.1Licensing provisions: GNU General Public License 3 (GPL)Programming language: C/C++Nature of problem: The solution of fluid flow problems in the vicinity of complex geometries mandates the utilisation of unstructured grids. However, this flexibility of unstructured mesh methods in dealing with complicated geometries comes at a cost of increased difficulty in extracting high performance out of modern processors. We provide implementations for a number of optimisations useful for improving the performance of unstructured CFD codes on modern multicore and manycore architectures.Solution method: grid renumbering via Reverse Cuthill–Mckee, code transformations necessary for enabling vectorisation, face colouring/reordering for removing dependencies at the face end-points when accumulating residuals, data layout transformations for reducing cache misses, hand-tuned gather and scatter primitives for in-register transpositions, software prefetching via auto-tuning and multithreading for exploiting SMT features of modern processors.

ELT-scale adaptive optics real-time control with the Intel Xeon Phi Many Integrated Core Architecture

We propose a solution to the increased computational demands of Extremely Large Telescope (ELT) scale adaptive optics (AO) real-time control with the Intel Xeon Phi Knights Landing (KNL) Many Integrated Core (MIC) Architecture. The computational demands of an AO real-time controller (RTC) scale with the fourth power of telescope diameter and so the next generation ELTs require orders of magnitude more processing power for the RTC pipeline than existing systems. The Xeon Phi contains a large number (> 64) of low power x86 CPU cores and high bandwidth memory integrated into a single socketed server CPU package. The increased parallelism and memory bandwidth are crucial to providing the performance for reconstructing wavefronts with the required precision for ELT scale AO. Here, we demonstrate that the Xeon Phi KNL is capable of performing ELT scale single conjugate AO real-time control computation at over 1.0 kHz with less than 20 {\mu}s RMS jitter. We have also shown that with a wavefront sensor camera attached the KNL can process the real-time control loop at up to 966 Hz, the maximum frame-rate of the camera, with jitter remaining below 20 {\mu}s RMS. Future studies will involve exploring the use of a cluster of Xeon Phis for the real-time control of the MCAO and MOAO regimes of AO. We find that the Xeon Phi is highly suitable for ELT AO real time control.