Compiler Directives Research Articles

Evaluating Performance Portability with the CMS Heterogeneous Pixel Reconstruction code

In the past years the landscape of tools for expressing parallel algorithms in a portable way across various compute accelerators has continued to evolve significantly. There are many technologies on the market that provide portability between CPU, GPUs from several vendors, and in some cases even FPGAs. These technologies include C++ libraries such as Alpaka and Kokkos, compiler directives such as OpenMP, the SYCL open specification that can be implemented as a library or in a compiler, and standard C++ where the compiler is solely responsible for the offloading. Given this developing landscape, users have to choose the technology that best fits their applications and constraints. For example, in the CMS experiment the experience so far in heterogeneous reconstruction algorithms suggests that the full application contains a large number of relatively short computational kernels and memory transfer operations. In this work we use a stand-alone version of the CMS heterogeneous pixel reconstruction code as a realistic use case of HEP reconstruction software that is capable of leveraging GPUs effectively. We summarize the experience of porting this code base from CUDA to Alpaka, Kokkos, SYCL, std::par, and OpenMP offloading. We compare the event processing throughput achieved by each version on NVIDIA and AMD GPUs as well as on a CPU, and compare those to what a native version of the code achieves on each platform.

AutoScaleDSE: A Scalable Design Space Exploration Engine for High-Level Synthesis

High-Level Synthesis (HLS) has enabled users to rapidly develop designs targeted for FPGAs from the behavioral description of the design. However, to synthesize an optimal design capable of taking better advantage of the target FPGA, a considerable amount of effort is needed to transform the initial behavioral description into a form that can capture the desired level of parallelism. Thus, a design space exploration (DSE) engine capable of optimizing large complex designs is needed to achieve this goal. We present a new DSE engine capable of considering code transformation, compiler directives (pragmas), and the compatibility of these optimizations. To accomplish this, we initially express the structure of the input code as a graph to guide the exploration process. To appropriately transform the code, we take advantage of ScaleHLS based on the multi-level compiler infrastructure (MLIR). Finally, we identify problems that limit the scalability of existing DSEs, which we name the “design space merging problem.” We address this issue by employing a Random Forest classifier that can successfully decrease the number of invalid design points without invoking the HLS compiler as a validation tool. We evaluated our DSE engine against the ScaleHLS DSE, outperforming it by a maximum of 59×. We additionally demonstrate the scalability of our design by applying our DSE to large-scale HLS designs, achieving a maximum speedup of 12× for the benchmarks in the MachSuite and Rodinia set.

Toward Ultrahigh-Resolution E3SM Land Modeling on Exascale Computers

We present an ultra-high-resolution E3SM land model (uELM) for high-fidelity land simulations targeting new Exascale computers. After considering modeling infrastructure compatibility and ELM software features, we designed a parallel model for the uELM development targeting hybrid architectures of new US Exascale computers. We also described a function unit test framework to expedite the piece-wise code porting (with compiler directives), verification, and global variable management. Furthermore, in this study, we report an early uELM model development using OpenACC within a function unit test framework on a pre-Exascale computer, demonstrate the performance of a uLEM submodel with a 3.0-time speedup, and summarize the code porting experience regarding global variable handling, deepcopy, memory reduction, and parallel loop reconstruction.

A 3D transient CFD model to predict heat and moisture transfer in on-farm stored grain silo through parallel computing using compiler directives: Impact of discretization methods on solution efficacy

Computational fluid dynamics model parameters highly depend on the storage structure, grain conditions, and airflow properties. Solution methods obtained for the lab-scale bins cannot be extrapolated for a relatively large on-farm silo. To validate the above hypothesis, a model was developed and validated with stored barley aeration in a 1000t silo. A mathematical function was used to initialize the discrete initial conditions in the silo followed by using DEFINE_macros to execute parallel computing. Time-step analysis was conducted followed by optimization of the solution methods in terms of the accuracy and time frame of the model’s output. Results showed that with a time-step of 4 s, temperature and moisture error were 1.1–1.7 °C and 0.3% wb while mean relative deviation were 3.1–4.4% and 2.2%, respectively. COUPLED algorithm resulted in a similar accuracy as of SIMPLE scheme except within spatial and transient formulations. However, the former algorithm was observed to take almost 190% more time than the latter scheme, limiting the simulation efficiency in an on-farm silo. Green Gauss Node-based gradient technique was found to be the appropriate for discretizing large silos. Model showed 45 kJ kg−1 energy emission that decreased the cooling potential of air in silo. As field evaluation of aeration strategies are time-consuming, this model can be used to obtain results that could shape the stored grain management practice.

A New Multi-Target Compiler Architecture for Edge-Devices and Cloud Management

Edge computing is the concept where the computation is handled at edge-devices. The transfer of the computation from servers to edge-devices will decrease the massive amount of data transfer generated by edge-devices. There are several efficient management tools for setup and connection purposes, but these management tools cannot provide a unified programming system from a single source code/project. Even though it is possible to control each device efficiently, a global view of the computation is missing in a programming project that includes several edge-devices for computation and data analysis purposes, and the devices need to be programmed individually. A generic workflow engine might automate part of the problem using standard interfaces and predefined objects running on edge-devices. Nevertheless, the approach fails in fine-tuning each edge-device since the computation cannot be moved easily among devices. This paper introduces a new compiler architecture to control and program edge-devices from a single source code. The source code can be distributed to multiple edge-devices using simple compiler directives, and the transfer and communication of the source code with multiple devices are handled transparently. Fine-tuning the source code and code movement between devices becomes very efficient in editing and time. The proposed architecture is a lightweight system with fine-tuned computation and distribution among devices.

Code Transformation Impact on Compiler-based Optimization: A Case Study in the CMSSW

In this paper, we study the benefit of applying loop transformations to a part of module in the CMS software. Particularly, we focus at the effect of loop transformations in term of performance improvement from the optimization process of compilers. Loop optimizations have been considered at low-level phase, such as loop unrolling using compiler directive. For high-level code transformations such as index set splitting and loop reordering, we adopt the polyhedral model to simplify the transformations. In this study, our loop optimization has been evaluated quantitatively. We study the impact on loops execution speed up and its instruction executed. Our observation shows that high-level loop optimizations can reduce both execution time and the number of instruction. This behavior suggested that simple loop transformations can trigger other optimizations. Moreover, we not only improve the overall performance, but also reduce the number of instruction. The results show that loop optimizations yield the speed up between 1.5 and 1.7.

High‐performance SIMD modular arithmetic for polynomial evaluation

SummaryTwo essential problems in computer algebra, namely polynomial factorization and polynomial greatest common divisor computation, can be efficiently solved thanks to multiple polynomial evaluations in two variables using modular arithmetic. In this article, we focus on the efficient computation of such polynomial evaluations on one single CPU core. We first show how to leverage SIMD (single instruction, multiple data) computing for modular arithmetic on AVX2 and AVX‐512 units, using both intrinsics and OpenMP compiler directives. Then we manage to increase the operational intensity and to exploit instruction‐level parallelism in order to increase the compute efficiency of these polynomial evaluations. All this results in the end to performance gains up to about 5x on AVX2 and 10x on AVX‐512.

A Layered Mapping of Ada 202X to OpenMP

The OpenMP specification defines a set of compiler directives, library routines, and environment variables that together represent the OpenMP Application Programming Interface, and is currently defined for C, C++, and Fortran. The forthcoming version of Ada, currently dubbed Ada 202X, includes lightweight parallelism features, in particular parallel blocks and parallel loops. All versions of Ada, since its inception in 1983, have included "tasking," which corresponds to what are traditionally considered "heavyweight" parallelism features, or simply "concurrency" features. Ada "tasks" typically map to what are called "kernel threads," in that the operating system manages them and schedules them. However, one of the goals of lightweight parallelism is to reduce overhead by doing more of the management outside the kernel of the operating system, using a light-weight-thread (LWT) scheduler. The OpenMP library routines support both levels of threading, but for Ada 202X, the main interest is in making use of OpenMP for its lightweight thread scheduling capabilities.

Correct program parallelisations

A commonly used approach to develop deterministic parallel programs is to augment a sequential program with compiler directives that indicate which program blocks may potentially be executed in parallel. This paper develops a verification technique to reason about such compiler directives, in particular to show that they do not change the behaviour of the program. Moreover, the verification technique is tool-supported and can be combined with proving functional correctness of the program. To develop our verification technique, we propose a simple intermediate representation (syntax and semantics) that captures the main forms of deterministic parallel programs. This language distinguishes three kinds of basic blocks: parallel, vectorised and sequential blocks, which can be composed using three different composition operators: sequential, parallel and fusion composition. We show how a widely used subset of OpenMP can be encoded into this intermediate representation. Our verification technique builds on the notion of iteration contract to specify the behaviour of basic blocks; we show that if iteration contracts are manually specified for single blocks, then that is sufficient to automatically reason about data race freedom of the composed program. Moreover, we also show that it is sufficient to establish functional correctness on a linearised version of the original program to conclude functional correctness of the parallel program. Finally, we exemplify our approach on an example OpenMP program, and we discuss how tool support is provided.

Extending the OpenCHK Model with advanced checkpoint features

One of the major challenges in using extreme scale systems efficiently is to mitigate the impact of faults. Application-level checkpoint/restart (CR) methods provide the best trade-off between productivity, robustness, and performance.There are many solutions implementing CR at the application level. They all provide advanced I/O capabilities to minimize the overhead introduced by CR. Nevertheless, there is still room for improvement in terms of programmability and flexibility, because end-users must manually serialize and deserialize application state using low-level APIs, modify the flow of the application to consider restarts, or rewrite CR code whenever the backend library changes.In this work, we propose a set of compiler directives and clauses that allow users to specify CR operations in a simple way. Our approach supports the common CR features provided by all the CR libraries. However, it can also be extended to support advanced features that are only available in some CR libraries, such as differential checkpointing, the use of HDF5 format, and the possibility of using fault-tolerance-dedicated threads.The result of our evaluation revealed a high increase in programmability. On average, we reduced the number of lines of code by 71%, 94%, and 64% for FTI, SCR, and VeloC, respectively, and no additional overhead was perceived using our solution compared to using the backend libraries directly. Finally, portability is enhanced because our programming model allows the use of any backend library without changing any code.

Debugging Parallel Programs in DVM-System

DVM-system is designed for the development of parallel programs of scientific and technical calculations in the C-DVMH and Fortran-DVMH languages. These languages use a single DVMH-model of parallel programming model and are an extension of the standard C and Fortran languages with parallelism specifications in the form of compiler directives. The DVMH model makes it possible to create efficient parallel programs for heterogeneous computing clusters, in the nodes of which accelerators, graphic processors or Intel Xeon Phi coprocessors can be used as computing devices along with universal multi-core processors. The article describes the method of debugging parallel programs in DVM-system, as well as new features of DVM-debugger.

The Using of DVM-System for Developing of a Program for Calculations of the Problem of Radiation Magnetic Gas Dynamics and Research of Plasma Dynamics in the QSPA Channel

DVM-system is designed for the development of parallel programs of scientific and technical calculations in the C-DVMH and Fortran-DVMH languages. These languages use a single DVMH-model of parallel programming model and are an extension of the standard C and Fortran languages with parallelism specifications in the form of compiler directives. The DVMH model makes it possible to create efficient parallel programs for heterogeneous computing clusters, in the nodes of which accelerators, graphic processors or Intel Xeon Phi coprocessors can be used as computing devices along with universal multi-core processors. The article describes the experience of the successful using of DVM-system to develop a parallel software code for calculating the problem of radiation magnetic gas dynamics and for research of plasma dynamics in the QSPA channel.

Investigation of Data Dependencies by Dynamic Analysis of Sapfor

The use of pointers and indirect memory accesses in the program, as well as the complex control flow are some of the main weaknesses of the static analysis of programs. The program properties investigated by this analysis are too conservative to accurately describe program behavior and hence they prevent parallel execution of the program. The application of dynamic analysis allows us to expand the capabilities of semi-automatic parallelization. In the SAPFOR system (System FOR Automated Parallelization), a dynamic analysis tool has been implemented, based on on the instrumentation of the LLVM representation of an analyzed program, which allows the system to explore programs in both C and Fortran programming languages. The capabilities of the static analysis implemented in SAPFOR are used to reduce the overhead program execution, while maintaining the completeness of the analysis. The use of static analysis allows to reduce the number of analyzed memory accesses and to ignore scalar variables, which can be explored in a static way. The developed tool was tested on performance tests from the NAS Parallel Benchmarks package for C and Fortran languages. The implementation of dynamic analysis, in addition to traditional types of data dependencies (flow, anit, output), allows us to determine privitizable variables and a possibility of pipeline execution of loops. Together with the capabilities of DVM and OpenMP these greatly facilitates program parallelization and simplify insertion of the appropriate compiler directives.

Progress in Dvm-System

DVM-system is designed for the development of parallel programs of scientific and technical calculations in the C-DVMH and Fortran-DVMH languages. These languages use a single DVMH-model of parallel programming model and are an extension of the standard C and Fortran languages with parallelism specifications in the form of compiler directives. The DVMH model makes it possible to create efficient parallel programs for heterogeneous computing clusters, in the nodes of which accelerators, graphic processors or Intel Xeon Phi coprocessors can be used as computing devices along with universal multi-core processors. The article presents new features of DVM-system that have been developed recently.

Numerical multi-loop integration on heterogeneous many-core processors

We report on multi-loop integral computations executed on a PEZY/Exascaler large-scale (immersion cooling) computing system. The programming model requires a host program written in C++ with a PZCL (OpenCL-like) kernel. However the kernel can be generated by the Goose compiler interface, which allows parallelizing program loops according to compiler directives. As an advantage, the executable derived from a program instrumented with Goose pragmas can be run on multiple devices and multiple nodes without changes to the program. We use lattice rules and lattice copy (composite) rules on PEZY to approximate integrals for multi-loop self-energy diagrams with and without masses. GPU results are also given and the performnce on the different architectures is compared.

Automatic Port to OpenACC/OpenMP for Physical Parameterization in Climate and Weather Code Using the CLAW Compiler

In order to benefit from emerging high-performance computing systems, weather and climate models need to be adapted to run efficiently on different hardware architectures such as accelerators. This is a major challenge for existing community models that represent extremely large codebase written in Fortran. Large parts of the code can be ported using OpenACC compiler directives but for time-critical components such as physical parameterizations, code restructuring and optimizations specific to a hardware architecture are necessary to obtain high performance. In an effort to retain a single source code for multiple target architectures, the CLAW Compiler and the CLAW Single Column Abstraction were introduced. We report on the extension of the CLAW SCA to handle ELEMENTAL functions and subroutines. We demonstrate the new capability on the JSBACH land surface scheme of the ICON climate model. With the extension, JSBACH can be automatically ported to OpenACC or OpenMP for accelerators with minimal to no change to the original code.

OpenACC acceleration for the [formula omitted] algorithm in Nek5000

Due to its high performance and throughput capabilities, GPU-accelerated computing is becoming a popular technology in scientific computing, in particular using programming models such as CUDA and OpenACC. The main advantage with OpenACC is that it enables to simply port codes in their “original” form to GPU systems through compiler directives, thus allowing an incremental approach. An OpenACC implementation is applied to the CFD code Nek5000 for simulation of incompressible flows, based on the spectral-element method. The work follows up previous implementations and focuses now on the PN−PN−2 method for the spatial discretization of the Navier–Stokes equations. Performance results of the ported code show a speed-up of up to 3.1 on multi-GPU for a polynomial order N>11.

OpenMP parallel implementation of stiffly stable time-stepping projection/GMRES(ILU(0)) implicit simulation of incompressible fluid flows on shared-memory, multicore architecture

This work showcases that simply by using a few standard OpenMP compiler directives and without any major source code modifications, simulations of both steady and unsteady incompressible fluid flows by the implicit stiffly stable time-stepping projection method could easily be ported to run in parallel on any shared-memory, multicore systems with reasonably good computation performance. The shared-memory, multicore architecture is the state-of-the-art hardware environment for modern microprocessors, including computation nodes of supercomputers, workstation, and personal desktop and laptop. This work has modified the original semi-implicit, stiffly stable time-stepping projection method to be fully implicit. Following the modified method, the approximation of incompressible Navier Stokes (INS) equations is broken into a sequence of steps. Each step consists of three stages, dealing with advection, pressure and viscosity of the INS equations respectively. Each stage of every individual step is governed by its own governing equation(s), physical discretization of which transforms the partial-differential equation(s) to a set of linear algebraic equations, which in general matrix form are Ax=b. The ILU(0) preconditioner and the generalized minimum residual (GMRES) algorithm are used for the preconditioning and solution of Ax=b, respectively. Using Kovasznay and Pearson vortex flows as reference problems for steady and unsteady incompressible flows, respectively, the profiling tests have demonstrated that for both flows, more than 99% of the whole simulation time is spent on the production of ILU(0) preconditioner and GMRES implementation, which are then singled out to be OpenMP parallelized. Benchmarked on a computation node of 16 CPU cores, it has been demonstrated that mean parallel efficiencies of 72% and 73% have been achieved for simulations of Kovasznay and Pearson vortex flows, respectively. When all 16 CPU cores are used, simulations of Kovaszanay and Pearson vortex flows have been accelerated by a factor of 8.0 and 8.17, respectively. Performance and efficiency of OpenMP parallel implementation depend on the hardware system as well. Running the Kovasznay flow simulation on a 12 CPU-core workstation, the mean parallel efficiency is 89%, in contrast to the mean parallel efficiency of 72% that has been achieved on the 16 CPU-core computation node.

Решение прикладных задач с использованием DVM-системы

DVM-system was designed to create parallel programs of scientific-technical computations in C-DVMH and Fortran-DVMH languages. These languages use the same model of parallel programming (DVMH-model) and are the extensions of standard C and Fortran languages by parallelism specifications, implemented as compiler directives. DVMH-model allows creating efficient parallel programs for heterogeneous computational clusters, the nodes of which use as computing devices not only universal multi-core processors but also can use attached accelerators (GPUs or Intel Xeon Phi coprocessors). This article describes the experience of parallelizing various application programs using DVM-system. The method of incremental or partial parallelization, the system's capabilities for working with unstructured grids, new tools for mapping MPI-programs to multi-core processors and accelerators are considered. The efficiency of parallel DVMH-programs on heterogeneous computing clusters K-10, K-100, Lomonosov and MVS-10P is investigated. The main advantages of DVM-approach for the development of parallel programs are described. The main features of DVM-system tools for performance analysis and functional debugging of parallel programs are presented. The directions for further development of DVM-system are determined.

Parallelization of Jump Flood-Cut Method in Shared Memory Multicore Systems

Graph cut approach has evidenced to be trenchant strategy for the segmentation of digital images. Primary restraint of existing graph cut method is that of acute arithmetic calculation which use up tremendous processing time to carry into action. This paper aims on parallelizing jump flooding technique and push re-label scheme with NSP and BFS methodology on shared memory multicore system using OpenMP compiler directives. OpenMP API includes a rich set of library routines and environment variables which enables the parallelization of JF-Cut method within a framework where parallelism is achieved by sharing a single common memory among multiple co-processors. OpenMP remains live area of interest for research in segmentation of digital images as it accelerates computational speed of complex scientific applications like image segmentation using multicore architecture in shared memory mode.