OpenMP Directives Research Articles

PragFormer: Data-Driven Parallel Source Code Classification with Transformers

AbstractMulti-core shared memory architectures have become ubiquitous in computing hardware nowadays. As a result, there is a growing need to fully utilize these architectures by introducing appropriate parallelization schemes, such as OpenMP worksharing-loop constructs, to applications. However, most developers find introducing OpenMP directives to their code hard due to pervasive pitfalls in managing parallel shared memory. To assist developers in this process, many compilers, as well as source-to-source (S2S) translation tools, have been developed over the years, tasked with inserting OpenMP directives into code automatically. In addition to having limited robustness to their input format, these compilers still do not achieve satisfactory coverage and precision in locating parallelizable code and generating appropriate directives. Recently, many data-driven AI-based code completion (CC) tools, such as GitHub CoPilot, have been developed to ease and improve programming productivity. Leveraging the insights from existing AI-based programming-assistance tools, this work presents a novel AI model that can serve as a parallel-programming assistant. Specifically, our model, named PragFormer, is tasked with identifying loops that can benefit from conversion to parallel worksharing-loop construct (OpenMP directive) and even predict the need for specific data-sharing attributes clauses on the fly. We created a unique database, named Open-OMP, specifically for this goal. Open-OMP contains over 32,000 unique code snippets from different domains, half of which contain OpenMP directives, while the other half do not. We experimented with different model design parameters for these tasks and showed that our best-performing model outperforms a statistically-trained baseline as well as a state-of-the-art S2S compiler. In fact, it even outperforms the popular generative AI model of ChatGPT. In the spirit of advancing research on this topic, we have already released source code for PragFormer as well as Open-OMP dataset to public. Moreover, an interactive demo of our tool, as well as a Hugging Face webpage to experiment with our tool, are already available.

Using hardware-transactional-memory support to implement speculative task execution

Loops take up most of the time of computer programs, so optimizing them so that they run in the shortest time possible is a continuous task. However, this task is not negligible; on the contrary, it is an open area of research since many irregular loops are hard to parallelize. Generally, these loops have loop-carried (DOACROSS) dependencies and the appearance of dependencies could depend on the context. Many techniques have been studied to be able to parallelize these loops efficiently; however, for example in the OpenMP standard there is no efficient way to parallelize them. This article presents Speculative Task Execution (STE), a technique that enables the execution of OpenMP tasks in a speculative way to accelerate certain hot-code regions (such as loops) marked by OpenMP directives. It also presents a detailed analysis of the application of Hardware Transactional Memory (HTM) support for executing tasks speculatively and describes a careful evaluation of the implementation of STE using HTM on modern machines. In particular, we consider the scenario in which speculative tasks are generated by the OpenMP taskloop construct (Speculative Taskloop (STL)). As a result, it provides evidence to support several important claims about the performance of STE over HTM in modern processor architectures. Experimental results reveal that: (a) by implementing STL on top of HTM for hot-code regions, speed-ups of up to 5.39× can be obtained in IBM POWER8 and of up to 2.41× in Intel processors using 4 cores; and (b) STL-ROT, a variant of STL using rollback-only transactions (ROTs), achieves speed-ups of up to 17.70× in IBM POWER9 processor using 20 cores.

Ion-molecule collision cross-section calculations using trajectory parallelization in distributed systems

Ion Mobility coupled with Mass Spectrometry (IM-MS) stands as a strong analytical method for structurally characterizing complex molecules. In IM-MS, the sample under investigation is ionized and propelled by an electric field into a drift tube, which collides against a buffer gas. The separation of the ion gas phase is then measured through the differences in their rotationally averaged Collision Cross-Section (CCS) values. The effectiveness of the measured Collision Cross-Section (CCS) for structural characterization critically depends on the validation against theoretical calculations. This validation process relies on intensive molecular mechanics simulations, which can be computationally demanding, especially for large systems such as molecular assemblies and viruses. Therefore, reliable and fast CCS calculations are needed to help interpret IM-MS experimental data. This work presents the MassCCS software, which considerably increases the CCS simulation performance by implementing a linked-cell-based algorithm, incorporating High-Performance Computing (HPC) techniques. We performed extensive tests regarding the system size, shape, and number of CPU cores. Experimental results reveal speedups up to 3 orders of magnitude faster than Collision Simulator for Ion Mobility Spectrometry (CoSIMS) and High-Performance Collision Cross Section (HPCCS), optimized solutions for CCS simulations, for a single node execution. In addition, we extended MassCCS at the inter-node level by employing OpenMP Cluster (OMPC). OMPC is an innovative programming model designed for the development of HPC applications. It streamlines the development process and simplifies software maintenance using only OpenMP directives. Notably, OMPC delivers a performance level comparable to a pure MPI implementation. This enhancement enabled expensive CCS calculations using nitrogen buffer gas for large systems such as human adenovirus with ∼11 million atoms in just ∼4 min, making MassCCS the most performant software nowadays, to the best of our knowledge. MassCCS is available as free software for Academic use at https://github.com/cces-cepid/massccs.

HeteroPP: A directive‐based heterogeneous cooperative parallel programming framework

AbstractHeterogeneous platforms composed of multiple different types of computing devices (such as CPUs, GPUs, and Intel MICs) have been widely used recently. However, most of parallel applications developed in such a heterogeneous platform usually only utilize a certain kind of computing device due to the lack of easy‐to‐use heterogeneous cooperative parallel programming models. To reduce the difficulty of heterogeneous cooperative parallel programming, a directive‐based heterogeneous cooperative parallel programming framework called HeteroPP is proposed. HeteroPP provides an easier way for programmers to fully exploit multiple different types of computing devices to concurrently and cooperatively perform data‐parallel applications on heterogeneous platforms. An extension to OpenMP directives and clauses is proposed to make it possible for programmers to easily offload a data‐parallel compute kernel to multiple different types of computing devices. A source‐to‐source compiler is designed to help programmers to automatically generate multiple device‐specific compute kernels that can be concurrently and cooperatively performed on heterogeneous platforms. Many experiments are conducted with 12 typical data‐parallel applications implemented with HeteroPP on a heterogeneous CPU‐GPU‐MIC platform. The results show that HeteroPP not only greatly simplifies the heterogeneous cooperative parallel programming, but also can fully utilize the CPUs, GPU, and MIC to efficiently perform these applications.

Monte Carlo Simulation of Heteroepitaxial Growth of Ge on Pit-patterned Si(100) Substrate and Parallelization of Calculations

Methods and algorithms of Monte Carlo simulation of heteroepitaxial growth of Ge on Si are considered. The computational complexity analysis of simulation program blocks is carried out. Parallelization of time-consuming cycles using directives of the open standard OpenMP is proposed. The Monte Carlo simulation of heteroepitaxial growth of Ge on pitpatterned Si substrate was carried out. The position of Ge nanoislands is determined by the competition between the nucleation process of three-dimensional islands due to the accumulation of elastic strain in the growing layer and the process of diffusion of Ge atoms into the pits serving as a drain.

Accelerating Coupled-Cluster Calculations with GPUs: An Implementation of the Density-Fitted CCSD(T) Approach for Heterogeneous Computing Architectures Using OpenMP Directives.

An algorithm is presented for the coupled-cluster singles, doubles, and perturbative triples correction [CCSD(T)] method based on the density fitting or the resolution-of-the-identity (RI) approximation for performing calculations on heterogeneous computing platforms composed of multicore CPUs and graphics processing units (GPUs). The directive-based approach to GPU offloading offered by the OpenMP application programming interface has been employed to adapt the most compute-intensive terms in the RI-CCSD amplitude equations with computational costs scaling as , , and (where NO and NV denote the numbers of correlated occupied and virtual orbitals, respectively) and the perturbative triples correction to execute on GPU architectures. The pertinent tensor contractions are performed using an accelerated math library such as cuBLAS or hipBLAS. Optimal strategies are discussed for splitting large data arrays into tiles to fit them into the relatively small memory space of the GPUs, while also minimizing the low-bandwidth CPU-GPU data transfers. The performance of the hybrid CPU-GPU RI-CCSD(T) code is demonstrated on pre-exascale supercomputers composed of heterogeneous nodes equipped with NVIDIA Tesla V100 and A100 GPUs and on the world's first exascale supercomputer named "Frontier", the nodes of which consist of AMD MI250X GPUs. Speedups within the range 4-8× relative to the recently reported CPU-only algorithm are obtained for the GPU-offloaded terms in the RI-CCSD amplitude equations. Applications to polycyclic aromatic hydrocarbons containing 16-66 carbon atoms demonstrate that the acceleration of the hybrid CPU-GPU code for the perturbative triples correction relative to the CPU-only code increases with the molecule size, attaining a speedup of 5.7× for the largest circumovalene molecule (C66H20). The GPU-offloaded code enables the computation of the perturbative triples correction for the C60 molecule using the cc-pVDZ/aug-cc-pVTZ-RI basis sets in 7 min on Frontier when using 12,288 AMD GPUs with a parallel efficiency of 83.1%.

Power Efficient Strassen’s Algorithm using AVX512 and OpenMP in a Multi-core Architecture

This paper presents an effective implementation of Strassen's algorithm for matrix-matrix multiplication on shared memory multi-core architecture. The proposed algorithm aims to augment the computation speed in terms of GFLOPS performance on average 4.5 and 4.1 times faster than Eigen and OpenBLAS, respectively while reducing the power consumption to as low as possible. Our algorithm relies on using AVX512 intrinsics, loop unrolling factor, and OpenMP directives. A new 2D blocking data allocation pattern is proposed for Strassen's algorithm to provide optimized cache temporal and spatial locality. The proposed implementation reduced not only the amount of main memory but also the burden of unnecessary memory allocation/deallocation and data transferring for each level of recursion in Strassen's algorithm. Moreover, the proposed algorithm consumed, on average, 4.25 and 3.67 times lower energy than the multiplication functions of the Eigen and OpenBLAS libraries, respectively. To measure the computational performance with the awareness of power consumption, GFLOPS per Watt (GFPW) is calculated, which out- performed on average 3.78 and 3.47 times higher than those of Eigen and OpenBLAS libraries, respectively.

Resource-Efficient Use of Modern Processor Architectures For Numerically Solving Cardiac Ionic Cell Models.

A central component in simulating cardiac electrophysiology is the numerical solution of nonlinear ordinary differential equations, also called cardiac ionic cell models, that describe cross-cell-membrane ion transport. Biophysically detailed cell models often require a considerable amount of computation, including calls to special mathematical functions. This paper systematically studies how to efficiently use modern multicore CPUs for this costly computational task. We start by investigating the code restructurings needed to effectively enable compiler-supported SIMD vectorisation, which is the most important performance booster in this context. It is found that suitable OpenMP directives are sufficient for achieving both vectorisation and parallelisation. We then continue with an evaluation of the performance optimisation technique of using lookup tables. Due to increased challenges for automated vectorisation, the obtainable benefits of lookup tables are dependent on the hardware platforms chosen. Throughout the study, we report detailed time measurements obtained on Intel Xeon, Xeon Phi, AMD Epyc and two ARM processors including Fujitsu A64FX, while attention is also paid to the impact of SIMD vectorisation and lookup tables on the computational accuracy. As a realistic example, the benefits of performance enhancement are demonstrated by a 109-run ensemble on the Oakforest-PACS system, where code restructurings and SIMD vectorisation yield an 84% reduction in computing time, corresponding to 63,270 node hours.

Accelerating an iterative eigensolver for nuclear structure configuration interaction calculations on GPUs using OpenACC

To accelerate the solution of large eigenvalue problems arising from many-body calculations in nuclear physics on distributed-memory parallel systems equipped with general-purpose Graphic Processing Units (GPUs), we modified a previously developed hybrid MPI/OpenMP implementation of an eigensolver written in Fortran 90 by using an OpenACC directives based programming model. Such an approach requires making minimal changes to the original code and enables a smooth migration of large-scale nuclear structure simulations from a distributed-memory many-core CPU system to a distributed GPU system. However, in order to make the OpenACC based eigensolver run efficiently on GPUs, we need to take into account the architectural differences between a many-core CPU and a GPU device. Consequently, the optimal way to insert OpenACC directives may be different from the original way of inserting OpenMP directives. We point out these differences in the implementation of sparse matrix–vector multiplications with multiple vectors, which constitutes the main cost of the eigensolver, as well as other differences in the preconditioning step and dense linear algebra operations. We compare the performance of the OpenACC based implementation executed on multiple GPUs with the performance on distributed-memory many-core CPUs, and demonstrate significant speedup achieved on GPUs compared to the on-node performance of a many-core CPU. We also show that the overall performance improvement of the eigensolver on multiple GPUs is more modest due to the communication overhead among different MPI ranks.

Content queries and in-depth analysis on version-controlled software

Writing scientific code usually implies the need to coordinate and conflate the contributions of several scientific programmers. Using Git hosting services eases this process, because the hosting services offer many features, which assist in collaborated work on code. The well-established hosting service GitHub has seen continuous growth in terms of number of users, repositories and commits over the last few years; therefore it offers a large data source of scientific codes as well as social interaction of associated scientific programmers.We present a tool, which allows to easily search through relevant GitHub repositories and perform more advanced analyses, which cannot be conducted solely with the GitHub API. Our tool combines benefits from online as well as offline approaches to retrieve and analyse data to optimise time of execution and consumption of storage.We discuss possible use cases and demonstrate the tool's capabilities by investigating the popularity of OpenMP directives in the scientific community.

High performance and energy efficient sobel edge detection

Sobel edge detection is widely used in computer vision and image processing but its processing time becomes a serious problem in real-time environments, especially when an image is very large. Instead of utilizing a hardware-accelerated approach, we propose a purely software-based method which is simpler and cheaper. Our algorithm reduces the number of arithmetic operations and data loads, so that processing speed is increased and energy consumption reduced. The processing time is further reduced by the use of AVX intrinsics and OpenMP directives which distribute the workload among the AVX engines in a multi-core architecture. Our algorithm reduces the number of arithmetic operations by 22.73% compared to that of the state-of-the-art Sobel (SOAS) algorithm, while the number of data loads are reduced by 43.75% compared to SOAS. Performance and energy consumption comparisons between our algorithm and SOAS, as well as with the Sobel functions offered by the OpenCV and IPP libraries are investigated, and the results demonstrate that a multi-core version of our algorithm, implemented by AVX intrinsics, is on average 3.20, 9.34, and 13.99 times faster than IPP, SOAS, and OpenCV respectively. Also, it consumes an average of 2.91, 8.43, and 11.21 times less energy than IPP, SOAS, and OpenCV. Our algorithm, utilizing software modifications alone, benefits from both shorter development time and reduced cost compared to hardware approaches relying on an FPGA, ASIC, or GPU, making it more suitable for resource-constrained environments.

Numerical integration of overlap electron densities: Parallelization strategies for a good load balancing using OpenMP

Here we present a comprehensive assessment of how different OpenMP directives influence the computation speedup for the creation of the overlap electron density using linear combinations of atomic Gaussian-type orbitals. Also, we present the computational performance of a simple algorithm for numerical integration, based on the division of integration space in subspaces (with adaptive sizes considering its integral values - ASII). This algorithm is implemented and compared with the VEGAS algorithm with integrand parallelization. Here we present the performance comparison of 12 different sets of OpenMP directives and collapse clauses, and 15 different basis sets, that were tested for Br2 and H2O2 model systems. The OpenMP directives and clauses tests pointed to the schedule = dynamic or static, Nindex = 1, and collapse (2) clause as the best strategy for a load-balanced parallel computation. The ASII method presents speedup performance compared with a modern parallel implementation of the VEGAS algorithm. The best speedup obtained for the computation at a 64 core platform was ca. 48.0. A GitHub link with the built library and a simple example are also available.

An Improvement of the Matrix-Matrix Multiplication Speed using 2D-Tiling and AVX512 Intrinsics for Multi-Core Architectures

Matrix-matrix multiplication is a time-consuming operation in scientific and engineering applications. When the matrix size is large, it will take a lot of computation time, resulting in slow software which is unacceptable in real-time applications. In this paper, 2D-tiling, loop unrolling, data padding, OpenMP directives, and AVX512 intrinsics are utilized to increase the speed of matrix-matrix multiplication on multi-core architectures. Our algorithm, tested on a Core i9-7900X machine, is more than two times faster than the operations offered by the OpenBLAS and Eigen libraries for single and double precision floating-point matrices. We also propose an equation for parameter tuning which allows our algorithm to be adapted to process any size of matrix on CPUs with different cache organizations.

Three-dimensional discontinuous deformation analysis with explicit contact formulation and block-wise multicore CPU acceleration

In this study, an explicit three-dimensional discontinuous deformation analysis (3D-DDA) is parallelized. Using explicit contact force circumvents the open-close iteration and makes the global stiffness matrix block-diagonal. Post-judgment of contacts is split into two parts to unify the data access pattern, constant acceleration Newmark method is adopted to guarantee unconditional stability of time integration, and square-root-free Cholesky decomposition is adopted to solve small linear equations efficiently. To exploit the multicore CPU with few changes in code and achieve good modularity, block-wise parallelization is adopted, which can realize the full-stage parallelization with just several lines of OpenMP directives. There is no need to modify block specific algorithms, such as contact detection and block information update, for the proposed parallelization, which results in the most efficient parallelization scheme to date. Correctness and efficiency are validated using several numerical examples. The speed-up ratio of 5.0–6.5 is achieved on an 8-core CPU.

Fast Overlap Detection between Hard-Core Colloidal Cuboids and Spheres. The OCSI Algorithm

Collision between rigid three-dimensional objects is a very common modelling problem in a wide spectrum of scientific disciplines, including Computer Science and Physics. It spans from realistic animation of polyhedral shapes for computer vision to the description of thermodynamic and dynamic properties in simple and complex fluids. For instance, colloidal particles of especially exotic shapes are commonly modelled as hard-core objects, whose collision test is key to correctly determine their phase and aggregation behaviour. In this work, we propose the Oriented Cuboid Sphere Intersection (OCSI) algorithm to detect collisions between prolate or oblate cuboids and spheres. We investigate OCSI’s performance by bench-marking it against a number of algorithms commonly employed in computer graphics and colloidal science: Quick Rejection First (QRI), Quick Rejection Intertwined (QRF) and a vectorized version of the OBB-sphere collision detection algorithm that explicitly uses SIMD Streaming Extension (SSE) intrinsics, here referred to as SSE-intr. We observed that QRI and QRF significantly depend on the specific cuboid anisotropy and sphere radius, while SSE-intr and OCSI maintain their speed independently of the objects’ geometry. While OCSI and SSE-intr, both based on SIMD parallelization, show excellent and very similar performance, the former provides a more accessible coding and user-friendly implementation as it exploits OpenMP directives for automatic vectorization.

OpenMP Implementation of a Novel Potential-Field-Data Source-Growth-Based Inversion Approach for 3D Salt Imaging in Deepwater Gulf of Mexico

Potential-field-data imaging of complex geological features in deepwater salt-tectonic regions in the Gulf of Mexico remains an open active research field. There is still a lack of resolution in seismic imaging methods below and in the surroundings of allochthonous salt bodies. In this work, we present a novel three-dimensional potential-field-data simultaneous inversion method for imaging of salt features. This new approach incorporates a growth algorithm for source estimation, which progressively recovers geological structures by exploring a constrained parameter space; restrictions are posed from a priori geological knowledge of the study area. The algorithm is tested with synthetic data corresponding to a real complex salt-tectonic geological setting commonly found in exploration areas of deepwater Gulf of Mexico. Due to the huge amount of data involved in three-dimensional inversion of potential field data, the use of parallel computing techniques becomes mandatory. In this sense, to alleviate computational burden, an easy to implement parallelization strategy for the inversion scheme through OpenMP directives is presented. The methodology was applied to invert and integrate gravity, magnetic and full tensor gradient data of the study area.

Improving the performance of fragments of computer code with high time costs based on parallel programming technologies

This article describes the process of converting the program from single-threaded to multithreaded mode. The influence of the number of threads of execution on the total execution time of the code. The problem of spending much time to run programs solved by incremental parallelization when the code is designed for sequential execution, are added to the parallelizing directives. Are similar libraries for working with threads in other programming languages and the advantages of OpenMP over other libraries. Considered a model of parallelizing OpenMP directives, which is time-consuming to run the snippet in multi-threaded mode. The article also presents example code implemented using the C++ language, which built the parallelizing directives of OpenMP. In this model, the estimated execution time of a code snippet of a program with different number of threads. The study showed that with increasing the number of threads the run time is significantly reduced on the average in 1,5–2 times. This happens until the moment when the number of threads will be equal to the number of processor cores. If the number of threads becomes greater than the number of processor cores, the execution time of the program remains virtually unchanged. Are the OpenMP library allows to significantly shorten the program, which can be critical for works of real-time systems, to reduce the time of creating multithreaded applications is simple inserts directives of the library, to reduce the number of errors made by the programmer in the course of working with multi-threaded applications.

The behavior of vertically non-homogeneous elastic solids under internal rectangular loads

This paper examines the internal rectangular loading problems for an elastic halfspace where the shear modulus varies exponentially or linearly and the Poisson’s ratio keeps constant or varies linearly with depth. The numerical method is developed through applying the fundamental solution of layered elastic solids and integrating numerically it over the loading area. The adaptive integration of the displacement and traction integrals over the loading area is designed to calculate the nearly singular integral for the source point close to an element. The discretization approach is applied to deal with an arbitrarily depth-heterogeneous elastic solid. OpenMP directives are used to parallelize the internal loop, which controls element iterations so that a high computing speed can be obtained. For an axisymmetric internal loading problem, the displacements obtained with the present formulation are in a very good agreement with existing closed-form solutions. Finally, stresses and displacements in non-homogeneous halfspaces induced by horizontally and vertically uniform rectangular loadings are presented. Results illustrate the effect of non-homogeneous properties on the stress and displacement fields.

HDOT — An approach towards productive programming of hybrid applications

A wealth of important scientific and engineering applications are configured for use on high performance computing architectures using functionality found in the MPI specification. This specification provides application developers with a straightforward means for implementing their ideas for execution on distributed-memory parallel processing computers. OpenMP directives provide a means for operating on shared-memory regions of those computers. With the advent of machines composed of many-core processors, the strict synchronisation required by the bulk synchronous parallel (BSP) communication model can hinder performance increases. This is due to the complexity to handle load imbalances, to reduce serialisation imposed by blocking communication patterns, to overlap communication with computation and, finally, to deal with increasing memory overheads. The MPI specification provides advanced features such as non-blocking calls or shared memory to mitigate some of these factors. However, applying these features efficiently usually requires significant changes on the application structure.Task parallel programming models are being developed as a means of mitigating the abovementioned issues but without requiring extensive changes on the application code. In this work, we present a methodology to develop hybrid applications based on tasks called hierarchical domain over-decomposition with tasking (HDOT). This methodology overcomes most of the issues found on MPI-only and traditional hybrid MPI+OpenMP applications. However, by emphasising the reuse of data partition schemes from process-level and applying them to task-level, it enables a natural coexistence between MPI and shared-memory programming models. The proposed methodology shows promising results in terms of programmability and performance measured on a set of applications.

Automatic Parallelization using Open MP Directives

With the increase in the advent of parallel computing, it has become necessary to write OpenMP programs to achieve better speedup and to exploit parallel hardware efficiently. However, to achieve this, the programmers are required to understand OpenMP directives and clauses, the dependencies in their code, etc. A small mistake made by them, such as wrongly analysing a dependency or wrong data scoping of a variable, can result in an incorrect or inefficient program. In this paper, we propose a system which can automate the process of parallelization of a serial C code. The system accepts a serial program as input and generates the corresponding parallel code in OpenMP without altering the core logic of the program. The system has used different data scoping and work sharing constructs available in OpenMP platform.The system designed here aims at parallelizing “for” loops, “while” loops, nested “for” loops and recursive structures.The system has parallelized “for” loop by considering the induction variable. And converted “while” loop to “for” loop for parallelization. The system is tested by providing several programs such as matrix addition, quick sort, linear search etc. as input. The execution time of programs before and after parallelization is determined and a graph is plotted to help visualize the decrease in execution time.