Parallelization Of Code Research Articles

An assessment of large language models for OpenMP-based code parallelization: a user perspective

Large language models have sparked a lot of attention in the research community in recent days, especially with the introduction of practical tools such as ChatGPT and Github Copilot. Their ability to solve complex programming tasks was also shown in several studies and commercial solutions increasing the interest in using them for software development in different fields. High performance computing is one of such fields, where parallel programming techniques have been extensively used to utilize raw computing power available in contemporary multicore and manycore processors. In this paper, we perform an evaluation of the ChatGPT and Github Copilot tools for OpenMP-based code parallelization using a proposed methodology. We used nine different benchmark applications which represent typical parallel programming workloads and compared their OpenMP-based parallel solutions produced manually and using ChatGPT and Github Copilot in terms of obtained speedup, applied optimizations, and quality of the solution. ChatGPT 3.5 and Github Copilot installed with Visual Studio Code 1.88 were used. We concluded that both tools can produce correct parallel code in most cases. However, performance-wise, ChatGPT can match manually produced and optimized parallel code only in simpler cases, as it lacks a deeper understanding of the code and the context. The results are much better with Github Copilot, where much less effort is needed to obtain correct and performant parallel solution.

PragFormer: Data-Driven Parallel Source Code Classification with Transformers

AbstractMulti-core shared memory architectures have become ubiquitous in computing hardware nowadays. As a result, there is a growing need to fully utilize these architectures by introducing appropriate parallelization schemes, such as OpenMP worksharing-loop constructs, to applications. However, most developers find introducing OpenMP directives to their code hard due to pervasive pitfalls in managing parallel shared memory. To assist developers in this process, many compilers, as well as source-to-source (S2S) translation tools, have been developed over the years, tasked with inserting OpenMP directives into code automatically. In addition to having limited robustness to their input format, these compilers still do not achieve satisfactory coverage and precision in locating parallelizable code and generating appropriate directives. Recently, many data-driven AI-based code completion (CC) tools, such as GitHub CoPilot, have been developed to ease and improve programming productivity. Leveraging the insights from existing AI-based programming-assistance tools, this work presents a novel AI model that can serve as a parallel-programming assistant. Specifically, our model, named PragFormer, is tasked with identifying loops that can benefit from conversion to parallel worksharing-loop construct (OpenMP directive) and even predict the need for specific data-sharing attributes clauses on the fly. We created a unique database, named Open-OMP, specifically for this goal. Open-OMP contains over 32,000 unique code snippets from different domains, half of which contain OpenMP directives, while the other half do not. We experimented with different model design parameters for these tasks and showed that our best-performing model outperforms a statistically-trained baseline as well as a state-of-the-art S2S compiler. In fact, it even outperforms the popular generative AI model of ChatGPT. In the spirit of advancing research on this topic, we have already released source code for PragFormer as well as Open-OMP dataset to public. Moreover, an interactive demo of our tool, as well as a Hugging Face webpage to experiment with our tool, are already available.

LUAEMA: A Loop Unrolling Approach Extending Memory Accessing for Vector Very-Long-Instruction-Word Digital Signal Processor with Multiple Register Files

Loop unrolling can provide more instruction-level parallelism opportunities for code and enables a greater range of instruction pipeline scheduling. In high-performance very-long-instruction-word (VLIW) digital signal processors (DSPs), there are special registers to address. To further improve the instruction-level parallelism of code for such DSPs by making full use of these registers, in this paper, we propose a more effective loop unrolling approach through extending memory accessing (LUAEMA). In this approach, the final unrolling factor is computed by a model in which every register kind and every memory accessing operation are considered. For basic digital signal processing algorithms, the unrolling factor under the LUAEMA is larger than that under the conventional loop unrolling approach. We also provide the opportunity to reduce the number of instructions in a loop during the code transformation of loop unrolling. The experimental results show that the loop unrolling approach proposed in this paper can achieve an average speedup ratio ranging from 1.14 to 1.81 compared with the conventional loop unrolling approach. For some algorithms, the peak speedup ratio is up to 2.11.

Schematic Program Proofs with Abstract Execution

We propose Execution, a static verification framework based on symbolic execution and dynamic frames for proving properties of schematic programs. Since a schematic program may potentially represent infinitely many concrete programs, Execution can analyze infinitely many programs at once. Trading off expressiveness and automation, the framework allows proving many interesting (universal, behavioral) properties fully automatically. Its main application are correctness proofs of program transformations represented as pairs of schematic programs. We implemented Execution in a deductive verification framework and designed a graphical workbench supporting the modeling process. Execution has been applied to correct code refactoring, analysis of the cost impact of transformation rules, and parallelization of sequential code. Using our framework, we found and reported several bugs in the refactoring engines of the Java IDEs IntelliJ IDEA and Eclipse, which were acknowledged and fixed.

On the parallelization of multipacting simulation codes for the design of particle accelerator components

Particle trajectory and collision simulation is a critical step of the design and construction of novel particle accelerator components. However it requires a huge computational effort which can slow down the design process. We started from a sequential simulation program which is used to study an event called Multipacting. Our work explains the physical problem that is simulated and the implications it can have on the behavior of the components. Then we analyze the original program’s operation to find the best options for parallelization. We first developed a parallel version of the Multipacting simulation and were able to accelerate the execution up to ∼35×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim 35\ imes $$\\end{document} with 48 or 56 cores. In the best cases, parallelization efficiency was maintained up to 16 cores (∼95\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim 95$$\\end{document}%) and the speed-up plateaus at around 40–48 cores. When this first parallelization effort was tried for multi-power simulations, we found that parallelism was severely limited with a maximum of 20×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$20\ imes $$\\end{document} speed-up. For this reason, we introduced a new method to improve the parallelization efficiency for this second use case. This method uses a shared processor pool for all simulations of electrons (OnePool). OnePool improved scalability by pushing the speed-up to over 32×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$32\ imes $$\\end{document}.

Mixed quantum/classical theory (MQCT) approach to the dynamics of molecule-molecule collisions in complex systems.

We developed a general theoretical approach and a user-ready computer code that permit study of the dynamics of collisional energy transfer and ro-vibrational energy exchange in complex molecule-molecule collisions. The method is a mixture of classical and quantum mechanics. The internal ro-vibrational motion of collision partners is treated quantum mechanically using a time-dependent Schrödinger equation that captures many quantum phenomena including state quantization and zero-point energy, propensity and selection rules for state-to-state transitions, quantum symmetry and interference phenomena. A significant numerical speed up is obtained by describing the translational motion of collision partners classically, using the Ehrenfest mean-field trajectory approach. Within this framework a family of approximate methods for collision dynamics is developed. Several benchmark studies for diatomic and triatomic molecules, such as H2O and ND3 collided with He, H2 and D2, show that the results of MQCT are in good agreement with full-quantum calculations in a broad range of energies, especially at high collision energies where they become nearly identical to the full quantum results. Numerical efficiency of the method and massive parallelism of the MQCT code permit us to embrace some of the most complicated collisional systems ever studied, such as C6H6 + He, CH3COOH + He and H2O + H2O. Application of MQCT to the collisions of chiral molecules such as CH3CHCH2O + He, and to molecule-surface collisions is also possible and will be pursued in the future.

A space-time multigrid method for poroelasticity equations with random hydraulic conductivity

In this work, we propose a space-time multigrid method for solving a finite difference discretization of the linear Biot’s model in two space dimensions considering random hydraulic conductivity. This method is an extension of the one developed by Franco et al. [1] and which was applied to the heat equation. Particularly for the poroelasticity problem, we need to use the Tri-Diagonal Matrix Algorithm (TDMA) for systems of equations with multiple variables (block TDMA) solver on the planes xt and yt with zebra-type wise-manner (the domain is divided into planes and solve each odd plane first and then each even plane independently [2]). On the planes xy, the F-cycle multigrid with fixed-stress smoother is improved by red-black relaxation. A fixed-stress smoother is compound solver, where the mechanical and flow parts are addressed by two distinctive solvers and symmetric Gauss-Seidel iteration. This is the basis of the proposed method which does not converge only with the standard line-in-time red-black solver [1]. This combination allows the code parallelization. The use of standard coarsening associated with line or plane smoothers in the strong coupling direction allows an application in real world problems and, in particular, the random hydraulic conductivity case. This would not be possible using the standard space-time method with semicoarsening in the strong coupling direction. The spatial and temporal approximations of the problem are performed, respectively, by using Central Difference Scheme (CDS) and implicit Euler methods. The robustness and excellent performance of the multigrid method, even with random hydraulic conductivity, are illustrated with numerical experiments through the average convergence factor.

Parallelization of a 3D FDTD code and physics studies of electromagnetic wave propagation in fusion plasmas

Numerical codes for electromagnetic wave propagation in magnetized plasmas are mainly based on frequency-domain asymptotic methods, which provide a fast solution and are thus valuable for experiment design and control applications. However, in several cases of practical interest (e.g. mode conversion), these tools run close to their limits of validity and should be compared to full-wave solutions. The code RFFW solves Maxwell's equations with the finite-difference time-domain method in 3D geometry, for scenarios involving high-frequency waves with arbitrary electric field spectrum in plasmas with axisymmetric equilibrium. In fusion-related problems, the code may conduct investigations of wave propagation and absorption relevant to auxiliary plasma heating and current drive, reflectometry and instability control. The code has been parallelized with a hybrid OpenMP-MPI scheme, which has allowed exploiting the much larger processing power and memory of current-day supercomputers. In this work, we present the main aspects of the physics implemented in the code, and also refer shortly to the parallelization scheme. Furthermore, we show results that exhibit the strong scaling performance of the code, and examine cases of electron-cyclotron heating application in medium-sized tokamaks.

PyKleeBarcode: Enabling representation of the whole animal kingdom in information space.

As biological sequence databases continue growing, so do the insight that they promise to shed on the shape of the genetic diversity of life. However, to fulfil this promise the software must remain usable, be able to accommodate a large amount of data and allow use of modern high performance computing infrastructure. In this study we present a reimplementation as well as an extension of a technique using indicator vectors to compute and visualize similarities between sets of nucleotide sequences. We have a flexible and easy to use python program relying on standard and open-source libraries. Our tool allows analysis of very large complement of sequences using code parallelization, as well as by providing routines to split a computational task in smaller and manageable subtasks whose results are then merged. This implementation also facilitates adding new sequences into an indicator vector-based representation without re-computing the whole set. The efficient synthesis of data into knowledge is no trivial matter given the size and rapid growth of biological sequence databases. Based on previous results regarding the properties of indicator vectors, the open-source approach proposed here efficiently and flexibly supports comparative analysis of genetic diversity at a large scale. Our software is freely available at: https://github.com/WandrilleD/pyKleeBarcode.

Coupled Lattice Boltzmann Modeling Framework for Pore-Scale Fluid Flow and Reactive Transport.

In this paper, we propose a modeling framework for pore-scale fluid flow and reactive transport based on a coupled lattice Boltzmann model (LBM). We develop a modeling interface to integrate the LBM modeling code parallel lattice Boltzmann solver and the PHREEQC reaction solver using multiple flow and reaction cell mapping schemes. The major advantage of the proposed workflow is the high modeling flexibility obtained by coupling the geochemical model with the LBM fluid flow model. Consequently, the model is capable of executing one or more complex reactions within desired cells while preserving the high data communication efficiency between the two codes. Meanwhile, the developed mapping mechanism enables the flow, diffusion, and reactions in complex pore-scale geometries. We validate the coupled code in a series of benchmark numerical experiments, including 2D single-phase Poiseuille flow and diffusion, 2D reactive transport with calcite dissolution, as well as surface complexation reactions. The simulation results show good agreement with analytical solutions, experimental data, and multiple other simulation codes. In addition, we design an AI-based optimization workflow and implement it on the surface complexation model to enable increased capacity of the coupled modeling framework. Compared to the manual tuning results proposed in the literature, our workflow demonstrates fast and reliable model optimization results without incorporating pre-existing domain knowledge.

The principles of building a machine-learning-based service for converting sequential code into parallel code

This article presents a novel approach for automating the parallelization of programming code using machine learning. The approach centers on a two-phase algorithm, incorporating a training phase and a transformation phase. In the training phase, a neural network is trained using data in the form of Abstract Syntax Trees, with Word2Vec being employed as the primary model for converting the syntax tree into numerical arrays. The choice of Word2Vec is attributed to its efficacy in encoding words with less reliance on context, compared to other natural language processing models such as GloVe and FastText. During the transformation phase, the trained model is applied to new sequential code, transforming it into parallel programming code. The article discusses in detail the mechanisms behind the algorithm, the rationale for the selection of Word2Vec, and the subsequent processing of code data. This methodology introduces an intelligent, automated system capable of understanding and optimizing the syntactic and semantic structures of code for parallel computing environments. The article is relevant for researchers and practitioners seeking to enhance code optimization techniques through the integration of machine learning models.

Combining stream with data parallelism abstractions for multi-cores

Stream processing applications have seen an increasing demand with the raised availability of sensors, IoT devices, and user data. Modern systems can generate millions of data items per day that require to be processed timely. To deal with this demand, application programmers must consider parallelism to exploit the maximum performance of the underlying hardware resources. In this work, we introduce improvements to stream processing applications by exploiting fine-grained data parallelism (via Map and MapReduce) inside coarse-grained stream parallelism stages. The improvements are including techniques for identifying data parallelism in sequential codes, a new language, semantic analysis, and a set of definition and transformation rules to perform source-to-source parallel code generation. Moreover, we investigate the feasibility of employing higher-level programming abstractions to support the proposed optimizations. For that, we elect SPar programming model as a use case, and extend it by adding two new attributes to its language and implementing our optimizations as a new algorithm in the SPar compiler. We conduct a set of experiments in representative stream processing and data-parallel applications. The results showed that our new compiler algorithm is efficient and that performance improved by up to 108.4x in data-parallel applications. Furthermore, experiments evaluating stream processing applications towards the composition of stream and data parallelism revealed new insights. The results showed that such composition may improve latencies by up to an order of magnitude. Also, it enables programmers to exploit different degrees of stream and data parallelism to accomplish a balance between throughput and latency according to their necessity.

Analytical performance estimation during code generation on modern GPUs

Automatic code generation is frequently used to create implementations of algorithms specifically tuned to particular hardware and application parameters. The code generation process involves the selection of adequate code transformations, tuning parameters, and parallelization strategies. We propose an alternative to time-intensive autotuning, scenario-specific performance models, or black-box machine learning to select the best-performing configuration.This paper identifies the relevant performance-defining mechanisms for memory-intensive GPU applications through a performance model coupled with an analytic hardware metric estimator. This enables a quick exploration of large configuration spaces to identify highly efficient code candidates with high accuracy.We examine the changes of the A100 GPU architecture compared to the predecessor V100 and address the challenges of how to model the data transfer volumes through the new memory hierarchy.We show how our method can be coupled to the “pystencils” stencil code generator, which is used to generate kernels for a range-four 3D-25pt stencil and a complex two-phase fluid solver based on the Lattice Boltzmann Method. For both, it delivers a ranking that can be used to select the best-performing candidate.The method is not limited to stencil kernels but can be integrated into any code generator that can generate the required address expressions.

Arbitrarily Parallelizable Code: A Model of Computation Evaluated on a Message-Passing Many-Core System

The number of processing elements per solution is growing. From embedded devices now employing (often heterogeneous) multi-core processors, across many-core scientific computing platforms, to distributed systems comprising thousands of interconnected processors, parallel programming of one form or another is now the norm. Understanding how to efficiently parallelize code, however, is still an open problem, and the difficulties are exacerbated across heterogeneous processing, and especially at run time, when it is sometimes desirable to change the parallelization strategy to meet non-functional requirements (e.g., load balancing and power consumption). In this article, we investigate the use of a programming model based on series-parallel partial orders: computations are expressed as directed graphs that expose parallelization opportunities and necessary sequencing by construction. This programming model is suitable as an intermediate representation for higher-level languages. We then describe a model of computation for such a programming model that maps such graphs into a stack-based structure more amenable to hardware processing. We describe the formal small-step semantics for this model of computation and use this formal description to show that the model can be arbitrarily parallelized, at compile and runtime, with correct execution guaranteed by design. We empirically support this claim and evaluate parallelization benefits using a prototype open-source compiler, targeting a message-passing many-core simulation. We empirically verify the correctness of arbitrary parallelization, supporting the validity of our formal semantics, analyze the distribution of operations within cores to understand the implementation impact of the paradigm, and assess execution time improvements when five micro-benchmarks are automatically and randomly parallelized across 2 × 2 and 4 × 4 multi-core configurations, resulting in execution time decrease by up to 95% in the best case.

Design and Implementation for Nonblocking Execution in GraphBLAS: Tradeoffs and Performance

GraphBLASis a recent standard that allows the expression of graph algorithms in the language of linear algebra and enables automatic code parallelization and optimization. GraphBLAS operations are memory bound and may benefit from data locality optimizations enabled by nonblocking execution. However, nonblocking execution remains under-evaluated. In this article, we present a novel design and implementation that investigates nonblocking execution in GraphBLAS. Lazy evaluation enables runtime optimizations that improve data locality, and dynamic data dependence analysis identifies operations that may reuse data in cache. The nonblocking execution of an arbitrary number of operations results in dynamic parallelism, and the performance of the nonblocking execution depends on two parameters, which are automatically determined, at run-time, based on a proposed analytic model. The evaluation confirms the importance of nonblocking execution for various matrices of three algorithms, by showing up to 4.11× speedup over blocking execution as a result of better cache utilization. The proposed analytic model makes the nonblocking execution reach up to 5.13× speedup over the blocking execution. The fully automatic performance is very close to that obtained by using the best manual configuration for both small and large matrices. Finally, the evaluation includes a comparison with other state-of-the-art frameworks for numerical linear algebra programming that employ parallel execution and similar optimizations to those discussed in this work, and the presented nonblocking execution reaches up to 16.1× speedup over the state of the art.

Simulating the Euler equations on multiple GPUs using Python

GPUs have become a household name in High Performance Computing (HPC) systems over the last 15 years. However, programming GPUs is still largely a manual and arduous task, which requires expert knowledge of the physics, mathematics, and computer science involved. Even though there have been large advances in automatic parallelization and GPU execution of serial code, it is still difficult to fully utilize the GPU hardware with such approaches. Many core numeric GPU codes are therefore still mostly written using low level C/C++ or Fortran for the host code. Several studies have shown that using higher level languages, such as Python, can make software development faster and with fewer bugs. We have developed a simulator based on PyCUDA and mpi4py in Python for solving the Euler equations on Cartesian grids. Our framework utilizes the GPU, and can automatically run on clusters using MPI as well as on shared-memory systems. Our framework allows the programmer to implement low-level details in CUDA C/C++, which is important to achieve peak performance, whilst still benefiting from the productivity of Python. We show that our framework achieves good weak and strong scaling. Our weak scaling achieves more than 94% efficiency on a shared-memory GPU system and more than 90% efficiency on a distributed-memory GPU system, and our strong scaling is close to perfect on both shared-memory and distributed-memory GPU systems.

An efficient hardware supported and parallelization architecture for intelligent systems to overcome speculative overheads

In the last few decades, technology advancements have paved the way for the creation of intelligent and autonomous systems that utilize complex calculations which are both time-consuming and central processing unit intensive. As a consequence, parallel processing systems are gaining popularity to enhance overall computer performance. Programmers should be able to efficiently utilize available hardware resources with parallelization in an ideal world. Through the automatic parallelization of sequential code, multithreading can be executed without extra supervision. However, a wide range of software dependencies prevents this from being feasible. An architectural framework for speculative parallelization along with an efficient memory analysis and computational algorithms for the code generation are proposed that can provide optimal performance. Furthermore, a suitable support of hardware design as a runtime library to the proposed architectural framework is presented which can be used to recover misspeculated results during execution to minimize speculative parallelism overhead. The implementation makes use of the Low-Level Virtual Machine compiler infrastructure and is tested on numerous benchmarks, thus making it highly scalable in terms of programming languages and architectures. According to our experimental results, there is significant potential for speedup increase. In comparison to the overall function speedup, that is, geomean speedup of 5.2× approximately when using the proposed architecture without hardware support, the proposed architectural framework and algorithm with hardware support give an average geomean speedup of 7.0× approximately on the given benchmark which is written in C/C++.

MapReduce accelerated attribute reduction based on neighborhood entropy with Apache Spark

Attribute reduction is nowadays an extremely important data preprocessing technique in the field of data mining, which has gained much attention due to its ability to provide better generalization performance and learning speed for analysis model. Rough set theory offers a systematic powerful framework for attribute reduction in terms of classificatory and decision abilities under uncertainty. In this paper, we present a parallel neighborhood entropy-based attribute reduction method with neighborhood rough sets that uses the Apache Spark cluster computing model to realize the parallelization of algorithm in a distributed computing environment. In leveraging the horizontal partitioning strategy to alleviate the task of data parallelism, three quantitative measures of attribute sets, i.e., neighborhood approximation accuracy, neighborhood credibility and coverage degrees are parallelized to accelerate the computation of decision neighborhood entropy during the heuristic search iterative process. A novel parallel heuristic attribute reduction algorithm is then developed by employing several operations from Spark API to ease the code parallelization. Extensive experimental results indicate the superiority and notable strengths of the proposed algorithm in terms of the criteria for evaluating parallel performance, i.e., scalability and extensibility.

From C/C++ Code to High-Performance Dataflow Circuits

High-level synthesis (HLS) tools typically generate statically scheduled datapaths. Static scheduling implies that the resulting circuits have a hard time exploiting parallelism in code with potential memory dependences, with control dependences, or where performance is limited by long latency control decisions. In this work, we describe an HLS approach which generates dynamically scheduled, dataflow circuits out of imperative code. We detail a complete set of rules to transform a standard compiler intermediate representation into a high-performance dataflow circuit that is able to dynamically resolve memory dependences and adapt its behavior on the fly to particular control flow decisions and operation latencies. Compared to a traditional HLS tool, the result is a different tradeoff between performance and circuit complexity: statically scheduled circuits display the best performance per cost in regular applications, but general-purpose, irregular, and control-dominated computing tasks require the runtime flexibility of dynamic scheduling. Therefore, enabling dynamic behavior in HLS is key to dealing with the increasing computational demands of new contexts and broader application domains.

SPACE: 3D parallel solvers for Vlasov-Maxwell and Vlasov-Poisson equations for relativistic plasmas with atomic transformations

A parallel, relativistic, three-dimensional particle-in-cell code SPACE has been developed for the simulation of electromagnetic fields, relativistic particle beams, and plasmas. In addition to the standard second-order Particle-in-Cell (PIC) algorithm, SPACE includes efficient novel algorithms to resolve atomic physics processes such as multi-level ionization of plasma atoms, recombination, and electron attachment to dopants in dense neutral gases. SPACE also contains a highly adaptive particle-based method, called Adaptive Particle-in-Cloud (AP-Cloud), for solving the Vlasov-Poisson problems. It eliminates the traditional Cartesian mesh of PIC and replaces it with an adaptive octree data structure. The code's algorithms, structure, capabilities, parallelization strategy, and performance have been discussed. Typical examples of SPACE applications to accelerator science and engineering problems are described.