Graphics Processing Units Code Research Articles

Accelerating Lagrangian transport simulations on graphics processing units: performance optimizations of Massive-Parallel Trajectory Calculations (MPTRAC) v2.6

Abstract. Lagrangian particle dispersion models are indispensable tools for the study of atmospheric transport processes. However, Lagrangian transport simulations can become numerically expensive when large numbers of air parcels are involved. To accelerate these simulations, we made considerable efforts to port the Massive-Parallel Trajectory Calculations (MPTRAC) model to graphics processing units (GPUs). Here we discuss performance optimizations of the major bottleneck of the GPU code of MPTRAC, the advection kernel. Timeline, roofline, and memory analyses of the baseline GPU code revealed that the application is memory-bound, and performance suffers from near-random memory access patterns. By changing the data structure of the horizontal wind and vertical velocity fields of the global meteorological data driving the simulations from structure of arrays (SoAs) to array of structures (AoSs) and by introducing a sorting method for better memory alignment of the particle data, performance was greatly improved. We evaluated the performance on NVIDIA A100 GPUs of the Jülich Wizard for European Leadership Science (JUWELS) Booster module at the Jülich Supercomputing Center, Germany. For our largest test case, transport simulations with 108 particles driven by the European Centre for Medium-Range Weather Forecasts (ECMWF) ERA5 reanalysis, we found that the runtime for the full set of physics computations was reduced by 75 %, including a reduction of 85 % for the advection kernel. In addition to demonstrating the benefits of code optimization for GPUs, we show that the runtime of central processing unit (CPU-)only simulations is also improved. For our largest test case, we found a runtime reduction of 34 % for the physics computations, including a reduction of 65 % for the advection kernel. The code optimizations discussed here bring the MPTRAC model closer to applications on upcoming exascale high-performance computing systems and will also be of interest for optimizing the performance of other models using particle methods.

A GPU based accelerated solver for simulation of heat transfer during metal casting process

The metal casting process, which is one of the key drivers of the manufacturing industry, involves several physical phenomena occurring simultaneously like fluid flow, phase change, and heat transfer which affect the casting yield and quality. Casting process modeling involves numerical modeling of these phenomena on a computer. In recent decades, this has become an inevitable tool for foundry engineers to make defect-free castings. To expedite computational time graphics processing units (GPUs) are being increasingly used in the numerical modeling of heat transfer and fluid flow. Initially, in this work a CPU based implicit solver code is developed for solving the 3D unsteady energy equation including phase change numerically using finite volume method which predicts the thermal profile during solidification in the metal casting process in a completely filled mold. To address the computational bottleneck, which is identified as the linear algebraic solver based on the bi-conjugate gradient stabilized method, a GPU-based code is developed using Compute Unified Device Architecture toolkit and was implemented on the GPU. The CPU and GPU based codes are then validated against a commercial casting simulation code FLOW-3D CAST® for a simple casting part and against in-house experimental results for gravity die casting of a simple geometry. Parallel performance is analyzed for grid sizes ranging from 10 × 10 × 10 to 210 × 210 × 210 and for three time-step sizes. The performance of the GPU code based on occupancy and throughput is also investigated. The GPU code exhibits a maximum speedup of 308× compared to the CPU code for a grid size of 210 × 210 × 210 and a time-step size of 2 s.

A Simplified GPU Implementation of the Hybrid Lattice Boltzmann Model for Three-Dimensional High Rayleigh Number Flows

This paper provides an analysis of the numerical performance of a hybrid computational fluid dynamics (CFD) solver for 3D natural convection. We propose to use the lattice Boltzmann equations with the two-relaxation time approximation for the fluid flow, whereas thermodynamics is described by the macroscopic energy equation with the finite difference solution. An in-house parallel graphics processing unit (GPU) code is written in MATLAB. The execution time of every single step of the algorithm is studied. It is found that the explicit finite difference scheme is not as stable as the implicit one for high Rayleigh numbers. The most time-consuming steps are energy and collide, while stream, boundary conditions, and macroscopic parameters recovery are executed in no time, despite the grid size under consideration. GPU code is more than 30 times faster than a typical low-end central processing unit-based code. The proposed hybrid model can be used for real-time simulation of physical systems under laminar flow behavior and on mid-range segment GPUs.

Three-dimensional third-order gas-kinetic scheme on hybrid unstructured meshes for Euler and Navier–Stokes equations

In this paper, a third-order gas-kinetic scheme is developed on the three-dimensional hybrid unstructured meshes for the compressible inviscid and viscous flows. In the classical weighted essentially non-oscillatory (WENO) scheme, the high-order spatial accuracy is achieved by the non-linear combination of lower order polynomials. However, for the hybrid unstructured meshes, the procedures, including the selection of candidate stencils and calculation of linear weights, become extremely complicated, especially for three-dimensional problems. To overcome the drawbacks, a third-order WENO reconstruction is developed on the hybrid unstructured meshes, including tetrahedron, pyramid, prism and hexahedron. A general strategy is given for the selection of big stencil and sub-stencils for hybrid meshes, and the topologically independent linear weights are used in the spatial reconstruction. A unified interpolation is also used for the volume integral of different types of control volumes, as well as the flux integration over different cell interfaces. Incorporate with the two-stage fourth-order temporal discretization, the explicit high-order gas-kinetic schemes are developed for unsteady problems. With the lower–upper symmetric Gauss–Seidel (LU-SGS) methods and Jacobi iteration, the implicit high-order gas-kinetic schemes are developed for steady problems. To accelerate the computation, both explicit and implicit scheme is implemented with the graphics processing unit (GPU) using compute unified device architecture (CUDA). Especially, for implicit parallel computation, Jacobi iteration is used. The speedup of GPU code suggests the potential of high-order gas-kinetic schemes for the large scale computation. A variety of numerical examples, from the subsonic to supersonic flows, are presented to validate the accuracy and robustness for both inviscid and viscous flows.

Quantum Mechanics/Molecular Mechanics Simulations on NVIDIA and AMD Graphics Processing Units.

We have ported and optimized the graphics processing unit (GPU)-accelerated QUICK and AMBER-based ab initio quantum mechanics/molecular mechanics (QM/MM) implementation on AMD GPUs. This encompasses the entire Fock matrix build and force calculation in QUICK including one-electron integrals, two-electron repulsion integrals, exchange-correlation quadrature, and linear algebra operations. General performance improvements to the QUICK GPU code are also presented. Benchmarks carried out on NVIDIA V100 and AMD MI100 cards display similar performance on both hardware for standalone HF/DFT calculations with QUICK and QM/MM molecular dynamics simulations with QUICK/AMBER. Furthermore, with respect to the QUICK/AMBER release version 21, significant speedups are observed for QM/MM molecular dynamics simulations. This significantly increases the range of scientific problems that can be addressed with open-source QM/MM software on state-of-the-art computer hardware.

High-performance GPU-accelerated evaluation of electron repulsion integrals

ABSTRACT A novel methodology for the evaluation of two electron integrals up to f functions using Graphics Processing Units (GPUs) is presented. The Head-Gordon-Pople recursion relationships are solved via a simple heuristic methodology to minimize the number of evaluated intermediates in the recursion trees. Automatic code generation is used to generate highly optimized CUDA kernels. A novel approach for f functions is presented in which integral classes are split into smaller subclasses to minimize register pressure and exploit additional parallelism at the cost of recomputing a small number of intermediates. Alongside optimized kernels, the ERI evaluation scheme works in conjunction with an efficient work distribution scheme which guarantees load-balancing during computation. The new HGP scheme shows excellent speedups of 2 × to above 60 × against existing GPU code. Additionally, when coupled with digestion into the Fock matrix, the scaling is excellent on up to 7 GPUs with an 85% parallel efficiency for the 6-31G(d) basis set.

Esoteric Pull and Esoteric Push: Two Simple In-Place Streaming Schemes for the Lattice Boltzmann Method on GPUs

I present two novel thread-safe in-place streaming schemes for the lattice Boltzmann method (LBM) on graphics processing units (GPUs), termed Esoteric Pull and Esoteric Push, that result in the LBM only requiring one copy of the density distribution functions (DDFs) instead of two, greatly reducing memory demand. These build upon the idea of the existing Esoteric Twist scheme, to stream half of the DDFs at the end of one stream-collide kernel and the remaining half at the beginning of the next, and offer the same beneficial properties over the AA-Pattern scheme—reduced memory bandwidth due to implicit bounce-back boundaries and the possibility of swapping pointers between even and odd time steps. However, the streaming directions are chosen in a way that allows the algorithm to be implemented in about one tenth the amount of code, as two simple loops, and is compatible with all velocity sets and suitable for automatic code-generation. The performance of the new streaming schemes is slightly increased over Esoteric Twist due to better memory coalescence. Benchmarks across a large variety of GPUs and CPUs show that for most dedicated GPUs, performance differs only insignificantly from the One-Step Pull scheme; however, for integrated GPUs and CPUs, performance is significantly improved. The two proposed algorithms greatly facilitate modifying existing code to in-place streaming, even with extensions already in place, such as demonstrated here for the Free Surface LBM implementation FluidX3D. Their simplicity, together with their ideal performance characteristics, may enable more widespread adoption of in-place streaming across LBM GPU codes.

Three-dimensional discontinuous Galerkin based high-order gas-kinetic scheme and GPU implementation

In this paper, the discontinuous Galerkin based high-order gas-kinetic schemes (DG-HGKS) are developed for the three-dimensional Euler and Navier–Stokes equations. Different from the traditional discontinuous Galerkin (DG) methods with Riemann solvers, the current method adopts a kinetic evolution process, which is provided by the integral solution of Bhatnagar–Gross–Krook (BGK) model. In the weak formulation of DG method, a time-dependent evolution function is provided, and both inviscid and viscous fluxes can be calculated uniformly. The temporal accuracy is achieved by the two-stage fourth-order discretization, and the second-order gas-kinetic solver is adopted for the fluxes over the cell interface and the fluxes inside a cell. Numerical examples, including accuracy tests and Taylor–Green vortex problem, are presented to validate the efficiency and accuracy of DG-HGKS. Both optimal convergence and super-convergence are achieved by the current scheme. The comparison between DG-HGKS and high-order gas-kinetic scheme with weighted essential non-oscillatory reconstruction (WENO-HGKS) is also given, and the numerical performances are comparable with the approximate number of degree of freedom. To accelerate the computation, the DG-HGKS is implemented with the graphics processing unit (GPU) using compute unified device architecture (CUDA). The obtained results are also compared with those calculated by the central processing units (CPU) code in terms of the computational efficiency. The speedup of GPU code suggests the potential of high-order gas-kinetic schemes for the large scale computation.

Massive-Parallel Trajectory Calculations version 2.2 (MPTRAC-2.2): Lagrangian transport simulations on graphics processing units (GPUs)

Abstract. Lagrangian models are fundamental tools to study atmospheric transport processes and for practical applications such as dispersion modeling for anthropogenic and natural emission sources. However, conducting large-scale Lagrangian transport simulations with millions of air parcels or more can become rather numerically costly. In this study, we assessed the potential of exploiting graphics processing units (GPUs) to accelerate Lagrangian transport simulations. We ported the Massive-Parallel Trajectory Calculations (MPTRAC) model to GPUs using the open accelerator (OpenACC) programming model. The trajectory calculations conducted within the MPTRAC model were fully ported to GPUs, i.e., except for feeding in the meteorological input data and for extracting the particle output data, the code operates entirely on the GPU devices without frequent data transfers between CPU and GPU memory. Model verification, performance analyses, and scaling tests of the Message Passing Interface (MPI) – Open Multi-Processing (OpenMP) – OpenACC hybrid parallelization of MPTRAC were conducted on the Jülich Wizard for European Leadership Science (JUWELS) Booster supercomputer operated by the Jülich Supercomputing Centre, Germany. The JUWELS Booster comprises 3744 NVIDIA A100 Tensor Core GPUs, providing a peak performance of 71.0 PFlop s−1. As of June 2021, it is the most powerful supercomputer in Europe and listed among the most energy-efficient systems internationally. For large-scale simulations comprising 108 particles driven by the European Centre for Medium-Range Weather Forecasts' fifth-generation reanalysis (ERA5), the performance evaluation showed a maximum speed-up of a factor of 16 due to the utilization of GPUs compared to CPU-only runs on the JUWELS Booster. In the large-scale GPU run, about 67 % of the runtime is spent on the physics calculations, conducted on the GPUs. Another 15 % of the runtime is required for file I/O, mostly to read the large ERA5 data set from disk. Meteorological data preprocessing on the CPUs also requires about 15 % of the runtime. Although this study identified potential for further improvements of the GPU code, we consider the MPTRAC model ready for production runs on the JUWELS Booster in its present form. The GPU code provides a much faster time to solution than the CPU code, which is particularly relevant for near-real-time applications of a Lagrangian transport model.

An Automated Tool for Analysis and Tuning of GPU-Accelerated Code in HPC Applications

The US Department of Energy’s fastest supercomputers and forthcoming exascale systems employ Graphics Processing Units (GPUs) to increase the computational performance of compute nodes. However, the complexity of GPU architectures makes tailoring sophisticated applications to achieve high performance on GPU-accelerated systems a major challenge. At best, prior performance tools for GPU code only provide coarse-grained tuning advice at the kernel level. In this article, we describe GPA, a performance advisor that suggests potential code optimizations at a hierarchy of levels, including individual lines, loops, and functions. To gather the fine-grained measurements needed to produce such insights, GPA uses instruction sampling and binary instrumentation to monitor execution of GPU code. At the time of this writing, GPU instruction sampling is only available on NVIDIA GPUs. To understand performance losses, GPA uses data flow analysis to approximately attribute measured instruction stalls back to their causes. GPA then analyzes patterns of stalls using information about a program’s structure and the GPU architecture to identify optimization strategies that address inefficiencies observed. GPA then employs detailed performance models to estimate the potential speedup that each optimization might provide. Experiments with benchmarks and applications show that GPA provides useful advice for tuning GPU code. We applied GPA to analyze and tune a collection of codes on NVIDIA V100 and A100 GPUs. GPA suggested optimizations that it estimates will accelerate performance across the set of codes by a geometric mean of 1.21×. Applying these optimizations suggested by GPA accelerated these codes by a geometric mean of 1.19×.

Autotuning GPU code for acceleration of CGH calculation

Computer-generated hologram (CGH) has a problem in that the calculation time is huge, we use a graphics processing unit (GPU) to enable fast calculation for CGH. Optimization of GPU calculation is studied to bring out the best performance of the GPU. We propose an autotuning method for GPU code that can be applied when the scene or calculation environment changes in CGH for practical use. We created a system that finds the optimal parameters by presenting three parameters for tuning in CGH. We applied autotuning to multiple objects and different GPUs and found optimal parameters for each. In addition, it was confirmed that the calculation time was 1.1 to 1.7 times faster by the autotuning.

Faster Self-Consistent Field (SCF) Calculations on GPU Clusters.

A novel implementation of the self-consistent field (SCF) procedure specifically designed for high-performance execution on multiple graphics processing units (GPUs) is presented. The algorithm offloads to GPUs the three major computational stages of the SCF, namely, the calculation of one-electron integrals, the calculation and digestion of electron repulsion integrals, and the diagonalization of the Fock matrix, including SCF acceleration via DIIS. Performance results for a variety of test molecules and basis sets show remarkable speedups with respect to the state-of-the-art parallel GAMESS CPU code and relative to other widely used GPU codes for both single and multi-GPU execution. The new code outperforms all existing multi-GPU implementations when using eight V100 GPUs, with speedups relative to Terachem ranging from 1.2× to 3.3× and speedups of up to 28× over QUICK on one GPU and 15× using eight GPUs. Strong scaling calculations show nearly ideal scalability up to 8 GPUs while retaining high parallel efficiency for up to 18 GPUs.

Analysis of GPU Computation of Parabolic, Bessel, Wright and Riemann Zeta Functions

This paper deals with GPU computing of special mathematical functions that are used in Fractional Calculus. The graphics processing unit (GPU) has grown to be an integral part of nowadays’s mainstream computing structures. The special mathematical functions are an integral part of Fractional Calculus. This paper deals with a novel parallel approach for computing special mathematical functions used in Fractional Calculus. NVIDIA’s GPU hardware is used to speed up the parallel algorithm. A comparison of the sequential code, vectorized code and GPU code is performed. We have successfully reduced the computation time of special mathematical functions using the parallel computing capabilities of GPU.

GPU-accelerated Monte Carlo simulation of MV-CBCT

Monte Carlo simulation (MCS) is one of the most accurate computation methods for dose calculation and image formation in radiation therapy. However, the high computational complexity and long execution time of MCS limits its broad use. In this paper, we present a novel strategy to accelerate MCS using a graphic processing unit (GPU), and we demonstrate the application in mega-voltage (MV) cone-beam computed tomography (CBCT) simulation. A new framework that generates a series of MV projections from a single simulation run is designed specifically for MV-CBCT acquisition. A Geant4-based GPU code for photon simulation is incorporated into the framework for the simulation of photon transport through a phantom volume. The FastEPID method, which accelerates the simulation of MV images, is modified and integrated into the framework. The proposed GPU-based simulation strategy was tested for its accuracy and efficiency in a Catphan 604 phantom and an anthropomorphic pelvis phantom with beam energies at 2.5 MV, 6 MV, and 6 MV FFF. In all cases, the proposed GPU-based simulation demonstrated great simulation accuracy and excellent agreement with measurement and CPU-based simulation in terms of reconstructed image qualities. The MV-CBCT simulation was accelerated by factors of roughly 900–2300 using an NVIDIA Tesla V100 GPU card against a 2.5 GHz AMD Opteron™ Processor 6380.

High-Performance, Graphics Processing Unit-Accelerated Fock Build Algorithm.

We present a high-performance, GPU (graphics processing unit)-accelerated algorithm for building the Fock matrix. The algorithm is designed for efficient calculations on large molecular systems and uses a novel dynamic load balancing scheme that maximizes the GPU throughput and avoids thread divergence that could occur due to integral screening. Additionally, the code adopts a novel ERI digestion algorithm that exploits all forms of permutational symmetry, combines efficiently the evaluation of both Coulomb and exchange terms together, and eliminates explicit thread synchronization requirements. Performance results obtained using a number of large molecules reveal remarkable speedups up to 24.4× with respect to the QUICK GPU code and up to 237× with respect to the GAMESS CPU parallel code.

Evaluation of Clustering Algorithms on GPU-Based Edge Computing Platforms.

Internet of Things (IoT) is becoming a new socioeconomic revolution in which data and immediacy are the main ingredients. IoT generates large datasets on a daily basis but it is currently considered as “dark data”, i.e., data generated but never analyzed. The efficient analysis of this data is mandatory to create intelligent applications for the next generation of IoT applications that benefits society. Artificial Intelligence (AI) techniques are very well suited to identifying hidden patterns and correlations in this data deluge. In particular, clustering algorithms are of the utmost importance for performing exploratory data analysis to identify a set (a.k.a., cluster) of similar objects. Clustering algorithms are computationally heavy workloads and require to be executed on high-performance computing clusters, especially to deal with large datasets. This execution on HPC infrastructures is an energy hungry procedure with additional issues, such as high-latency communications or privacy. Edge computing is a paradigm to enable light-weight computations at the edge of the network that has been proposed recently to solve these issues. In this paper, we provide an in-depth analysis of emergent edge computing architectures that include low-power Graphics Processing Units (GPUs) to speed-up these workloads. Our analysis includes performance and power consumption figures of the latest Nvidia’s AGX Xavier to compare the energy-performance ratio of these low-cost platforms with a high-performance cloud-based counterpart version. Three different clustering algorithms (i.e., k-means, Fuzzy Minimals (FM), and Fuzzy C-Means (FCM)) are designed to be optimally executed on edge and cloud platforms, showing a speed-up factor of up to for the GPU code compared to sequential counterpart versions in the edge platforms and energy savings of up to 150% between the edge computing and HPC platforms.

Boosting Free-Energy Perturbation Calculations with GPU-Accelerated NAMD.

Harnessing the power of graphics processing units (GPUs) to accelerate molecular dynamics (MD) simulations in the context of free-energy calculations has been a longstanding effort toward the development of versatile, high-performance MD engines. We report a new GPU-based implementation in NAMD of free-energy perturbation (FEP), one of the oldest, most popular importance-sampling approaches for the determination of free-energy differences that underlie alchemical transformations. Compared to the CPU implementation available since 2001 in NAMD, our benchmarks indicate that the new implementation of FEP in traditional GPU code is about four times faster, without any noticeable loss of accuracy, thereby paving the way toward more affordable free-energy calculations on large biological objects. Moreover, we have extended this new FEP implementation to a code path highly optimized for a single-GPU node, which proves to be up to nearly 30 times faster than the CPU implementation. Through optimized GPU performance, the present developments provide the community with a cost-effective solution for conducting FEP calculations. The new FEP-enabled code has been released with NAMD 3.0.

Toward large-scale simulation of residual stress and distortion in wire and arc additive manufacturing

This study aims to advance the structural analysis of wire and arc additive manufacturing (WAAM) by considering the thermomechanical features inherent in direct energy deposition. Simulation approaches including the iterative substructure method (ISM), dynamic mesh refining method (DMRM), and graphics processing unit (GPU) based explicit finite element method (FEM) were developed for accelerating additive manufacturing stress analysis that is very time consuming by conventional numerical methods. ISM and DMRM take advantage of the strong nonlinearity phenomenon near the moving heat source by reducing the global iterations and refining the local mesh, respectively. In addition to the use of GPUs, the explicit FEM is accelerated by a time scaling technique based on the inherent strain concept. The residual stress and distortion of two large builds were analyzed, showing very consistent numerical results and good agreement with experiments. Compared with the commercial software Abaqus, the novel approaches reduced the computational cost substantially without compromising accuracy. The GPU code showed the highest computational efficiency (30∼70×), while DMRM and ISM had acceleration factors of 9× and 3×, respectively. Such high-fidelity modeling approaches will be very useful for building up a digital twin of WAAM to reduce development time and cost.

Implementation and performance analysis of the massively parallel method of characteristics based on GPU

The method of characteristics (MOC) is one of the most common methods for solving the neutron transport equation in practical application. Researches have been focused on the acceleration techniques and the parallel algorithm for improving the efficiency of MOC. The Graphics Processing Unit (GPU) provides an alternative method of parallelizing the MOC neutron transport sweep. In this work, a GPU-paralleled 2D MOC code is implemented, which employs the diamond difference (DD) scheme and the step characteristics (SC) scheme. Different parallel schemes which are ray-level, energy-group-level, and polar-angle-level, are analyzed to choose the proper parallel scheme. The C5G7 2D benchmark is calculated to verify the accuracy and efficiency of the code in different schemes with single precision and double precision. The bottlenecks of the GPU code are identified and the code is classified into three categories, which are compute-bound, memory-bound, and latency-bound, according to the performance analysis model introduced in this paper. In addition, corresponding optimization strategies are applied to improve the performance according to the analysis. Moreover, the speed, power efficiency, and hardware cost are compared for CPU and GPU based on a fictitious quarter core PWR problem. Numerical results demonstrate that the energy group-level parallelization can obtain the optimal performance on GPU. Optimization strategies are effective to improve the efficiency of the calculation on GPU, which indicates that the performance analysis model is useful and effective to locate the limitation of the code. Moreover, the GPU-version code is about 30 times faster than the CPU-version code with double precision and about 100 times faster with single precision, while the desired accuracy is kept. And the GPU delivers superior performance in both speed, energy efficiency, and hardware cost.

GPU acceleration and performance of the particle-beam-dynamics code Elegant

Elegant is an accelerator physics and particle-beam dynamics code widely used for modeling and design of a variety of high-energy particle accelerators and accelerator-based systems. In this paper we discuss a recently developed version of the code that can take advantage of CUDA-enabled graphics processing units (GPUs) to achieve significantly improved performance for a large class of simulations that are important in practice. The GPU version is largely defined by a framework that simplifies implementations of the fundamental kernel types that are used by Elegant: particle operations, reductions, particle loss, histograms, array convolutions and random number generation. Accelerated performance on the Titan Cray XK-7 supercomputer is approximately 6–10 times better with the GPU than all the CPU cores associated with the same node count. In addition to performance, the maintainability of the GPU-accelerated version of the code was considered a key design objective. Accuracy with respect to the CPU implementation is also a core consideration. Four different methods are used to ensure that the accelerated code faithfully reproduces the CPU results. Program summaryProgram Title: Kernels from the GPU-accelerated ElegantProgram Files doi:http://dx.doi.org/10.17632/jc465zy7p5.1Licensing provisions: MITProgramming language: C/C++/CUDANature of problem: The original design of the Elegant accelerator physics code was implemented on central processing units with message-passing interface parallelization. This implementation is not able to use next-generation multicore systems.Solution method: In this package we develop routines based on the CUDA language extensions to C++ that enable porting the Elegant code to be run on graphics processing units (GPUs). Special consideration is given to algorithms that require collective communication on the GPU.Additional comments including restrictions and unusual features: The full Elegant source code is freely available from Argonne National Laboratory and these distributions include the GPU code in the later releases.