GPGPU Research Articles

OpenCL-accelerated first-principles calculations of all-electron quantum perturbations on HPC resources.

We have proposed, for the first time, an OpenCL implementation for the all-electron density-functional perturbation theory (DFPT) calculations in FHI-aims, which can effectively compute all its time-consuming simulation stages, i.e., the real-space integration of the response density, the Poisson solver for the calculation of the electrostatic potential, and the response Hamiltonian matrix, by utilizing various heterogeneous accelerators. Furthermore, to fully exploit the massively parallel computing capabilities, we have performed a series of general-purpose graphics processing unit (GPGPU)-targeted optimizations that significantly improved the execution efficiency by reducing register requirements, branch divergence, and memory transactions. Evaluations on the Sugon supercomputer have shown that notable speedups can be achieved across various materials.

GANBISS: a new GPU accelerated N-body code for binary star systems

We present a GPU accelerated N-body integrator using the Bulirsch–Stoer method, called GANBISS (GPU accelerated n-body code for binary star systems). It is designed to simulate the dynamical evolution of planetesimal disks in binary star systems which contain some thousand disk objects. However, it can also be used for studies of non-interacting massless bodies where up to 50 million objects can be studied in a simulation. GANBISS shows the energy and angular momentum conservation behavior of non-symplectic integration methods. The code is written in CUDA C and can be run on NVIDIA GPUs of compute capability of at least 3.5. A comparison of GPU and CPU computations indicates a speed-up of the GPU performance of up to 100 times—depending on the number of disk objects.

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

Signature and Log-Signature for the Study of Empirical Distributions Generated with GANs

In this paper, we address the research gap in efficiently assessing Generative Adversarial Network (GAN) convergence and goodness of fit by introducing the application of the Signature Transform to measure similarity between image distributions. Specifically, we propose the novel use of Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) Signature, along with Log-Signature, as alternatives to existing methods such as Fréchet Inception Distance (FID) and Multi-Scale Structural Similarity Index Measure (MS-SSIM). Our approach offers advantages in terms of efficiency and effectiveness, providing a comprehensive understanding and extensive evaluations of GAN convergence and goodness of fit. Furthermore, we present innovative analytical measures based on statistics by means of Kruskal–Wallis to evaluate the goodness of fit of GAN sample distributions. Unlike existing GAN measures, which are based on deep neural networks and require extensive GPU computations, our approach significantly reduces computation time and is performed on the CPU while maintaining the same level of accuracy. Our results demonstrate the effectiveness of the proposed method in capturing the intrinsic structure of the generated samples, providing meaningful insights into GAN performance. Lastly, we evaluate our approach qualitatively using Principal Component Analysis (PCA) and adaptive t-Distributed Stochastic Neighbor Embedding (t-SNE) for data visualization, illustrating the plausibility of our method.

High-performance GPU computing of phase-field lattice Boltzmann simulations for dendrite growth with natural convection

The effect of natural convection on dendrite morphology is investigated through three-dimensional large-scale phase-field lattice Boltzmann simulations using a block-structured adaptive mesh refinement scheme with the mother-leaf method in a parallel-GPU environment. The simulations confirmed that downward buoyancy enhances the growth of the primary and secondary arms, and upward buoyancy delays the growth of those arms. In addition, the effect of natural convection on the solidification morphologies gradually decreased as the primary arm tip reached the top of the computational domain and finally stopped. Furthermore, in the longer simulation under purely isothermal diffusive conditions, detachment of the secondary arms owing to curvature-driven fragmentation was observed. A large-scale non-isothermal dendrite growth simulation was also conducted, wherein it was observed that the tip growth rate of the primary arm was delayed, and the secondary arm spacing was larger than that in the isothermal condition.

Efficient parallelization for 3D-3V sparse grid Particle-In-Cell: Single GPU architectures

In the present paper, an efficient General Purpose Graphical Processing Unit (GPGPU)-based implementation of sparse grid Particle-In-Cell (PIC) methods is proposed. The parallelization, implementing novel strategies specific to sparse-PIC methods and tailored to GPU architectures, provides speed-ups1 as large as 100 on a single Tesla V100 GPU, with respect to sequential Computing Processing unit (CPU) execution; and a four order of magnitude reduction of the computational time in comparison with a standard PIC sequential CPU simulation. In addition, the simple implementation of the parallelization with the OpenACC framework offers portability to a large class of accelerators.

Architectural Design Model Guided On-Demand Power Management of Energy-Efficient GPGPU for SLAM

Simultaneously localization and mapping (SLAM) is a core component in many embedded domains, e.g., robots, augmented and virtual reality. Due to SLAM’s high demand on computation resources, general-purpose graphic processing units (GPGPUs) are often used as its processing engine. Meanwhile, embedded systems usually have strict power constraint. Thus, how to deliver required performance for SLAM, yet still meet the power limit, is a great challenge faced by GPGPU designer. In this work, we discover the general principles of designing energy-efficient GPGPU for SLAM as “many SMs, enough SPs and registers, small caches”, by analyzing the implication of individual design parameters on both performance and power. Then, we conduct large-scale design space exploration and fit the Pareto frontier with a two-term exponential model. Further, we construct gradient boosting decision tree (GBDT)-based design models to predict the performance and power given the design parameters. The evaluation shows that our GBDT-based models can achieve [Formula: see text]3% mean average percentage error, which significantly outperform other machine learning models. With these models, a kernel’s requirement on hardware resources can be well understood. Based on such knowledge, we introduce design model guided power management strategies, including power gating and dynamic frequency and voltage scaling (DFVS). Overall, by combining these two power management strategies, we can improve the energy delay product by 36%.

Numerical Simulation of Diversion Tunnel Excavation Using GPU-Accelerated Anisotropic Random Fields

In this study, a combined anisotropic random field and GPU-accelerated Cholesky decomposition algorithm is proposed for diversion tunnel excavation. With the MATLAB programming control be used to combine the advantages of GPU and CPU computing, the efficient method of generating large numerical model random fields is realized. Based on the geological structure characteristics of red-layered soft rocks in central Yunnan, the anisotropic rock random field and tunnel excavation with different rotation angles are simulated. The results show that the variability of rotational anisotropy random fields and mechanical parameters of the red-bedded soft rocks in central Yunnan has a large influence on the deformation and stress state of the surrounding rocks at the excavation face of the tunnel. This study can provide theoretical and technical support for the design and construction of relevant rock engineering in the red-layered soft rock area in central Yunnan.

Highly-parallelized simulation of a pixelated LArTPC on a GPU

The rapid development of general-purpose computing on graphics processing units (GPGPU) is allowing the implementation of highly-parallelized Monte Carlo simulation chains for particle physics experiments. This technique is particularly suitable for the simulation of a pixelated charge readout for time projection chambers, given the large number of channels that this technology employs. Here we present the first implementation of a full microphysical simulator of a liquid argon time projection chamber (LArTPC) equipped with light readout and pixelated charge readout, developed for the DUNE Near Detector. The software is implemented with an end-to-end set of GPU-optimized algorithms. The algorithms have been written in Python and translated into CUDA kernels using Numba, a just-in-time compiler for a subset of Python and NumPy instructions. The GPU implementation achieves a speed up of four orders of magnitude compared with the equivalent CPU version. The simulation of the current induced on 10^3 pixels takes around 1 ms on the GPU, compared with approximately 10 s on the CPU. The results of the simulation are compared against data from a pixel-readout LArTPC prototype.

Techniques, Tricks, and Algorithms for Efficient GPU-Based Processing of Higher Order Hyperbolic PDEs

GPU computing is expected to play an integral part in all modern Exascale supercomputers. It is also expected that higher order Godunov schemes will make up about a significant fraction of the application mix on such supercomputers. It is, therefore, very important to prepare the community of users of higher order schemes for hyperbolic PDEs for this emerging opportunity. We focus on three broad and high-impact areas where higher order Godunov schemes are used. The first area is computational fluid dynamics (CFD). The second is computational magnetohydrodynamics (MHD) which has an involution constraint that has to be mimetically preserved. The third is computational electrodynamics (CED) which has involution constraints and also extremely stiff source terms. Together, these three diverse uses of higher order Godunov methodology, cover many of the most important applications areas. In all three cases, we show that the optimal use of algorithms, techniques and tricks, along with the use of OpenACC, yields superlative speedups on GPUs! As a bonus, we find a most remarkable and desirable result: some higher order schemes, with their larger operations count per zone, show better speedup than lower order schemes on GPUs. In other words, the GPU is an optimal stratagem for overcoming the higher computational complexities of higher order schemes! Several avenues for future improvement have also been identified. A scalability study is presented for a real-world application using GPUs and comparable numbers of high-end multicore CPUs. It is found that GPUs offer a substantial performance benefit over comparable number of CPUs, especially when all the methods designed in this paper are used.

A GPU-Accelerated Two-Dimensional Hydrodynamic Model for Unstructured Grids

The precision of numerical overland flow models is limited by their computational cost. A GPU-accelerated 2D shallow flow model is developed to overcome this challenge in this study. The model employs a Godunov-type finite volume method (FVM) to solve shallow water equations (SWEs) with unstructured grids, while also considering rainfall, infiltration, bottom slope, and friction source terms. The numerical simulation demonstrates that this model has well-balanced and robust properties. In an experiment of urban rain-runoff and flood, the accuracy and stability of the model are further demonstrated. The model is programmed with CUDA, and each numerical computation term is processed in parallel to adopt multi-thread GPU acceleration technology. With the GPU computation framework, this model can achieve a speeding up ration around 75 to single-thread CPU in the dam-break flow for a large-scale application.

GANDSE: Generative Adversarial Network-based Design Space Exploration for Neural Network Accelerator Design

With the popularity of deep learning, the hardware implementation platform of deep learning has received increasing interest. Unlike the general purpose devices, e.g., CPU or GPU, where the deep learning algorithms are executed at the software level, neural network hardware accelerators directly execute the algorithms to achieve higher energy efficiency and performance improvements. However, as the deep learning algorithms evolve frequently, the engineering effort and cost of designing the hardware accelerators are greatly increased. To improve the design quality while saving the cost, design automation for neural network accelerators was proposed, where design space exploration algorithms are used to automatically search the optimized accelerator design within a design space. Nevertheless, the increasing complexity of the neural network accelerators brings the increasing dimensions to the design space. As a result, the previous design space exploration algorithms are no longer effective enough to find an optimized design. In this work, we propose a neural network accelerator design automation framework named GANDSE, where we rethink the problem of design space exploration, and propose a novel approach based on the generative adversarial network (GAN) to support an optimized exploration for high-dimension large design space. The experiments show that GANDSE is able to find the more optimized designs in negligible time compared with approaches including multilayer perceptron and deep reinforcement learning.

DycSe: A Low-Power, Dynamic Reconfiguration Column Streaming-Based Convolution Engine for Resource-Aware Edge AI Accelerators

Edge AI accelerators are utilized to accelerate the computation in edge AI devices such as image recognition sensors on robotics, door lockers, drones, and remote sensing satellites. Instead of using a general-purpose processor (GPP) or graphic processing unit (GPU), an edge AI accelerator brings a customized design to meet the requirements of the edge environment. The requirements include real-time processing, low-power consumption, and resource-awareness, including resources on field programmable gate array (FPGA) or limited application-specific integrated circuit (ASIC) area. The system’s reliability (e.g., permanent fault tolerance) is essential if the devices target radiation fields such as space and nuclear power stations. This paper proposes a dynamic reconfigurable column streaming-based convolution engine (DycSe) with programmable adder modules for low-power and resource-aware edge AI accelerators to meet the requirements. The proposed DycSe design does not target the FPGA platform only. Instead, it is an intellectual property (IP) core design. The FPGA platform used in this paper is for prototyping the design evaluation. This paper uses the Vivado synthesis tool to evaluate the power consumption and resource usage of DycSe. Since the synthesis tool is limited to giving the final complete system result in the designing stage, we compare DycSe to a commercial edge AI accelerator for cross-reference with other state-of-the-art works. The commercial architecture shares the competitive performance within the low-power ultra-small (LPUS) edge AI scopes. The result shows that DycSe contains 3.56% less power consumption and slight resources (1%) overhead with reconfigurable flexibility.

Porting the WAVEWATCH III (v6.07) wave action source terms to GPU

Abstract. Surface gravity waves play a critical role in several processes, including mixing, coastal inundation, and surface fluxes. Despite the growing literature on the importance of ocean surface waves, wind–wave processes have traditionally been excluded from Earth system models (ESMs) due to the high computational costs of running spectral wave models. The development of the Next Generation Ocean Model for the DOE’s (Department of Energy) E3SM (Energy Exascale Earth System Model) Project partly focuses on the inclusion of a wave model, WAVEWATCH III (WW3), into E3SM. WW3, which was originally developed for operational wave forecasting, needs to be computationally less expensive before it can be integrated into ESMs. To accomplish this, we take advantage of heterogeneous architectures at DOE leadership computing facilities and the increasing computing power of general-purpose graphics processing units (GPUs). This paper identifies the wave action source terms, W3SRCEMD, as the most computationally intensive module in WW3 and then accelerates them via GPU. Our experiments on two computing platforms, Kodiak (P100 GPU and Intel(R) Xeon(R) central processing unit, CPU, E5-2695 v4) and Summit (V100 GPU and IBM POWER9 CPU) show respective average speedups of 2× and 4× when mapping one Message Passing Interface (MPI) per GPU. An average speedup of 1.4× was achieved using all 42 CPU cores and 6 GPUs on a Summit node (with 7 MPI ranks per GPU). However, the GPU speedup over the 42 CPU cores remains relatively unchanged (∼ 1.3×) even when using 4 MPI ranks per GPU (24 ranks in total) and 3 MPI ranks per GPU (18 ranks in total). This corresponds to a 35 %–40 % decrease in both simulation time and usage of resources. Due to too many local scalars and arrays in the W3SRCEMD subroutine and the huge WW3 memory requirement, GPU performance is currently limited by the data transfer bandwidth between the CPU and the GPU. Ideally, OpenACC routine directives could be used to further improve performance. However, W3SRCEMD would require significant code refactoring to make this possible. We also discuss how the trade-off between the occupancy, register, and latency affects the GPU performance of WW3.

High-performance computation toward large-scale underwater acoustics modelling

The complex nature of the ocean has long presented a challenge for researchers in the field of oceanography, including in the area of underwater acoustics. As a result, significant efforts have been made to develop accurate numerical models to better understand and study the ocean. Established models such as Acoustic Toolbox and Range-dependent Acoustic Models have proven to be effective for modelling sound propagation. However, these models were designed to run on single-core computers, and there is potential to optimise their performance on modern systems. The use of General-purpose graphics processing units (GPGPUs) and Single Instruction/Multiple Data (SIMD) computing units is one way to achieve this optimization. In addition to improving the computational speed of established models, these optimized models can also contribute to implementing more sophisticated modelling approaches, such as broad-band modelling. In this study, we will discuss the optimized performance of established codes, particularly those based on parabolic equation methods (PE), and also discuss the results of advanced modelling techniques.

SE-CNN: Convolution Neural Network Acceleration via Symbolic Value Prediction

CNNs are difficult to achieve inter-layer parallelism because of the data dependence between layers. In the paper, we propose Symbolic-Execution CNN (SE-CNN), which breaks data dependence between CNN layers via value prediction. Our insight is that in post-trained CNNs, only a subset (less than 10%) of neurons are activated (producing non-zero values) to identify patterns in inputs while most of the other neurons remain silent (producing zeros). This is because given an input image there are only a limited number of features presented. Thus, within the CNN, the neurons that are sensitive to the given features are more exercised than the others. Based on this insight, SE-CNN works in two successive phases: A parallel computation phase and a serial correction phase. In the parallel computation phase, each CNN layer starts computation simultaneously based on predicted inputs: we predict most of the neurons having zeros as inputs. For non-zero input neurons we predict their inputs for the next input image the same as the previous ones. In the serial correction phase, each layer compares the predicted inputs with the real ones to correct its computation results if necessary. If a neuron has predicted correctly its input during the parallel phase, thus the corresponding neuron passes its serial phase. Otherwise the neuron will amend its prediction with a light-weight result amendment mechanism based on the real inputs. We implement SE-CNN on top of the streaming processor of a state-of-the-art general purpose GPU (GPGPU) architecture, adding marginal hardware overheads in area and power consumption. We also provide application programming interfaces (APIs) so that CNNs that have already been implemented can directly enjoy the benefits of our technique. We utilize GPGPU-sim as our experimental platform, benchmarked with 9 well-accepted CNNs from recent years’ ILSVRC contests. Experimental results show that compared to other three state-of-the-art GPU based CNN acceleration mechanisms, SE-CNN can averagely achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$13.4\times $ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$10.4\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$7.9\times $ </tex-math></inline-formula> (maximally <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$22.0\times $ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$18.7\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$16.4\times $ </tex-math></inline-formula> ) CNN execution speedup while maintaining over 95% of classification accuracy.

Software-hardware co-design for accelerating large-scale graph convolutional network inference on FPGA

Inspired by convolutional neural networks, graph convolutional networks (GCNs) have been proposed for processing non-Euclidean graph data and successfully been applied in recommendation systems, smart traffic, etc. However, subject to the sparsity and irregularity of GCN models, the complex execution pattern of large-scale GCN poses huge challenges to the efficient inference on general purpose CPUs and GPUs, such as workload imbalance and irregular memory access. Therefore, we propose a software-hardware co-design framework for low-latency GCN inference on field programmable gate array. Specifically, at the algorithm level, we propose an attention-mechanism-based graph sparsification approach to reduce the redundant relation in the graph structure and alleviate irregularity without losing accuracy. Then, at the hardware design level, based on the sparsified graph, we propose a two-stage hardware architecture that supports the two phases with a distinct execution mode in the GCN. In order to achieve low-latency computation, edge-level and feature-level parallelism are exploited in the aggregation phase. In addition, a graph partition strategy is exploited to efficiently improve data reuse. The experimental results demonstrate that our proposed framework can achieve 739× speedup compared to CPU, 13.7× speedup compared to GPU on average and 6.8× speedup compared to state-of-the-art accelerators.

Experimental and LBM analysis of medium-Reynolds number fluid flow around NACA0012 airfoil

PurposeThe purpose of this paper is the examination of fluid flow around NACA0012 airfoil, with the aim of the numerical validation between the experimental results in the wind tunnel and the Lattice Boltzmann method (LBM) analysis, for the medium Reynolds number (Re = 191,000). The LBM–large Eddy simulation (LES) method described in this paper opens up opportunities for faster computational fluid dynamics (CFD) analysis, because of the LBM scalability on high performance computing architectures, more specifically general purpose graphics processing units (GPGPUs), pertaining at the same time the high resolution LES approach.Design/methodology/approachProcess starts with data collection in open-circuit wind tunnel experiment. Furthermore, the pressure coefficient, as a comparative variable, has been used with varying angle of attack (2°, 4°, 6° and 8°) for both experiment and LBM analysis. To numerically reproduce the experimental results, the LBM coupled with the LES turbulence model, the generalized wall function (GWF) and the cumulant collision operator with D3Q27 velocity set has been used. Also, a mesh independence study has been provided to ensure result congruence.FindingsThe proposed LBM methodology is capable of highly accurate predictions when compared with experimental data. Besides, the special significance of this work is the possibility of experimental and CFD comparison for the same domain dimensions.Originality/valueConsidering the quality of results, root-mean-square error (RMSE) shows good correlations both for airfoil’s upper and lower surface. More precisely, maximal RMSE for the upper surface is 0.105, whereas 0.089 for the lower surface, regarding all angles of attack.

Evaluation of Portable Programming Models to Accelerate LArTPC Detector Simulations

The Liquid Argon Time Projection Chamber (LArTPC) technology is widely used in high energy physics experiments, including the upcoming Deep Underground Neutrino Experiment (DUNE). Accurately simulating LArTPC detector responses is essential for analysis algorithm development and physics model interpretations. Accurate LArTPC detector response simulations are computationally demanding, and can become a bottleneck in the analysis workflow. Compute devices such as General-Purpose Graphics Processing Units (GPGPUs) have the potential to substantially accelerate simulations compared to traditional CPU-only processing. The software development for these compute accelerators often carries the cost of specialized code refactorization and porting to match the target hardware architecture. With the rapid evolution and increased diversity of the computer architecture landscape, it is highly desirable to have a portable solution that also maintains reasonable performance. We report our ongoing effort in evaluating Kokkos as a basis for this portable programming model using LArTPC simulations in the context of the Wire-Cell Toolkit, a C++ library for LArTPC simulations, data analysis, reconstruction and visualization.

High-throughput binding free energy simulations: Applications in drug discovery.

Computational drug discovery methods are becoming more and more prevalent in the drug discovery pipeline. Binding free energy (BFE) simulations are a class of molecular dynamics-based free energy methods that utilize a nonphysical “alchemical” thermodynamic pathway to efficiently and rigorously compute protein-ligand complex binding free energies. With the advent of GPU computing, BFE simulations can be performed on a timescale reasonable for small molecule drug discovery applications. Combining these methods with machine learning methods such as active learning, one can rapidly explore large chemical spaces with a fraction of the cost of brute force simulations. Herein, we describe a framework for the high-throughput and highly automated screening of a large number of chemically diverse small molecule ligands for the LRRK2 WDR domain using BFE calculations with thermodynamic integration. LRRK2 is the most commonly mutated gene in familial Parkinson's Disease, and its WDR domain is an understudied drug target with no known molecular inhibitors. This work was done as a submission to the Critical Assessment of Computational Hit-Finding Experiments (CACHE) Challenge. We also discuss various methods to minimize computational cost while maintaining accuracy and make recommendations for future high-throughput screening campaigns.