Multi-core Graphics Processing Units Research Articles

Many-core algorithms for high-dimensional gradients on phylogenetic trees.

Advancements in high-throughput genomic sequencing are delivering genomic pathogen data at an unprecedented rate, positioning statistical phylogenetics as a critical tool to monitor infectious diseases globally. This rapid growth spurs the need for efficient inference techniques, such as Hamiltonian Monte Carlo (HMC) in a Bayesian framework, to estimate parameters of these phylogenetic models where the dimensions of the parameters increase with the number of sequences N. HMC requires repeated calculation of the gradient of the data log-likelihood with respect to (wrt) all branch-length-specific (BLS) parameters that traditionally takes O(N2) operations using the standard pruning algorithm. A recent study proposes an approach to calculate this gradient in O(N), enabling researchers to take advantage of gradient-based samplers such as HMC. The CPU implementation of this approach makes the calculation of the gradient computationally tractable for nucleotide-based models but falls short in performance for larger state-space size models, such as Markov-modulated and codon models. Here, we describe novel massively parallel algorithms to calculate the gradient of the log-likelihood wrt all BLS parameters that take advantage of graphics processing units (GPUs) and result in many fold higher speedups over previous CPU implementations. We benchmark these GPU algorithms on three computing systems using three evolutionary inference examples exploring complete genomes from 997 dengue viruses, 62 carnivore mitochondria and 49 yeasts, and observe a >128-fold speedup over the CPU implementation for codon-based models and >8-fold speedup for nucleotide-based models. As a practical demonstration, we also estimate the timing of the first introduction of West Nile virus into the continental Unites States under a codon model with a relaxed molecular clock from 104 full viral genomes, an inference task previously intractable. We provide an implementation of our GPU algorithms in BEAGLE v4.0.0 (https://github.com/beagle-dev/beagle-lib), an open-source library for statistical phylogenetics that enables parallel calculations on multi-core CPUs and GPUs. We employ a BEAGLE-implementation using the Bayesian phylogenetics framework BEAST (https://github.com/beast-dev/beast-mcmc).

Distributed Multi-GPU Ab Initio Density Matrix Renormalization Group Algorithm with Applications to the P-Cluster of Nitrogenase.

The presence of many degenerate d/f orbitals makes polynuclear transition-metal compounds, such as iron-sulfur clusters in nitrogenase, challenging for state-of-the-art quantum chemistry methods. To address this challenge, we present the first distributed multi-graphics processing unit (GPU) ab initio density matrix renormalization group (DMRG) algorithm suitable for modern high-performance computing (HPC) infrastructures. The central idea is to parallelize the most computationally intensive part─the multiplication of O(K2) operators with a trial wave function, where K is the number of spatial orbitals, by combining operator parallelism for distributing the workload with a batched algorithm for performing contractions on GPU. With this new implementation, we are able to reach an unprecedentedly large bond dimension D = 14,000 on 48 GPUs (NVIDIA A100 80 GB SXM) for an active space model (114 electrons in 73 active orbitals) of the P-cluster, which is nearly 3 times larger than the bond dimensions reported in previous DMRG calculations for the same system using only central processing units (CPUs).

Graph Neural Network-based surrogate model for granular flows

Accurate simulation of granular flow dynamics is crucial for assessing geotechnical risks, including landslides and debris flows. Traditional numerical methods are limited by their computational cost in simulating large-scale systems. Statistical or machine learning-based models offer alternatives. Still, they are largely empirical, based on limited parameters. Due to their permutation-dependent learning, traditional machine learning-based models require huge training data to generalize. To resolve these problems, we use a graph neural network (GNN), a state-of-the-art machine learning architecture that learns local interactions. Graphs represent the state of dynamically changing granular flows and their interactions. We implement a multi-Graphics Processing Units (GPU) GNN simulator (GNS) capable of handling different material types. We demonstrate the capability of GNS by modeling granular flow interactions with barriers. GNS takes the granular flow’s current state and predicts the next state using Euler explicit integration by learning the local interaction laws. We train GNS on different granular trajectories. We then assess its performance by predicting granular column collapse and interaction with barriers. GNS accurately predicts flow dynamics for column collapses with different aspect ratios and interaction with barriers with configurations unseen during training. GNS is up to a few thousand times faster than high-fidelity numerical simulators.

Toward an extreme-scale electronic structure system.

Electronic structure calculations have the potential to predict key matter transformations for applications of strategic technological importance, from drug discovery to material science and catalysis. However, a predictive physicochemical characterization of these processes often requires accurate quantum chemical modeling of complex molecular systems with hundreds to thousands of atoms. Due to the computationally demanding nature of electronic structure calculations and the complexity of modern high-performance computing hardware, quantum chemistry software has historically failed to operate at such large molecular scales with accuracy and speed that are useful in practice. In this paper, novel algorithms and software are presented that enable extreme-scale quantum chemistry capabilities with particular emphasis on exascale calculations. This includes the development and application of the multi-Graphics Processing Unit (GPU) library LibCChem 2.0 as part of the General Atomic and Molecular Electronic Structure System package and of the standalone Extreme-scale Electronic Structure System (EXESS), designed from the ground up for scaling on thousands of GPUs to perform high-performance accurate quantum chemistry calculations at unprecedented speed and molecular scales. Among various results, we report that the EXESS implementation enables Hartree-Fock/cc-pVDZ plus RI-MP2/cc-pVDZ/cc-pVDZ-RIFIT calculations on an ionic liquid system with 623 016 electrons and 146 592 atoms in less than 45minusing 27 600 GPUs on the Summit supercomputer with a 94.6% parallel efficiency.

Kohn–Sham time-dependent density functional theory with Tamm–Dancoff approximation on massively parallel GPUs

We report a high-performance multi graphics processing unit (GPU) implementation of the Kohn–Sham time-dependent density functional theory (TDDFT) within the Tamm–Dancoff approximation. Our algorithm on massively parallel computing systems using multiple parallel models in tandem scales optimally with material size, considerably reducing the computational wall time. A benchmark TDDFT study was performed on a green fluorescent protein complex composed of 4353 atoms with 40,518 atomic orbitals represented by Gaussian-type functions, demonstrating the effect of distant protein residues on the excitation. As the largest molecule attempted to date to the best of our knowledge, the proposed strategy demonstrated reasonably high efficiencies up to 256 GPUs on a custom-built state-of-the-art GPU computing system with Nvidia A100 GPUs. We believe that our GPU-oriented algorithms, which empower first-principles simulation for very large-scale applications, may render deeper understanding of the molecular basis of material behaviors, eventually revealing new possibilities for breakthrough designs on new material systems.

A GPU-accelerated sharp interface immersed boundary method for versatile geometries

We present a Graphical Processing Unit (GPU) accelerated sharp interface Immersed Boundary (IB) method that can be applied to versatile geometries on a staggered Cartesian grid. The current IB solver predicts the flow around arbitrary surfaces of both finite and negligible thicknesses with improved accuracy near the sharp edges. The proposed methodology first uses a modified signed distance algorithm to track the complex geometries on the structured Cartesian grid accurately. Afterwards, we impose the boundary conditions by reconstructing the flow variables on the near boundary nodes in both fluid and solid domains. We have also shown a reduction of Spurious Force Oscillations (SFOs) near the moving boundaries with reduced divergence error. The accuracy of the present solver is demonstrated at low Reynolds numbers over different stationary and moving rigid geometries associated with sharp edges pertaining to several engineering applications. We have discussed the steps for GPU optimisation of the present solver. Our implementation ensures the concurrent execution of threads for the field extension-based velocity and pressure reconstruction algorithm on a GPU. More than 100x speedup is obtained on NVIDIA V100 GPU for the three-dimensional oscillating sphere simulation. It is observed that the speedup is higher for larger mesh sizes. The computational performance over both the multi-core Control Processing Units (CPUs) and NVIDIA GPUs (V100 and A100) using OpenACC is also provided for the insect flow simulation.

Multi-GPU thermal lattice Boltzmann simulations using OpenACC and MPI

We assess the performance of the hybrid Open Accelerator (OpenACC) and Message Passing Interface (MPI) approach for multi-graphics processing units (GPUs) accelerated thermal lattice Boltzmann (LB) simulation. The OpenACC accelerates computation on a single GPU, and the MPI synchronizes the information between multiple GPUs. With a single GPU, the two-dimension (2D) simulation achieved 1.93 billion lattice updates per second (GLUPS) with a grid number of 81932, and the three-dimension (3D) simulation achieved 1.04 GLUPS with a grid number of 3853, which is more than 76% of the theoretical maximum performance. On multi-GPUs, we adopt block partitioning, overlapping communications with computations, and concurrent computation to optimize parallel efficiency. We show that in the strong scaling test, using 16 GPUs, the 2D simulation achieved 30.42 GLUPS and the 3D simulation achieved 14.52 GLUPS. In the weak scaling test, the parallel efficiency remains above 99% up to 16 GPUs. Our results demonstrated that, with improved data and task management, the hybrid OpenACC and MPI technique is promising for thermal LB simulation on multi-GPUs.

Plant Disease Detection Using Deep Convolutional Neural Network

In this research, we proposed a novel 14-layered deep convolutional neural network (14-DCNN) to detect plant leaf diseases using leaf images. A new dataset was created using various open datasets. Data augmentation techniques were used to balance the individual class sizes of the dataset. Three image augmentation techniques were used: basic image manipulation (BIM), deep convolutional generative adversarial network (DCGAN) and neural style transfer (NST). The dataset consists of 147,500 images of 58 different healthy and diseased plant leaf classes and one no-leaf class. The proposed DCNN model was trained in the multi-graphics processing units (MGPUs) environment for 1000 epochs. The random search with the coarse-to-fine searching technique was used to select the most suitable hyperparameter values to improve the training performance of the proposed DCNN model. On the 8850 test images, the proposed DCNN model achieved 99.9655% overall classification accuracy, 99.7999% weighted average precision, 99.7966% weighted average recall, and 99.7968% weighted average F1 score. Additionally, the overall performance of the proposed DCNN model was better than the existing transfer learning approaches.

Smart City Construction and Management by Digital Twins and BIM Big Data in COVID-19 Scenario

With the rapid development of information technology and the spread of Corona Virus Disease 2019 (COVID-19), the government and urban managers are looking for ways to use technology to make the city smarter and safer. Intelligent transportation can play a very important role in the joint prevention. This work expects to explore the building information modeling (BIM) big data (BD) processing method of digital twins (DTs) of Smart City, thus speeding up the construction of Smart City and improve the accuracy of data processing. During construction, DTs build the same digital copy of the smart city. On this basis, BIM designs the building's keel and structure, optimizing various resources and configurations of the building. Regarding the fast data growth in smart cities, a complex data fusion and efficient learning algorithm, namely Multi- Graphics Processing Unit (GPU) , is proposed to process the multi-dimensional and complex BD based on the compositive rough set model. The Bayesian network solves the multi-label classification. Each label is regarded as a Bayesian network node. Then, the structural learning approach is adopted to learn the label Bayesian network's structure from data. On the P53-old and the P53-new datasets, the running time of Multi-GPU decreases as the number of GPUs increases, approaching the ideal linear speedup ratio. With the continuous increase of K value, the deterministic information input into the tag BN will be reduced, thus reducing the classification accuracy. When K = 3, MLBN can provide the best data analysis performance. On genbase dataset, the accuracy of MLBN is 0.982 ± 0.013. Through experiments, the BIM BD processing algorithm based on Bayesian Network Structural Learning (BNSL) helps decision-makers use complex data in smart cities efficiently.

A fine-grained parallelization of the immersed boundary method

We present new algorithms for the parallelization of Eulerian–Lagrangian interaction operations in the immersed boundary method. Our algorithms rely on two well-studied parallel primitives: key-value sort and segmented reduce. The use of these parallel primitives allows us to implement our algorithms on both graphics processing units (GPUs) and on other shared-memory architectures. We present strong and weak scaling tests on problems involving scattered points and elastic structures. Our tests show that our algorithms exhibit near-ideal scaling on both multicore CPUs and GPUs.

Heterogeneous gradient computing optimization for scalable deep neural networks

Nowadays, data processing applications based on neural networks cope with the growth in the amount of data to be processed and with the increase in both the depth and complexity of the neural networks architectures, and hence in the number of parameters to be learned. High-performance computing platforms are provided with fast computing resources, including multi-core processors and graphical processing units, to manage such computational burden of deep neural network applications. A common optimization technique is to distribute the workload between the processes deployed on the resources of the platform. This approach is known as data-parallelism. Each process, known as replica, trains its own copy of the model on a disjoint data partition. Nevertheless, the heterogeneity of the computational resources composing the platform requires to unevenly distribute the workload between the replicas according to its computational capabilities, to optimize the overall execution performance. Since the amount of data to be processed is different in each replica, the influence of the gradients computed by the replicas in the global parameter updating should be different. This work proposes a modification of the gradient computation method that considers the different speeds of the replicas, and hence, its amount of data assigned. The experimental results have been conducted on heterogeneous high-performance computing platforms for a wide range of models and datasets, showing an improvement in the final accuracy with respect to current techniques, with a comparable performance.

Accelerating Deep Learning Using Interconnect-Aware UCX Communication for MPI Collectives

Deep learning workloads on modern multi-graphics processing unit (GPU) nodes are highly dependent on intranode interconnects, such as NVLink and PCIe, for high-performance communication. In this article, we take on the challenge to design an interconnect-aware multipath GPU-to-GPU communication using unified communication X (UCX) to utilize all available bandwidth for both NVLink-based systems and those that use a mixture of NVLink and PCIe. Our proposed multipath data transfer mechanism pipelines and stripes the message across multiple intrasocket communication channels and memory regions to achieve 1.84× higher bandwidth for Open message passing interface (MPI) on NVLink-based systems and 1.23× on NVLink and PCIe systems. We then utilize this mechanism to propose a three-stage hierarchical, pipelined MPI_Allreduce design as well as a flat pipelined two-stage algorithm for two different node topologies. For large messages, our proposed algorithms achieve a high speedup when compared to other MPI implementations. We also observe significant speedup for the proposed MPI_Allreduce with Horovod + TensorFlow with a variety of deep learning models.

Calculation of Surface Offset Gathers Based on Reverse Time Migration and Its Parallel Computation with Multi-GPUs

As an important method for seismic data processing, reverse time migration (RTM) has high precision but involves high-intensity calculations. The calculation an RTM surface offset (shot–receiver distance) domain gathers provides intermediary data for an iterative calculation of migration and its velocity building. How to generate such data efficiently is of great significance to the industrial application of RTM. We propose a method for the calculation of surface offset gathers (SOGs) based on attribute migration, wherein, using migration calculations performed twice, the attribute profile of the surface offsets can be obtained, thus the image results can be sorted into offset gathers. Aiming at the problem of high-intensity computations required for RTM, we put forth a multi-graphic processing unit (GPU) calculative strategy, i.e., by distributing image computational domains to different GPUs for computation and by using the method of multi-stream calculations to conceal data transmission between GPUs. Ultimately, the computing original efficiency was higher relative to a single GPU, and more GPUs were used linearly. The test with a model showed that the attributive migration methods can correctly output SOGs, while the GPU parallel computation can effectively improve the computing efficiency. Therefore, it is of practical importance for this method to be expanded and applied in industries.

Lattice Boltzmann simulations of bubble interactions on GPU cluster

ABSTRACT A three-dimensional two-phase lattice Boltzmann model is adopted to simulate bubble interactions on a multi-graphics processing unit (GPU) cluster. Both the level of the predicted spurious velocity and Laplace’s law’s satisfaction are used to validate the numerical implementation. The deformation and the terminal velocity of a single-rising bubble are further simulated, and the predicted terminal Reynolds number is contrasted with the available measurement and prediction. Dual-bubble rising at initial in-line and off-line states are also investigated with both uniform and differential bubble radii. For two bubbles of the same size, the trailing bubble suffers a dramatic deformation after entering the wake region generated by the leading bubble. However, this phenomenon is less significant if the larger bubble is placed beneath the smaller bubble. Finally, the present numerical procedure exhibits good scalability for multi-GPU computations.

Real-time deep learning-based colorectal polyp localization on clinical video footage achievable with a wide array of hardware configurations.

Background and study aims Several computer-assisted polyp detection systems have been proposed, but they have various limitations, from utilizing outdated neural network architectures to a requirement for multi-graphics processing unit (GPU) processing, to validating on small or non-robust datasets. To address these problems, we developed a system based on a state-of-the-art convolutional neural network architecture able to detect polyps in real time on a single GPU and tested on both public datasets and full clinical examination recordings. Methods The study comprised 165 colonoscopy procedure recordings and 2678 still photos gathered retrospectively. The system was trained on 81,962 polyp frames in total and then tested on footage from 42 colonoscopies and CVC-ClinicDB, CVC-ColonDB, Hyper-Kvasir, and ETIS-Larib public datasets. Clinical videos were evaluated for polyp detection and false-positive rates whereas the public datasets were assessed for F1 score. The system was tested for runtime performance on a wide array of hardware. Results The performance on public datasets varied from an F1 score of 0.727 to 0.942. On full examination videos, it detected 94 % of the polyps found by the endoscopist with a 3 % false-positive rate and identified additional polyps that were missed during initial video assessment. The system’s runtime fits within the real-time constraints on all but one of the hardware configurations. Conclusions We have created a polyp detection system with a post-processing pipeline that works in real time on a wide array of hardware. The system does not require extensive computational power, which could help broaden the adaptation of new commercially available systems.

High Performance Graph Data Imputation on Multiple GPUs

In real applications, massive data with graph structures are often incomplete due to various restrictions. Therefore, graph data imputation algorithms have been widely used in the fields of social networks, sensor networks, and MRI to solve the graph data completion problem. To keep the data relevant, a data structure is represented by a graph-tensor, in which each matrix is the vertex value of a weighted graph. The convolutional imputation algorithm has been proposed to solve the low-rank graph-tensor completion problem that some data matrices are entirely unobserved. However, this data imputation algorithm has limited application scope because it is compute-intensive and low-performance on CPU. In this paper, we propose a scheme to perform the convolutional imputation algorithm with higher time performance on GPUs (Graphics Processing Units) by exploiting multi-core GPUs of CUDA architecture. We propose optimization strategies to achieve coalesced memory access for graph Fourier transform (GFT) computation and improve the utilization of GPU SM resources for singular value decomposition (SVD) computation. Furthermore, we design a scheme to extend the GPU-optimized implementation to multiple GPUs for large-scale computing. Experimental results show that the GPU implementation is both fast and accurate. On synthetic data of varying sizes, the GPU-optimized implementation running on a single Quadro RTX6000 GPU achieves up to 60.50× speedups over the GPU-baseline implementation. The multi-GPU implementation achieves up to 1.81× speedups on two GPUs versus the GPU-optimized implementation on a single GPU. On the ego-Facebook dataset, the GPU-optimized implementation achieves up to 77.88× speedups over the GPU-baseline implementation. Meanwhile, the GPU implementation and the CPU implementation achieve similar, low recovery errors.

Numerical investigation of flow behavior in double-rushton turbine stirred tank bioreactor

The current research focuses on the numerical simulation to scrutinize the divergent flow characteristics in a stirred tank bioreactor. The reactor is composed of a cylindrical flat-bottom tank equipped with two Rushton six-blade impellers and four baffles on the wall of the tank. The turbulent flow within the reactor is modeled with the Large Eddy Simulation model. The traditional subgrid model of Smagorinsky was used to solve turbulent small-scale structures with Smagorinsky constant C s = 0.1 The influence of the reactor components (i.e., cylindrical wall, baffles, shaft, and Rushton turbines) on the liquid-flow field has been obtained using the immersed boundary method. A highly parallelized computer code is developed to perform the simulation on a multi-core Graphics Processing Unit platform. The results are demonstrated in phase average flow velocities and turbulent statistics, with validation from the available experimental data reported in the literature.

Pore‐Scale Modeling of Drainage Displacement Patterns in Association With Geological Sequestration of CO2

AbstractWe investigate the immiscible displacement (drainage) of a wetting fluid in a porous medium by a nonwetting fluid using multi–graphics processing unit (GPU) lattice Boltzmann simulations with the aim of better understanding the pore‐scale processes involved in the geological sequestration of CO2. Correctly resolving the dynamics involved in multiphase flow in permeable media is of paramount importance for any numerical scheme. Generally, the average fluid flow is assumed to be at low Reynolds numbers Reav. Hence, by neglecting inertial effects, this immiscible displacement should be characterized by just two dimensionless numbers, namely, the capillary number Caav and the viscosity ratio, which quantify the ratio of the relevant forces, that is, the viscous and capillary forces. Our investigation reveals that inertial effects cannot be neglected in the range of typical capillary numbers associated with multiphase flow in permeable media. Even as the average Caav and Reav decrease away from the injection point, inertial effects remain important over a transient amount of time during abrupt Haines jumps, when the nonwetting phase passes from a narrow restriction to a wider pore space. The local Rel at the jump sites is orders of magnitude higher than the average Reav, with the local dynamics being decoupled from the externally imposed flow rate. Therefore, a full Navier‐Stokes solver should be used for investigating pore‐scale displacement processes. Using the Ohnesorge number to restrict the parameter selection process is essential, as this dimensionless number links Caav and Reav and reflects the thermophysical properties of a given system under investigation.

Multi-Node Multi-GPU Diffeomorphic Image Registration for Large-Scale Imaging Problems.

We present a Gauss-Newton-Krylov solver for large deformation diffeomorphic image registration. We extend the publicly available CLAIRE library to multi-node multi-graphics processing unit (GPUs) systems and introduce novel algorithmic modifications that significantly improve performance. Our contributions comprise (i) a new preconditioner for the reduced-space Gauss-Newton Hessian system, (ii) a highly-optimized multi-node multi-GPU implementation exploiting device direct communication for the main computational kernels (interpolation, high-order finite difference operators and Fast-Fourier-Transform), and (iii) a comparison with state-of-the-art CPU and GPU implementations. We solve a 2563-resolution image registration problem in five seconds on a single NVIDIA Tesla V100, with a performance speedup of 70% compared to the state-of-the-art. In our largest run, we register 20483 resolution images (25 B unknowns; approximately 152× larger than the largest problem solved in state-of-the-art GPU implementations) on 64 nodes with 256 GPUs on TACC's Longhorn system.

3D PHOVIS: 3D photoacoustic visualization studio

Photoacoustic (PA) imaging (or optoacoustic imaging) is a novel biomedical imaging method in biological and medical research. This modality performs morphological, functional, and molecular imaging with and without labels in both microscopic and deep tissue imaging domains. A variety of innovations have enhanced 3D PA imaging performance and thus has opened new opportunities in preclinical and clinical imaging. However, the 3D visualization tools for PA images remains a challenge. There are several commercially available software packages to visualize the generated 3D PA images. They are generally expensive, and their features are not optimized for 3D visualization of PA images. Here, we demonstrate a specialized 3D visualization software package, namely 3D Photoacoustic Visualization Studio (3D PHOVIS), specifically targeting photoacoustic data, image, and visualization processes. To support the research environment for visualization and fast processing, we incorporated 3D PHOVIS onto the MATLAB with graphical user interface and developed multi-core graphics processing unit modules for fast processing. The 3D PHOVIS includes following modules: (1) a mosaic volume generator, (2) a scan converter for optical scanning photoacoustic microscopy, (3) a skin profile estimator and depth encoder, (4) a multiplanar viewer with a navigation map, and (5) a volume renderer with a movie maker. This paper discusses the algorithms present in the software package and demonstrates their functions. In addition, the applicability of this software to ultrasound imaging and optical coherence tomography is also investigated. User manuals and application files for 3D PHOVIS are available for free on the website (www.boa-lab.com). Core functions of 3D PHOVIS are developed as a result of a summer class at POSTECH, “High-Performance Algorithm in CPU/GPU/DSP, and Computer Architecture.” We believe our 3D PHOVIS provides a unique tool to PA imaging researchers, expedites its growth, and attracts broad interests in a wide range of studies.