CUDA Implementation Research Articles

End-to-End Deployment of Winograd-Based DNNs on Edge GPU

The Winograd algorithm reduces the computational complexity of convolutional neural networks (CNNs) by minimizing the number of multiplications required for convolutions, making it particularly suitable for resource-constrained edge devices. Concurrently, most edge hardware accelerators utilize 8-bit integer arithmetic to enhance energy efficiency and reduce inference latency, requiring the quantization of CNNs before deployment. Combining Winograd-based convolution with quantization offers the potential for both performance acceleration and reduced energy consumption. However, prior research has identified significant challenges in this combination, particularly due to numerical instability and substantial accuracy degradation caused by the transformations required in the Winograd domain, making the two techniques incompatible on edge hardware. In this work, we describe our latest training scheme, which addresses these challenges, enabling the successful integration of Winograd-accelerated convolution with low-precision quantization while maintaining high task-related accuracy. Our approach mitigates the numerical instability typically introduced during the transformation, ensuring compatibility between the two techniques. Additionally, we extend our work by presenting a custom-optimized CUDA implementation of quantized Winograd convolution for NVIDIA edge GPUs. This implementation takes full advantage of the proposed training scheme, achieving both high computational efficiency and accuracy, making it a compelling solution for edge-based AI applications. Our training approach enables significant MAC reduction with minimal impact on prediction quality. Furthermore, our hardware results demonstrate up to a 3.4× latency reduction for specific layers, and a 1.44× overall reduction in latency for the entire DeepLabV3 model, compared to the standard implementation.

CUDA acceleration of MI-based feature selection methods

Feature selection algorithms are necessary nowadays for machine learning as they are capable of removing irrelevant and redundant information to reduce the dimensionality of the data and improve the quality of subsequent analyses. The problem with current feature selection approaches is that they are computationally expensive when processing large datasets. This work presents parallel implementations for Nvidia GPUs of three highly-used feature selection methods based on the Mutual Information (MI) metric: mRMR, JMI and DISR. Publicly available code includes not only CUDA implementations of the general methods, but also an adaptation of them to work with low-precision fixed point in order to further increase their performance on GPUs. The experimental evaluation was carried out on two modern Nvidia GPUs (Turing T4 and Ampere A100) with highly satisfactory results, achieving speedups of up to 283x when compared to state-of-the-art C implementations.

Parallel Implementation of Lightweight Secure Hash Algorithm on CPU and GPU Environments

Currently, cryptographic hash functions are widely used in various applications, including message authentication codes, cryptographic random generators, digital signatures, key derivation functions, and post-quantum algorithms. Notably, they play a vital role in establishing secure communication between servers and clients. Specifically, servers often need to compute a large number of hash functions simultaneously to provide smooth services to connected clients. In this paper, we present highly optimized parallel implementations of Lightweight Secure Hash (LSH), a hash algorithm developed in Korea, on server sides. To optimize LSH performance, we leverage two parallel architectures: AVX-512 on high-end CPUs and NVIDIA GPUs. In essence, we introduce a word-level parallel processing design suitable for AVX-512 instruction sets and a data parallel processing design appropriate for the NVIDIA CUDA platform. In the former approach, we parallelize the core functions of LSH using AVX-512 registers and instructions. As a result, our first implementation achieves a performance improvement of up to 50.37% compared to the latest LSH AVX-2 implementation. In the latter approach, we optimize the core operation of LSH with CUDA PTX assembly and apply a coalesced memory access pattern. Furthermore, we determine the optimal number of blocks/threads configuration and CUDA streams for RTX 2080Ti and RTX 3090. Consequently, in the RTX 3090 architecture, our optimized CUDA implementation achieves about a 180.62% performance improvement compared with the initially ported LSH implementation to the CUDA platform. As far as we know, this is the first work on optimizing LSH with AVX-512 and NVIDIA GPU. The proposed implementation methodologies can be used alone or together in a server environment to achieve the maximum throughput of LSH computation.

An Efficient Explicit Moving Particle Simulation Solver for Simulating Free Surface Flow on Multicore CPU/GPUs

The moving particle simulation (MPS) method is a simulation technique capable of calculating free surface and incompressible flows. As a particle-based method, MPS requires significant computational resources when simulating flow in a large-scale domain with a huge number of particles. Therefore, improving computational speed is a crucial aspect of current research in particle methods. In recent decades, many-core CPUs and GPUs have been widely utilized in scientific simulations to significantly enhance computational efficiency. However, the implementation of MPS on different types of hardware is not a trivial task. In this study, we present an implementation method for the explicit MPS that utilizes the Taichi parallel programming language. When it comes to CPU computing, Taichi’s computational efficiency is comparable to that of OpenMP. Nevertheless, when GPU computing is utilized, the acceleration of Taichi in parallel computing is not as fast as the CUDA implementation. Our developed explicit MPS solver demonstrates significant performance improvements in simulating dam-break flow dynamics.

Efficient GPU implementation of randomized SVD and its applications

Matrix decompositions are ubiquitous in machine learning, including applications in dimensionality reduction, data compression and deep learning algorithms. Typical solutions for matrix decompositions have polynomial complexity which significantly increases their computational cost and time. In this work, we leverage efficient processing operations that can be run in parallel on modern Graphical Processing Units (GPUs), predominant computing architecture used e.g. in deep learning, to reduce the computational burden of computing matrix decompositions. More specifically, we reformulate the randomized decomposition problem to incorporate fast matrix multiplication operations (BLAS-3) as building blocks. We show that this formulation, combined with fast random number generators, allows to fully exploit the potential of parallel processing implemented in GPUs. Our extensive evaluation confirms the superiority of this approach over the competing methods and we release the results of this research as a part of the official CUDA implementation.11https://docs.nvidia.com/cuda/cusolver/index.html.

CunuSHT: GPU accelerated spherical harmonic transforms on arbitrary pixelizations

ABSTRACT We present cunuSHT, a general-purpose Python package that wraps a highly efficient CUDA implementation of the non-uniform spin-0 spherical harmonic transform. The method is applicable to arbitrary pixelization schemes, including schemes constructed from equally spaced iso-latitude rings as well as completely non-uniform ones. The algorithm has an asymptotic scaling of $\mathcal {O}{(\ell _{\rm max}^3)}$ for maximum multipole $\ell _{\rm max}$ and can be made to achieve machine precision accuracy, considering band-limited transforms for which $N\approx \ell _{\rm max}^2$ (where N is the number of pixels in the map). While cunuSHT is developed for applications in cosmology in mind, it is applicable to various other interpolation problems on the sphere. We outperform the fastest available CPU algorithm at problem sizes $\ell _{\rm max}\sim 4\times 10^2$ and larger. The speed-up increases with the problem size and reaches a factor of up to 5 for problems with a non-uniform pixelization and $\ell _{\rm max}\gt 4\times 10^3$ when comparing a single modern GPU to a modern 32-core CPU. This performance is achieved by utilizing the double Fourier sphere method in combination with the non-uniform fast Fourier transform and by avoiding transfers between the host and device. For scenarios without GPU availability, cunuSHT wraps existing CPU libraries. cunuSHT is publicly available and includes tests, documentation, and demonstrations.

Optimized CUDA Implementation to Improve the Performance of Bundle Adjustment Algorithm on GPUs

Optimized CUDA Implementation to Improve the Performance of Bundle Adjustment Algorithm on GPUs

High level GPU-accelerated 2D PIV framework in Python

The Particle Image Velocimetry (PIV) method is widely used for optical measurment of flow velocity fields. This paper demonstrates the possibilities of using high-level libraries for GPU-accelerated PIV data analysis in Python. The Torch PIV library for the analysis of 2D PIV experiments based on the deep learning framework PyTorch with CUDA support was developed. The library implements a multi pass cross-correlation FFT PIV algorithm with an interrogation window shift. The chosen implementation does not require compilation from the user, has a compact codebase, is able to run both on the CPU and the GPU depending on the user choice, and also it is as flexible as the Python module. In this work, the performance of the CPU version of the developed method was compared with existing open source implementations. It is shown that the main functions of the developed module can be executed on the GPU at the speed of CUDA implementations. The developed library is tested on synthetic images and experimental data. Program SummaryProgram title: TorchPIV.CPC Library link to program files: https://doi.org/10.17632/yw43vjc36h.1.Developer's repository link: https://github.com/NikNazarov/TorchPIV.Licensing provisions: MIT license.Programming language: Python.Nature of problem: PIV experiments often require analyzing a large number of images in order to determine the statistical characteristics of the flow. GPUs are actively used in this field to speed up the data analysis process. Open-source software solutions to this problem usually require build and are difficult to integrate into the analysis pipelines.Solution method: The main feature of the developed module is its flexibility and simple distribution. The module is cross-platform, and installation does not require compilation from the user. The developed library can be imported as an ordinary Python module, at the same time it allows to get a significant performance gain when analyzing PIV experiments by using NVIDIA GPUs. Implementation in pure Python allows the module to serve as a backend for more complex experimental data processing systems. The core library of this method is one of the most reliable and widespread in the field of machine learning.

An Evaluation of Directive-Based Parallelization on the GPU Using a Parboil Benchmark

Heterogeneous architectures consisting of both central processing units and graphics processing units are common in contemporary computer systems. For that reason, several programming models have been developed to exploit available parallelism, such as low-level CUDA and OpenCL, and directive-based OpenMP and OpenACC. In this paper we explore and evaluate the applicability of OpenACC, which is a directive-based programming model for GPUs. We focus both on the performance and programming effort needed to parallelize the existing sequential algorithms for GPU execution. The evaluation is based on the benchmark suite Parboil, which consists of 11 different mini-applications from different scientific domains, both compute- and memory-bound. The results show that mini-apps parallelized with OpenACC can achieve significant speedups over sequential implementations and in some cases, even outperform CUDA implementations. Furthermore, there is less of a programming effort compared to low-level models, such as CUDA and OpenCL, because a majority of the work is left to the compiler and overall, the code needs less restructuring.

Real-Time Ego-Lane Detection in a Low-Cost Embedded Platform using CUDA-Based Implementation

This work assesses the effectiveness of heterogeneous computing based on a CUDA implementation for real-time ego-lane detection using a typical low-cost embedded computer. We propose and evaluate a CUDA-optimized algorithm using a heterogeneous approach based on the extraction of features from an aerial perspective image. The method incorporates well-known algorithms optimized to achieve a very efficient solution with high detection rates and combines techniques to enhance markings and remove noise. The CUDA-based solution is compared to an OpenCV library and to a serial CPU implementation. Practical experiments using TuSimple's image datasets were conducted in an NVIDIA's Jetson Nano embedded computer. The algorithm detects up to 97.9% of the ego lanes with an accuracy of 99.0% in the best-evaluated scenario. Furthermore, the CUDA-optimized method performs at rates greater than 300 fps in the Jetson Nano embedded system, speeding up 25 and 140 times the OpenCV and CPU implementations at the same platform, respectively. These results show that more complex algorithms and solutions can be employed for better detection rates while maintaining real-time requirements in a typical low-power embedded computer using a CUDA implementation.

CATE: A fast and scalable CUDA implementation to conduct highly parallelized evolutionary tests on large scale genomic data

Abstract Statistical tests for molecular evolution provide quantifiable insights into the selection pressures that govern a genome's evolution. Increasing sample sizes used for analysis leads to higher statistical power. However, this requires more computational nodes or longer computational time. CATE (CUDA Accelerated Testing of Evolution) is a computational solution to this problem comprised of two main innovations. The first is a file organization system coupled with a novel search algorithm and the second is a large‐scale parallelization of algorithms using both graphical processing unit (GPU) and central processing unit. CATE is capable of conducting evolutionary tests such as Tajima's D, Fu and Li's, and Fay and Wu's test statistics, McDonald–Kreitman Neutrality Index, Fixation Index and Extended Haplotype Homozygosity. CATE is magnitudes faster than standard tools with benchmarks estimating it being on average over 180 times faster. For instance, CATE processes all 54,849 human genes for all 22 autosomal chromosomes across the five super populations present in the 1000 Genomes Project in less than 30 min while counterpart software took 3.62 days. This proven framework has the potential to be adapted for GPU‐accelerated large‐scale parallel analyses of many evolutionary and genomic analyses.

Exploring the performance and portability of the k-means algorithm on SYCL across CPU and GPU architectures

The aim of SYCL is to reduce the gap between the performance and code portability of the main accelerators used in HPC, such as multi-vendor CPUs, GPUs, and FPGAs. To evaluate SYCL’s performance portability, this paper uses the k-means algorithm as a case study. The k-means algorithm is simple to code but can be complex to optimize. In this research, we compare our developed SYCL version with the most efficient implementations of CUDA and OpenMP. Our resulting SYCL code can potentially run on multi-vendor CPUs and GPUs. Additionally, we have created a hand-tuned SYCL variation that is optimized for specific device architectures (CPU, NVIDIA GPU, and Intel GPU) to evaluate the performance difference between a standard version and an optimized one. The results show that SYCL outperforms Intel GPUs and CPUs compared to the state-of-the-art He-Vialle version, while on NVIDIA GPUs SYCL offers equivalent performance compared to its native CUDA implementation.

CUDA implementation of the antlion optimization algorithm

A parallel version of Ant Lion Optimizer algorithm using the CUDA platform is introduced in this paper. Efficient kernel, memory and thread management approaches have been developed to maximize its performance. The new algorithm was tested against the canonical algorithm on 15 scalable problems of different sizes, with a total of 172 experiments. The solution costs and the execution times were compared using relative percentage difference and significance tests. The results showed the CUDA antlion algorithm significantly improves upon the execution time, while retaining the same solution quality.

Irregular alignment of arbitrarily long DNA sequences on GPU

The use of Graphics Processing Units to accelerate computational applications is increasingly being adopted due to its affordability, flexibility and performance. However, achieving top performance comes at the price of restricted data-parallelism models. In the case of sequence alignment, most GPU-based approaches focus on accelerating the Smith-Waterman dynamic programming algorithm due to its regularity. Nevertheless, because of its quadratic complexity, it becomes impractical when comparing long sequences, and therefore heuristic methods are required to reduce the search space. We present GPUGECKO, a CUDA implementation for the sequential, seed-and-extend sequence-comparison algorithm, GECKO. Our proposal includes optimized kernels based on collective operations capable of producing arbitrarily long alignments while dealing with heterogeneous and unpredictable load. Contrary to other state-of-the-art methods, GPUGECKO employs a batching mechanism that prevents memory exhaustion by not requiring to fit all alignments at once into the device memory, therefore enabling to run massive comparisons exhaustively with improved sensitivity while also providing up to 6x average speedup w.r.t. the CUDA acceleration of BLASTN.

Outperforming Sequential Full-Word Long Addition With Parallelization and Vectorization

The article presents algorithms for parallel and vectorized full-word addition of big unsigned integers with carry propagation. Because of the propagation, software parallelization and vectorization of non-polynomial addition of big integers have long been considered impractical due to data dependencies between digits of the operands. The presented algorithms are based upon parallel and vectorized detection of carry origins within elements of vector operands, masking bits which correspond to those elements and subsequent scalar addition of the resulting integers. The acquired bits can consequently be taken into account to adjust the sum using the proposed generalization of the Kogge-Stone method. Essentially, the article formalizes and experimentally verifies parallel and vectorized implementation of carry-lookahead adders applied at arbitrary granularity of data. This approach is noticeably beneficial for manycore, CUDA and vectorized implementation using AVX-512 with masked instructions. Experiments show that the parallel and vectorized implementations of the proposed algorithms can be multiple times faster compared to a sequential ripple-carry adder or adders based on redundant number systems such as one used in the GNU Multiple Precision library.

Efficient parallel implementation of crowd simulation using a hybrid CPU+GPU high performance computing system

In the paper we present a modern efficient parallel OpenMP+CUDA implementation of crowd simulation for hybrid CPU+GPU systems and demonstrate its higher performance over CPU-only and GPU-only implementations for several problem sizes including 10 000, 50 000, 100 000, 500 000 and 1 000 000 agents. We show how performance varies for various tile sizes and what CPU–GPU load balancing settings shall be preferred for various domain sizes among CPUs and GPUs of a high performance system with 2 Intel Xeon Silver multicore CPUs and 8 NVIDIA Quadro RTX 5000 GPUs. We then present how execution time depends on the number of agents as well as the number of CUDA streams used for parallel execution of several CUDA kernels. We discuss the design and implementation of an algorithm with CPU computational threads, GPU management threads, assignment of particular tasks to threads as well as usage of pinned memory and CUDA shared memory for maximizing performance.

Improving the Speed and Quality of Parallel Graph Coloring

Graph coloring assigns a color to each vertex of a graph such that no two adjacent vertices get the same color. It is a key building block in many applications. In practice, solutions that require fewer distinct colors and that can be computed faster are typically preferred. Various coloring heuristics exist that provide different quality versus speed tradeoffs. The highest-quality heuristics tend to be slow. To improve performance, several parallel implementations have been proposed. This paper describes two improvements of the widely used LDF heuristic. First, we present a “shortcutting” approach to increase the parallelism by non-speculatively breaking data dependencies. Second, we present “color reduction” techniques to boost the solution of LDF. On 18 graphs from various domains, the shortcutting approach yields 2.5 times more parallelism in the mean, and the color-reduction techniques improve the result quality by up to 20%. Our deterministic CUDA implementation running on a Titan V is 2.9 times faster in the mean and uses as few or fewer colors as the best GPU codes from the literature.

BitCracker: BitLocker meets GPUs

BitLocker is a full-disk encryption feature available in recent Windows versions. It is designed to protect data by providing encryption for entire volumes and it makes use of a number of different authentication methods. In this paper we present a solution, named BitCracker, to attempt the decryption, by means of a dictionary attack, of storage units encrypted by BitLocker with a user supplied password or the recovery password. To that purpose, we resort to Graphics Processing Units (GPU) that are, by now, widely used as general-purpose coprocessors in high performance computing applications. BitLocker decryption process requires the computation of a very large number of SHA-256 hashes and also AES, so we propose a very fast solution, highly tuned for Nvidia GPU, for both of them. We analyze the performance of our CUDA implementation on several Nvidia GPUs and we carry out a comparison of our SHA-256 hash with the Hashcat password cracker tool. Finally, we present our OpenCL version, recently released as a plugin of the John The Ripper tool.

Parallel Evaluation of Chebyshev Approximations: Applications in Astrodynamics

Approximations based on Chebyshev polynomials have several astrodynamic applications. The performance of these approximations can be improved by parallel implementations exploiting parallel architectures, such as OpenMP and CUDA. In this paper, we introduce the parallel implementation to two astrodynamic applications. The first is the gravitational finite element model (FEM): a piecewise Chebyshev approximation that replaces high degree and order gravitational spherical harmonic models (SHMs). Thus, much lower degree, locally valid functions can efficiently model and compute local gravity perturbations in parallel structure for efficient performance. For this model, the total gravity acceleration is split into a reference and disturbance term. The reference includes two-body plus J_2, which are relatively cheap to compute. The FEM approximates the higher-order gravity terms. It is developed from a 2D mesh grid covering a sphere of a specified radius, and a family of spherical shells is sampled using a cosine distribution in the radial direction. To reduce the required memory when seeking a specific accuracy, an adaptive version of the gravitational FEM is introduced. In addition, a parallel implementation of the FEM using OpenMP is preseneted. We show the runtime comparison for the 200 degree × 200 order EGM2008 SHM and the serial and parallel equivalent FEM algorithms. The other application is the Chebyshev-Picard method (CPM): a numerical integrator that solves an ordinary differential equation by approximating the integrand using a Chebyshev approximant and iterates over the trajectory via Picard iteration. A parallel CUDA implementation of the CPM method in conjunction with the EGM2008 SHM and the FEM is introduced. We present numerical examples for propagating four Earth-orbiting satellites considering both the 200times 200 EGM2008 SHM and the equivalent FEM representation to test the algorithm’s performance via parallel and serial computation (i.e., a single CPU thread).

Path integral radiative transfer via polyline representation allowing GPU implementation

Many research areas require simulating particle transport through scattering media. For instance, radiative transfer is useful for computer graphics and for neutron transport in nuclear physics. These transport simulations tend to be computationally expensive for problems involving large amounts of multiple scattering in generic geometries, requiring significant time to compute. Finding a fast solution for these types of problems remains an open area for research. Previous work shows that radiative transfer can be represented as a Feynman Path Integral over all paths between two points in a space. The path integral assigns a weight to each path based on the local curvature of the path, accumulating a transport kernel by summing the weights of all of the paths. Previous work demonstrated a Monte Carlo method for computing the radiative transfer Feynman path integral via repeatedly perturbing paths to generate new paths, using a discrete Frenet-Serret framework and an expensive root-solve calculation. The approach is highly parallelizable on the CPU, however computations require a supercomputer to complete in reasonable time. While a GPU implementation was considered for this previous approach, the root-solve and path representation mapped poorly to the GPU, hindering a reasonable implementation. In the present work, we propose a representation of paths as polylines, and a new curve perturbation method that guarantees production of a new unique path each execution while maintaining the boundary conditions of each path, but without the time-consuming root-solve. The new perturbation method’s structure maps well to the GPU, allowing implementation in CUDA which outperforms the CPU, and allows the utilization of GPU hardware.