GPU Platform Research Articles

PT-NAS: Designing efficient keypoint-based object detectors for desktop CPU platforms

Recently, keypoint-based object detectors have attracted widespread attention, due to their novel structure and excellent performance. However, in terms of their design, there are still two limitations: 1) Most keypoint-based methods are designed for GPU platforms, which makes them inefficient on desktop CPU platforms. 2) Existing works still rely heavily on manual design and prior knowledge. To this end, this work aims to offer a practical solution for designing CPU-efficient key-point detectors. First, we present a set of practical design guidelines by comparing different detection architectures. Following the proposed guidelines, we further develop a progressive three-phase network architecture search (PT-NAS) to achieve the automated design of detection architectures. Benefiting from our hierarchical search space and novel search pipeline, our PT-NAS not only achieves higher search efficiency, but also satisfies the practicality of CPU platforms. On the MS-COCO benchmark, we utilize our PT-NAS to generate several key-point detectors for fast inference on desktop CPUs. Finally, comprehensive comparison experiments prove that the proposed PT-NAS can produce new state-of-the-art keypoint-based detectors for CPU platforms.

CuSCNN: A Secure and Batch-Processing Framework for Privacy-Preserving Convolutional Neural Network Prediction on GPU.

The emerging topic of privacy-preserving deep learning as a service has attracted increasing attention in recent years, which focuses on building an efficient and practical neural network prediction framework to secure client and model-holder data privately on the cloud. In such a task, the time cost of performing the secure linear layers is expensive, where matrix multiplication is the atomic operation. Most existing mix-based solutions heavily emphasized employing BGV-based homomorphic encryption schemes to secure the linear layer on the CPU platform. However, they suffer an efficiency and energy loss when dealing with a larger-scale dataset, due to the complicated encoded methods and intractable ciphertext operations. To address it, we propose cuSCNN, a secure and efficient framework to perform the privacy prediction task of a convolutional neural network (CNN), which can flexibly perform on the GPU platform. Its main idea is 2-fold: (1) To avoid the trivia and complicated homomorphic matrix computations brought by BGV-based solutions, it adopts GSW-based homomorphic matrix encryption to efficiently enable the linear layers of CNN, which is a naive method to secure matrix computation operations. (2) To improve the computation efficiency on GPU, a hybrid optimization approach based on CUDA (Compute Unified Device Architecture) has been proposed to improve the parallelism level and memory access speed when performing the matrix multiplication on GPU. Extensive experiments are conducted on industrial datasets and have shown the superior performance of the proposed cuSCNN framework in terms of runtime and power consumption compared to the other frameworks.

Parallel Fine-Grained Comparison of Long DNA Sequences in Homogeneous and Heterogeneous GPU Platforms With Pruning

The parallelization of Smith-Waterman (SW) sequence comparison tools for long DNA sequences has been a big challenge over the years, requesting the use of several devices and sophisticated optimizations. Pruning is one of these optimizations, which can reduce considerably the amount of computation. This article proposes MultiBP, a sequence comparison solution in multiple GPUs with block pruning. Two MultiBP strategies are proposed. In static score-sharing, workload is statically distributed to the GPUs, and the best score is sent to neighbor GPUs to simulate a global view. In the dynamic strategy, execution is divided into cycles and workload is dynamically assigned, according to the GPUs processing rate. MultiBP was integrated to MASA-CUDAlign and tested in homogeneous and heterogeneous platforms, with different NVidia GPU architectures. The best results in our homogeneous and heterogeneous platforms were mostly obtained by the static and dynamic approaches, respectively. We also show that our decision module is able to select the best strategy in most cases. Finally, the comparison of the human and chimpanzee chromosomes 1 in a cluster with 512 V100 NVidia GPUs took 11 minutes and obtained the impressive rate of 82,822 GCUPS (Billions of Cells Updated per Second) which is, to our knowledge, the best performance for SW tools in GPUs.

Real-time optical flow processing on embedded GPU: an hardware-aware algorithm to implementation strategy

Determining the optical flow of a video is a compute-intensive task essential for computer vision. For achieving this processing in real time, the whole algorithm deployment chain must be thought of for efficiency first. The development is usually divided into two parts: first, designing an algorithm that meets precision constraints, then, implementing and optimizing its execution on the targeted platform. We argue that unifying those operations enhances performance on the embedded processor. This paper is based on an industrial use case of computer vision. The objective is to determine dense optical flow in real time on an embedded GPU platform: the Nvidia AGX Xavier. The CLG (combined local–global) optical flow method, initially chosen, is analyzed to understand the convergence speed of its underlying optimization problem. The Jacobi solver is selected for implementation because of its parallel nature. The whole multi-level processing is then ported to the GPU, using several specific optimization strategies. In particular, we analyze the impact of fusing the solver’s iterations with the roofline model. As a result, with a 30 W power budget, our implementation runs at 60FPS, on $$640 \times 512$$ images, with a four-level processing. Hopefully, this example should provide feedback on the issues that arise when trying to port a method to a parallel platform and serve for further implementations of computer vision algorithms on specialized hardware.

Deep reinforcement learning‐based autonomous parking design with neural network compute accelerators

AbstractWe describe the design and implementation of an autonomous prototype vehicle which finds an empty parking slot in a parking area, and parks itself in the empty parking slot, using neural networks based on deep reinforcement learning (RL). To perform an autonomous parking procedure for our prototype vehicle, two different artificial neural networks (ANNs) are trained using a deep RL Algorithm in a simulation environment and embedded into the computing platform of the prototype car. One of the ANNs enables the vehicle to drive autonomously in the parking environment. At the same time, an image processing algorithm is used to determine whether a parking slot is empty. When the image processing algorithm finds a suitable parking slot, a different ANN is activated and performs a safe parking procedure. However, ANN‐based machine learning techniques require high processing power and impose a high computational burden on embedded CPU and GPU platforms. To alleviate the computational burden, one can achieve higher performance and less power consumption using an application‐specific hardware design, where logic resources are fully exploited according to the algorithm of interest, in an energy‐efficient manner. In this article, hardware accelerators for our ANN models are designed and generated via the Vivado high‐level synthesis (HLS) tool, targeting an ARM based programmable SoC platform, ZedBoard. Our ANN accelerators have achieved a speedup of 17x as compared to an ARM software implementation. For deeper fully‐connected layers used in deep RL‐based solutions, function‐level parallelism (Vivado's dataflow) is employed to improve the computational efficiency. Our proposed stage‐level description for fully connected layers outperforms recent studies in terms of computation time.

A Fast Intrusion Detection Method for High-Speed Railway Clearance Based on Low-Cost Embedded GPUs.

The efficiency and the effectiveness of railway intrusion detection are crucial to the safety of railway transportation. Most current methods of railway intrusion detection or obstacle detection are inappropriate for large-scale applications due to their high cost or limited coverage. In this study, we present a fast and low-cost solution to intrusion detection of high-speed railways. As the solution to heavy computational burdens in the current convolutional-neural-network-based detection methods, the proposed method is mainly a novel neural network based on the SSD framework, which includes a feature extractor using an improved MobileNet and a lightweight and efficient feature fusion module. In addition, aiming to improve the detection accuracy of small objects, the feature map weights are introduced through convolution operation to fuse features at different scales. TensorRT is employed to optimize and deploy the proposed network in the low-cost embedded GPU platform, NVIDIA Jetson TX2, to enhance the efficiency. The experimental results show that the proposed methods achieved 89% mAP on the railway intrusion detection dataset, and the average processing time for a single frame was 38.6 ms on the Jetson TX2 module, which satisfies the need of real-time processing.

A High-throughput Parallel Viterbi Algorithm via Bitslicing

In this work, we present a novel bitsliced high-performance Viterbi algorithm suitable for high-throughput and data-intensive communication. A new column-major data representation scheme coupled with the bitsliced architecture is employed in our proposed Viterbi decoder that enables the maximum utilization of the parallel processing units in modern parallel accelerators. With the help of the proposed alteration of the data scheme, instead of the conventional bit-by-bit operations, 32-bit chunks of data are processed by each processing unit. This means that a single bitsliced parallel Viterbi decoder is capable of decoding 32 different chunks of data simultaneously. Here, the Viterbi’s Add-Compare-Select procedure is implemented with our proposed bitslicing technique, where it is shown that the bitsliced operations for the Viterbi internal functionalities are efficient in terms of their performance and complexity. We have achieved this level of high parallelism while keeping an acceptable bit error rate performance for our proposed methodology. Our suggested hard and soft-decision Viterbi decoder implementations on GPU platforms outperform the fastest previously proposed works by and , achieving 21.41 and 8.24 Gbps on Tesla V100, respectively.

Chauffeur: Benchmark Suite for Design and End-to-End Analysis of Self-Driving Vehicles on Embedded Systems

Self-driving systems execute an ensemble of different self-driving workloads on embedded systems in an end-to-end manner, subject to functional and performance requirements. To enable exploration, optimization, and end-to-end evaluation on different embedded platforms, system designers critically need a benchmark suite that enables flexible and seamless configuration of self-driving scenarios, which realistically reflects real-world self-driving workloads’ unique characteristics. Existing CPU and GPU embedded benchmark suites typically (1) consider isolated applications, (2) are not sensor-driven, and (3) are unable to support emerging self-driving applications that simultaneously utilize CPUs and GPUs with stringent timing requirements. On the other hand, full-system self-driving simulators (e.g., AUTOWARE, APOLLO) focus on functional simulation, but lack the ability to evaluate the self-driving software stack on various embedded platforms. To address design needs, we present Chauffeur, the first open-source end-to-end benchmark suite for self-driving vehicles with configurable representative workloads. Chauffeur is easy to configure and run, enabling researchers to evaluate different platform configurations and explore alternative instantiations of the self-driving software pipeline. Chauffeur runs on diverse emerging platforms and exploits heterogeneous onboard resources. Our initial characterization of Chauffeur on different embedded platforms – NVIDIA Jetson TX2 and Drive PX2 – enables comparative evaluation of these GPU platforms in executing an end-to-end self-driving computational pipeline to assess the end-to-end response times on these emerging embedded platforms while also creating opportunities to create application gangs for better response times. Chauffeur enables researchers to benchmark representative self-driving workloads and flexibly compose them for different self-driving scenarios to explore end-to-end tradeoffs between design constraints, power budget, real-time performance requirements, and accuracy of applications.

Learning to Train CNNs on Faulty ReRAM-based Manycore Accelerators

The growing popularity of convolutional neural networks (CNNs) has led to the search for efficient computational platforms to accelerate CNN training. Resistive random-access memory (ReRAM)-based manycore architectures offer a promising alternative to commonly used GPU-based platforms for training CNNs. However, due to the immature fabrication process and limited write endurance, ReRAMs suffer from different types of faults. This makes training of CNNs challenging as weights are misrepresented when they are mapped to faulty ReRAM cells. This results in unstable training, leading to unacceptably low accuracy for the trained model. Due to the distributed nature of the mapping of the individual bits of a weight to different ReRAM cells, faulty weights often lead to exploding gradients. This in turn introduces a positive feedback in the training loop, resulting in extremely large and unstable weights. In this paper, we propose a lightweight and reliable CNN training methodology using weight clipping to prevent this phenomenon and enable training even in the presence of many faults. Weight clipping prevents large weights from destabilizing CNN training and provides the backpropagation algorithm with the opportunity to compensate for the weights mapped to faulty cells. The proposed methodology achieves near-GPU accuracy without introducing significant area or performance overheads. Experimental evaluation indicates that weight clipping enables the successful training of CNNs in the presence of faults, while also reducing training time by 4 X on average compared to a conventional GPU platform. Moreover, we also demonstrate that weight clipping outperforms a recently proposed error correction code (ECC)-based method when training is carried out using faulty ReRAMs.

Efficient Implementation of Liquid Crystal Simulation Software on Modern HPC Platforms

In this paper we demonstrate the process of efficient porting a software package for Markov chain Monte Carlo (MCMC) simulations on a finite cubic lattice on multiple modern architectures: Pascal, Volta and Turing NVIDIA GPUs, NEC SX-Aurora TSUBASA vector engines and Intel Xeon Gold processors. In the studied software, MCMC methodology is used for simulations of liquid crystal structures, but it can be as well employed in a wide range of problems of mathematical physics and numerical methods. The main goals of this work are to determine the best software optimization strategy for this class of algorithms and to examine the speed and the efficiency of such simulations on modern HPC platforms. We evaluate the effects of various optimizations, such as using more suitable memory access patterns, multitasking for efficient utilization of massive parallelism on the target architectures, improved cache hit-rates, parallel workload balancing, etc. We perform a detailed performance analysis for each target platform using software tools such as nvprof, Ftrace and VTune. On this basis, we evaluate and compare the efficiency of the developed computational kernels on different platforms and subsequently rank these platforms by their performance. The results show that NVIDIA GPU and NEC SX-Aurora TSUBASA platforms, although at first glance seem very different, require similar optimization approaches in many cases due to similarities in data processing principles.

Best Practices for the Deployment of Edge Inference: The Conclusions to Start Designing

The number of Artificial Intelligence (AI) and Machine Learning (ML) designs is rapidly increasing and certain concerns are raised on how to start an AI design for edge systems, what are the steps to follow and what are the critical pieces towards the most optimal performance. The complete development flow undergoes two distinct phases; training and inference. During training, all the weights are calculated through optimization and back propagation of the network. The training phase is executed with the use of 32-bit floating point arithmetic as this is the convenient format for GPU platforms. The inference phase on the other hand, uses a trained network with new data. The sensitive optimization and back propagation phases are removed and forward propagation is only used. A much lower bit-width and fixed point arithmetic is used aiming a good result with reduced footprint and power consumption. This study follows the survey based process and it is aimed to provide answers such as to clarify all AI edge hardware design aspects from the concept to the final implementation and evaluation. The technology as frameworks and procedures are presented to the order of execution for a complete design cycle with guaranteed success.

High-performance implementation of evolutionary privacy-preserving algorithm for big data using GPU platform

Privacy protection has become a predominant concern in big data analysis. Privacy-Preserving Association Rule Mining (PPARM) is a process in which original data is transformed into a sanitized version to remove sensitive patterns. Although evolutionary PPARM approaches were developed, they are inefficient because the fitness function is overwhelmingly expensive. In this paper, an efficient algorithm, GPU-based Evolutionary Privacy-Preserving (GEPP), for improving the performance of PPARM is presented. We introduce two strategies for this reason: the first one is to develop a parallel indexing machine to generate retrieval index lists by parallelizing dataset scanning between GPU blocks, and the second strategy is the parallelization of the index lists for fitness function computation using CPU/GPU platform. In the second strategy, search mechanism and fitness function steps are performed on CPU and GPU, respectively. The scalability of GEPP is evaluated in terms of GPU characteristics, e.g., blocks and threads, data characteristics, e.g., the number of transactions, items, and density ratio, and patterns characteristics, e.g., the ratio of sensitive patterns and support thresholds. Experimental results show that our parallel implementation of the proposed approach obtains a speedup up to 36.6x, on average, compared to the CPU implementation using real and synthetic large-scale transaction datasets.

Edge Artificial Intelligence: Real-Time Noninvasive Technique for Vital Signs of Myocardial Infarction Recognition Using Jetson Nano

The history of medicine shows that myocardial infarction is one of the significant causes of death in humans. The rapid evolution in autonomous technologies, the rise of computer vision, and edge computing offers intriguing possibilities in healthcare monitoring systems. The major motivation of the work is to improve the survival rate during a cardiac arrest through an automatic emergency recognition system under ambient intelligence. We present a novel approach to chest pain and fall posture-based vital sign detection using an intelligence surveillance camera to address the emergency during myocardial infarction. A real-time embedded solution persuaded from “edge AI” is implemented using the state-of-the-art convolution neural networks: single shot detector Inception V2, single shot detector MobileNet V2, and Internet of Things embedded GPU platform NVIDIA’s Jetson Nano. The deep learning algorithm is implemented for 3000 indoor color image datasets: Nanyang Technological University Red Blue Green and Depth, NTU RGB + D dataset, and private RMS dataset. The research mainly pivots on two key factors in creating and training a CNN model to detect the vital signs and evaluate its performance metrics. We propose a model, which is cost-effective and consumes low power for onboard detection of vital signs of myocardial infarction and evaluated the metrics to achieve a mean average precision of 76.4% and an average recall of 80%.

A stable parallel algorithm for block tridiagonal toeplitz–block–toeplitz linear systems

In this paper, we present a direct parallel block WZ algorithm, named DPBWZA, for the solution of block tridiagonal toeplitz–block–toeplitz (TBT) linear system Ax=f. The algorithm is based on the proposed block WZ factorization of the coefficient matrix A. Existence of the block WZ factorization for block tridiagonal TBT block diagonally dominant matrix is proved. Error analysis of the parallel algorithm DPBWZA is presented and numerical stability of the algorithm is established. Numerical experiments are conducted to demonstrate the efficiency, stability and accuracy of the Direct Parallel Block WZ Algorithm on GPU platform. Forward and backward errors are computed and DPBWZA is found to be highly accurate, numerically stable and as efficient as the subroutine csrlsvlu of GPU accelerated cuSolverSP library.

Signal processing for a reverse‐GPS wildlife tracking system: CPU and GPU implementation experiences

AbstractWe present robust high‐performance implementations of signal‐processing tasks performed by a high‐throughput wildlife tracking system called ATLAS. The system tracks radio transmitters attached to wild animals by estimating the time of arrival of radio packets to multiple receivers (base stations). Time‐of‐arrival estimation of wideband radio signals is computationally expensive, especially in acquisition mode (when the time of transmission is not known, not even approximately). These computations are a bottleneck that limits the throughput of the system. We developed a sequential high‐performance CPU implementation of the computations a few years back, and more recently a GPU implementation. Both strive to balance performance with simplicity, maintainability, and development effort, as most real‐world codes do. The article reports on the two implementations and carefully evaluates their performance. The evaluations indicates that the GPU implementation dramatically improves performance and power‐performance relative to the sequential CPU implementation running on a desktop CPU typical of the computers in current base stations. Performance improves by more than 50X on a high‐end GPU and more than 4X with a GPU platform that consumes almost 5 times less power than the CPU platform. Performance‐per‐Watt ratios also improve (by more than 16X), and so do the price‐performance ratios.

Comparison of CPU and GPU Platforms in Problems of Wave Diagnostics

Comparison of CPU and GPU Platforms in Problems of Wave Diagnostics

AN IMPLICIT BLOCK JACOBI APPROACH FOR HIGH-ORDER FLUX RECONSTRUCTION METHOD

Recently, the flux reconstruction (FR) method has attracted more and more attentions for its simplicity and generality. However, it is still computationally expensive and time consuming when simulating the complex flow problems by FR method. There is a huge demand for developing appropriate efficient implicit solvers and parallel computing techniques for FR. This paper proposes an implicit high-order flux reconstruction solver on GPU platform based on the block Jacobi iteration method. As it is inefficient to solve the large global linear system resulting from spatial and implicit temporal discretization of FR directly. A block Jacobi approach is used to change the characteristics of the lift-hand matrix of the global linear system and this avoids the dependence of neighboring elements. Therefore, only the diagonal blocks of global matrix need to be stored and calculated. Then, the problem of solving the huge global linear system is transformed into solving a series of local linear equations simultaneously. Finally, these small local linear equations would be solved by the LU decomposition method in parallel on GPU platforms. Two typical cases, including subsonic flows over a bump and a NACA0012 airfoil, were simulated and compared with the multi-grid explicit Runge-Kutta scheme. The numerical results demonstrated that the present implicit method can reduce the iterations significantly. Meanwhile, the implicit solver has shown at least 10x speedup over the multi-grid Runge-Kutta scheme in all cases.

Cross-platform programming model for many-core lattice Boltzmann simulations.

We present a novel, hardware-agnostic implementation strategy for lattice Boltzmann (LB) simulations, which yields massive performance on homogeneous and heterogeneous many-core platforms. Based solely on C++17 Parallel Algorithms, our approach does not rely on any language extensions, external libraries, vendor-specific code annotations, or pre-compilation steps. Thanks in particular to a recently proposed GPU back-end to C++17 Parallel Algorithms, it is shown that a single code can compile and reach state-of-the-art performance on both many-core CPU and GPU environments for the solution of a given non trivial fluid dynamics problem. The proposed strategy is tested with six different, commonly used implementation schemes to test the performance impact of memory access patterns on different platforms. Nine different LB collision models are included in the tests and exhibit good performance, demonstrating the versatility of our parallel approach. This work shows that it is less than ever necessary to draw a distinction between research and production software, as a concise and generic LB implementation yields performances comparable to those achievable in a hardware specific programming language. The results also highlight the gains of performance achieved by modern many-core CPUs and their apparent capability to narrow the gap with the traditionally massively faster GPU platforms. All code is made available to the community in form of the open-source project stlbm, which serves both as a stand-alone simulation software and as a collection of reusable patterns for the acceleration of pre-existing LB codes.

Investigation of the efficiency optimization for the improved subgroup resonance self-shielding treatment on the GPU platform

The improved subgroup method adopts a fine energy mesh to handle the resonance interference effect, then the neutron slowing-down equation is solved for group condensation to obtain the multigroup cross section. A large calculating burden is raised in this process. In this paper, an acceleration scheme by the multigroup kernel method is investigated on both GPU and CPU platform. First, the subgroup fixed source and the neutron slowing-down equations are both defined to multigroup kernels to prevent the repetitive MOC sweep, and the group batch is introduced to save the memory usage. Then, the resonance treatment code is developed on the CPU/GPU heterogeneous clusters. Finally, the efficiency between the multigroup kernel and conventional group-by-group method is compared. The numerical verification shows that the multigroup kernel method in the GPU platform could effectively accelerate the solutions of both subgroup fixed source and neutron slowing-down equations.

Optimizing the LINPACK Algorithm for Large-Scale PCIe-Based CPU-GPU Heterogeneous Systems

There is a widening gap between GPU and other components (CPU, PCIe bus and communication network) in heterogeneous parallel system. The gap forces us to orchestrate cooperative execution among these components much more carefully than ever before. By taking the LINPACK benchmark as a case study, this article proposes a fine-grained pipelining algorithm on large-scale CPU-GPU heterogeneous cluster systems. First, we build an algorithmic model that reveals a new approach to GPU-centric and fine-grained pipelining algorithm design. Then, we present four model-driven pipelining algorithms that incrementally squeeze bubbles in the pipeline so that it is occupied by more useful floating-point calculations. The algorithms are implemented on both the AMD and NVIDIA GPU platforms. The finally optimized LINPACK program achieves 107 PFlops on 25, 600 GPUs (70 percent floating-point efficiency). Several insights have been drawn to suggest tradeoff of algorithm design, programming support, and architecture design.