GPU Applications Research Articles

A Comprehensive of CPU and GPU Performance and Applications in Autonomous Vehicles

The rapid advancement of autonomous vehicle technology over the past decade has significantly increased the complexity of intelligent transportation systems. This need is clearly reflected in the requirements for high-performance computing in autonomous vehicles — a requirement that artificial intelligence, machine learning and big data analytic have all come to meet through their integration. We need the CPU and GPU power to handle processing in real-time data, sensor fusion and decision-making. We investigate how the CPU and GPU perform in an autonomous driving scenario, concluding that these tasks are complementary to one another describing where each can be best used within a heterogeneous computing architecture. The research delves into how these computing chips can be best used under the demand for real-time processing, reliability and efficiency. The study is to better understand the techniques for improving CPU-GPU collaboration, and applies these new findings on performance-intensive tasks like image recognition or track construction. The technical part involves conducting driving scenarios in the real world on three different types of CPU and several GPU, with key metrics such as processing latency, power consumption and accuracy. The actual experimental results show the advantages of GPU parallel computing and deep learning, while CPU still has a good advantage in multi-tasking ability and logical calculation. The results are important for the implementation of future autonomous driving systems, highlighting heterogeneous computing architectures as a means to optimize and support safe, effective vehicle operations.

Easily porting material point methods codes to GPU

AbstractThe material point method (MPM) is computationally costly and highly parallelisable. With the plateauing of Moore’s law and recent advances in parallel computing, scientists without formal programming training might face challenges in developing fast scientific codes for their research. Parallel programming is intrinsically different to serial programming and may seem daunting to certain scientists, in particular for GPUs. However, recent developments in GPU application programming interfaces (APIs) have made it easier than ever to port codes to GPU. This paper explains how we ported our modular C++ MPM code to GPU without using low-level hardware APIs like CUDA or OpenCL. We aimed to develop a code that has abstracted parallelism and is therefore hardware agnostic. We first present an investigation of a variety of GPU APIs, comparing ease of use, hardware support and performance in an MPM context. Then, the porting process of to the Kokkos ecosystem is detailed, discussing key design patterns and challenges. Finally, our parallel C++ code running on GPU is shown to be up to 85 times faster than on CPU. Since Kokkos also supports Python and Fortran, the principles presented therein can also be applied to codes written in these languages.

Analyzing the impact of CUDA versions on GPU applications

CUDA toolkits are widely used to develop applications running on NVIDIA GPUs. They include compilers and are frequently updated to integrate state-of-the-art compilation techniques. Hence, many HPC users believe that the latest CUDA toolkit will improve application performance; however, considering results from CPU compilers, there are cases where this is not true. In this paper, we thoroughly evaluate the impact of CUDA toolkit version on the performance, power consumption, and energy consumption of GPU applications with four GPU architectures. Our results show that though the latest CUDA toolkit obtains the best performance, power consumption, and energy consumption for many applications in most cases, but we found a few exceptions. For such applications, we conducted an in-depth analysis using the SASS to identify why older CUDA toolkit achieve performance improvement. Our analysis showed that the factors that caused them are by three phenomena: aggressive loop unrolling, inefficient instruction scheduling, and the impact of host compilers.

CuCatch: A Debugging Tool for Efficiently Catching Memory Safety Violations in CUDA Applications

CUDA, OpenCL, and OpenACC are the primary means of writing general-purpose software for NVIDIA GPUs, all of which are subject to the same well-documented memory safety vulnerabilities currently plaguing software written in C and C++. One can argue that the GPU execution environment makes software development more error prone. Unlike C and C++, CUDA features multiple, distinct memory spaces to map to the GPU’s unique memory hierarchy, and a typical CUDA program has thousands of concurrently executing threads. Furthermore, the CUDA platform has fewer guardrails than CPU platforms that have been forced to incrementally adjust to a barrage of security attacks. Unfortunately, the peculiarities of the GPU make it difficult to directly port memory safety solutions from the CPU space. This paper presents cuCatch, a new memory safety error detection tool designed specifically for the CUDA programming model. cuCatch combines optimized compiler instrumentation with driver support to implement a novel algorithm for catching spatial and temporal memory safety errors with low performance overheads. Our experimental results on a wide set of GPU applications show that cuCatch incurs a 19% runtime slowdown on average, which is orders of magnitude faster than state-of-the-art debugging tools on GPUs. Moreover, our quantitative evaluation demonstrates cuCatch’s higher error detection coverage compared to prior memory safety tools. The combination of high error detection coverage and low runtime overheads makes cuCatch an ideal candidate for accelerating memory safety debugging for GPU applications.

3D least-squares reverse time migration in VTI media based on pseudoacoustic wave equation and multi-GPU parallel acceleration

As the high-density seismic acquisition is progressively becoming an essential mode of seismic exploration, developing high-precision imaging methods for 3D datasets has become an urgent industry requirement. Unfortunately, the two-dimensional least-squares reverse time migration (LSRTM) can hardly deal with the influence of the third dimension because the structure of reflectors is constantly changing in nearly all directions. Furthermore, anisotropy is very common in underground media, which makes the LSRTM based on isotropic assumption fail to locate the reflectors accurately and seriously limits the identification of geological stratification. Considering the above two problems, this paper derives the least-squares reverse time migration in three-dimensional vertical transversely isotropic media (3D-VTI-LSRTM) and develops a workflow which can realize 3D-LSRTM in VTI media based on this theory. Our workflow uses multi-GPU acceleration and heterogeneous CPU/GPU parallelism. Compared with the general parallel CPU workflow or single GPU workflow, it is feasible that the GPU application significantly improves computing efficiency in practical applications. In addition, this paper extends VTI-LSRTM from 2D to 3D, which solves the problem that the 2D algorithm cannot eliminate the lateral direction reflection artifacts. For the self-made 3D-VTI depression model and the 3D-VTI-SEG/EAGE salt model, we use four NVIDIA Tesla k40c graphics cards for testing. The results of the tests are correct and significantly improved compared to the 3D-VTI-RTM method.

Enhancing DEVITO GPU Allocator Using Unified Memory by NVIDIA

DEVITO is a framework whose objective is to implement optimized stencil computing. Its execution can be carried out both in the CPU and in GPU. For this reason, the data must be manipulated correctly so that, in case of executions in the GPU, they are present in the memory of the GPU at the time of the execution. Natively, DEVITO transfers data every time the operator is executed from OpenACC pragmas. This approach results in performance degradation when the operator is executed repeatedly. To prevent redundant copies and alleviate this bottleneck, an allocator based on unified memory was implemented, which makes manual data transfer between CPU and GPU unnecessary, significantly reducing data transfer time in GPU applications.

Gemini: Enabling Multi-Tenant GPU Sharing Based on Kernel Burst Estimation

Recent years have seen rapid adoption of GPUs in various types of platforms because of the tremendous throughput powered by massive parallelism. However, as the computing power of GPU continues to grow at a rapid pace, it also becomes harder to utilize these additional resources effectively with the support of GPU sharing. In this work, we designed and implemented <i>Gemini</i> , a user-space runtime scheduling framework to enable fine-grained GPU allocation control with support for multi-tenancy and elastic allocation, which are critical for cloud and resource providers. Our key idea is to introduce the concept of <i>kernel burst</i> , which refers to a group of consecutive kernels launched together without being interrupted by synchronous events. Based on the characteristics of kernel burst, we proposed a low overhead <i>event-driven monitor</i> and a <i>dynamic time-sharing scheduler</i> to achieve our goals. Our experiment evaluations using five types of GPU applications show that Gemini enabled multi-tenant and elastic GPU allocation with less than 5% performance overhead. Furthermore, compared to static scheduling, Gemini achieved 20% <inline-formula><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 30% performance improvement without requiring prior knowledge of applications.

Improving the Scalability of GPU Synchronization Primitives

General-purpose GPU applications increasingly use synchronization to enforce ordering between many threads accessing shared data. Accordingly, recently there has been a push to establish a common set of GPU synchronization primitives. However, the expressiveness of existing GPU synchronization primitives is limited. In particular the expensive GPU atomics often used to implement fine-grained synchronization make it challenging to implement efficient algorithms. Consequently, as GPU algorithms scale to millions or billions of threads, existing GPU synchronization primitives either scale poorly or suffer from livelock or deadlock issues because of heavy contention between threads accessing shared synchronization objects. We seek to overcome these inefficiencies by designing more efficient, scalable GPU barriers and semaphores. In particular, we show how multi-level sense reversing barriers and priority mechanisms for semaphores can be designed with the GPUs unique processing model in mind to improve performance and scalability of GPU synchronization primitives. Our results show that the proposed designs significantly improve performance compared to state-of-the-art solutions like CUDA Cooperative Groups and optimized CPU-style synchronization algorithms at medium and high contention levels, scale to an order of magnitude more threads, and avoid livelock in these situations unlike prior open source algorithms. Overall, across three modern GPUs the proposed barrier algorithm improves performance by an average of 33% over a GPU tree barrier algorithm and improves performance by an average of 34% over CUDA Cooperative Groups for five full-sized benchmarks at high contention levels; the new semaphore algorithm improves performance by an average of 83% compared to prior GPU semaphores.

A scalable and energy efficient GPU thread map for [formula omitted]-simplex domains

This work proposes a new GPU thread map for m-simplex domains that improves its speedup along with the m-dimension and is energy efficient compared to other state of the art approaches. The main contributions of this work are (i) the formulation of an improved new block-space map H:Zm↦Zm for regular orthogonal simplex domains, which is analyzed in terms of resource usage, and (ii) the experimental evaluation in terms of speedup and energy efficiency with respect to a bounding box approach. Results from the analysis show that H has a potential speedup of up to 2× and 6× for 2 and 3-simplices, respectively. Experimental evaluation shows that H is competitive for 2-simplices, reaching 1.2×∼2.0× of speedup for different tests, which is on par with the fastest state of the art approaches. For 3-simplices H reaches up to 1.3×∼6.0× of speedup making it the fastest. The extension of H to higher dimensional m-simplices is feasible and has a potential speedup that scales as m! given a proper selection of parameters r,β which are the scaling and replication factors of the geometry, respectively. In terms of energy consumption, although H is among the highest in power consumption, it compensates by its short duration, making it one of the most energy efficient approaches. The results of this work show that H is a scalable and energy efficient map that improves the efficiency of GPU applications that need to process m-simplex domains, such as Cellular Automata or PDE simulations, among others.

Analytical performance estimation during code generation on modern GPUs

Automatic code generation is frequently used to create implementations of algorithms specifically tuned to particular hardware and application parameters. The code generation process involves the selection of adequate code transformations, tuning parameters, and parallelization strategies. We propose an alternative to time-intensive autotuning, scenario-specific performance models, or black-box machine learning to select the best-performing configuration.This paper identifies the relevant performance-defining mechanisms for memory-intensive GPU applications through a performance model coupled with an analytic hardware metric estimator. This enables a quick exploration of large configuration spaces to identify highly efficient code candidates with high accuracy.We examine the changes of the A100 GPU architecture compared to the predecessor V100 and address the challenges of how to model the data transfer volumes through the new memory hierarchy.We show how our method can be coupled to the “pystencils” stencil code generator, which is used to generate kernels for a range-four 3D-25pt stencil and a complex two-phase fluid solver based on the Lattice Boltzmann Method. For both, it delivers a ranking that can be used to select the best-performing candidate.The method is not limited to stencil kernels but can be integrated into any code generator that can generate the required address expressions.

Learning based application driven energy aware compilation for GPU

The choice of correct compiler optimization is one of the major factors that determines the performance and power consumption of an application when executed on a target hardware. However, making such a choice is challenging since it requires knowledge of target architecture, code features and compiler pass interdependence. Moreover, the onus of choosing these optimization passes lie on the software developers who rely on heuristic solutions which often yield sub-optimal results. In this paper we proposed a machine learning based mechanism of determining the most energy aware compiler setting for a GPU application. Our proposed technique serves a dual purpose of reducing the energy consumption during execution on one hand and relieving the application developers from making the complex choices on the other. Our proposed approach displayed an average improvement of 11.04% and 5.25% improvement in power and energy efficiency respectively with respect to state-of-art techniques.

Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads (Extended Abstract)

Hazelwood et al. observed that at Facebook data centers, variations in user activity (e.g. due to diurnal load) resulted in low utilization periods with large pools of idle resources [4]. To make use of these resources, they proposed using machine learning training tasks. Analagous lowutilization periods have also been observed at the scale of individual GPUs when using both GPU-based inference [1] and training [6]. The proposed solution to this latter problem was colocating additional inference or training tasks on a single GPU.We go a step further than these previous studies by considering the GPU at the microarchitectural level rather than treating it as a black box. Broadly, we consider the following question: are current GPU application- and block-level scheduling mechanisms sufficient to guarantee predictable and low turnaround times for latency-sensitive inference requests, while also consistently making use of unoccupied resources for best-effort training tasks? To answer this question, we explore both NVIDIA's concurrency mechanisms and the characteristics of the workload itself. Complicating our analyses, the NVIDIA scheduling hierarchy is proprietary and some mechanisms (e.g., time-slicing) are not well-documented, so their behavior must be reverseengineered from empirical observation.

A study on GPU acceleration applied to 2D irregular packing problems

ABSTRACT Irregular packing problems are an important subject of study in C&P problems. An efficient solution can have a great economic and environmental impact. The main objective is to obtain a feasible layout, a configuration whereby items are completely placed inside one or more containers without overlap. Although many solutions in the literature are capable of achieving high density solutions for benchmark instances, they are limited to small and medium problems. The best packing algorithms adopt the overlap minimization approach, in which the overlap restriction is relaxed by adopting an overlap function. Thus, a study of parallel implementation is proposed to accelerate the overlap minimization solution and reduce the processing time, potentially allowing for the solution of more complex instances. The results showed high speedups for the parallelization of the local search algorithm, achieving an acceleration of up to 16x. Then, by applying this accelerated method to a packing algorithm, speedups of up to 4.5 were observed. Due to their stochastic nature, the tests were repeated several times for each instance and the average results were computed. These results demonstrated the potential for GPU application with irregular packing, which can be extended to achieve its full capability.

Repurposing GPU Microarchitectures with Light-Weight Out-Of-Order Execution

GPU is the dominant platform for accelerating general-purpose workloads due to its computing capacity and cost-efficiency. GPU applications cover an ever-growing range of domains. To achieve high throughput, GPUs rely on massive multi-threading and fast context switching to overlap computations with memory operations. We observe that among the diverse GPU workloads, there exists a significant class of kernels that fail to maintain a sufficient number of active warps to hide the latency of memory operations, and thus suffer from frequent stalling. We argue that the dominant Thread-Level Parallelism model is not enough to efficiently accommodate the variability of modern GPU applications. To address this inherent inefficiency, we propose a novel micro-architecture with lightweight Out-Of-Order execution capability enabling Instruction-Level Parallelism to complement the conventional Thread-Level Parallelism model. To minimize the hardware overhead, we carefully design our extension to highly re-use the existing micro-architectural structures and study various design trade-offs to contain the overall area and power overhead, while providing improved performance. We show that the proposed architecture outperforms traditional platforms by 23 percent on average for low-occupancy kernels, with an area and power overhead of 1.29 and 10.05 percent, respectively. Finally, we establish the potential of our proposal as a micro-architecture alternative by providing 16 percent speedup over a wide collection of 60 general-purpose kernels.

Tensorox: Accelerating GPU Applications via Neural Approximation on Unused Tensor Cores

Driven by the demands of deep learning, many hardware accelerators, including GPUs, have begun to include specialized tensor processing units to accelerate matrix operations. However, general-purpose GPU applications that have little or no large dense matrix operations cannot benefit from these tensor units. This article proposes Tensorox, a framework that exploits the half-precision tensor cores available on recent GPUs for approximable, non deep learning applications. In essence, a shallow neural network is trained based on the input-output mapping of the function to be approximated. The key innovation in our implementation is the use of the small and dimension-restricted tensor operations in Nvidia GPUs to run multiple instances of the approximation neural network in parallel. With the proper scaling and training methods, our approximation yielded an overall accuracy that is higher than naïvely running the original programs with half-precision. Furthermore, Tensorox allows for the runtime adjustment of the degree of approximation. For the 10 benchmarks we tested, we achieved speedups from 2× to 112× compared to the original in single precision floating point, while maintaining the error caused by the approximation to below 10 percent in most applications.

Uncovering input-sensitive energy bottlenecks in oversubscribed GPU workloads

Graph algorithms are at the core of data-intensive applications in many computational domains. Although accelerator-based implementations have the potential for delivering improved energy efficiency on today’s large-scale graph analytics, there are many challenges to reaching the desired goals. This paper examines the issues of GPU oversubscription and input sensitivity in energy efficient graph processing. We introduce the notion of energy signatures, a collection of runtime events that can determine fluctuations in power consumption as a function of input graph characteristics. The study shows that variations in volume, density, diameter and dispersion in graphs can alter the runtime behavior of GPU applications and these changes in program behavior in turn impact the energy efficiency of the application. Based on the insight gained from our study, we develop three input-aware optimization techniques to mitigate energy bottlenecks. On experiments conducted on a Unified Memory supported GPU, the optimizations yield a 11.97% energy savings on average on Breadth-First Search and Single-Source Shortest Path algorithms.

Research on Programming Model and Compilation Optimization Technology of Multi-Core GPU

GPGPU (General Purpose Computing on Graphics Processing Units) has been widely applied to high performance computing. However, GPU architecture and programming model are different from that of traditional CPU. Accordingly, it is rather challenging to develop efficient GPU applications. This paper focuses on the key techniques of programming model and compiler optimization for many-core GPU, and addresses a number of key theoretical and technical issues. This paper proposes a many-threaded programming model ab-Stream, which would transparentize architecture differences and provide an easy to parallel, easy to program, easy to extend and easy to tune programming model. In addition, this paper proposes memory optimization and data transfer transformation according to data classification. Firstly, this paper proposes data layout pruning based on classification memory, and then proposes Ta T (Transfer after Transformed) for transferring Strided data between CPU and GPU. Experimental results demonstrate that proposed techniques would significantly improve performance for GPGPU applications.

Byte-Select Compression

Cache-block compression is a highly effective technique for both reducing accesses to lower levels in the memory hierarchy (cache compression) and minimizing data transfers (link compression). While many effective cache-block compression algorithms have been proposed, the design of these algorithms is largely ad hoc and manual and relies on human recognition of patterns. In this article, we take an entirely different approach. We introduce a class of “byte-select” compression algorithms, as well as an automated methodology for generating compression algorithms in this class. We argue that, based on upper bounds within the class, the study of this class of byte-select algorithms has potential to yield algorithms with better performance than existing cache-block compression algorithms. The upper bound we establish on the compression ratio is 2X that of any existing algorithm. We then offer a generalized representation of a subset of byte-select compression algorithms and search through the resulting space guided by a set of training data traces. Using this automated process, we find efficient and effective algorithms for various hardware applications. We find that the resulting algorithms exploit novel patterns that can inform future algorithm designs. The generated byte-select algorithms are evaluated against a separate set of traces and evaluations show that Byte-Select has a 23% higher compression ratio on average. While no previous algorithm performs best for all our data sets which include CPU and GPU applications, our generated algorithms do. Using an automated hardware generator for these algorithms, we show that their decompression and compression latency is one and two cycles respectively, much lower than any existing algorithm with a competitive compression ratio.

Analyzing DUE Errors on GPUs With Neutron Irradiation Test and Fault Injection to Control Flow

As GPU applications expand, the reliability of GPU is drawing more attention since even reliability-demanding applications are executed on GPUs. Silent data corruption (SDC) is widely studied both in irradiation experiments and fault injection experiments. On the other hand, detectable uncorrected error (DUE) is not well studied. This work focuses on DUEs reported by the GPU driver and analyzes those observed in fault injection and neutron irradiation experiments, where faults are injected in the control flow to change the program counter value unexpectedly. The DUE errors of GPU engine exception, GPU memory page fault, and GPU processing stop are observed in both the experiments. On the other hand, the DUE error categorized as internal microcontroller halt by the GPU driver, which is not found in the fault injection experiment, is observed frequently, suggesting the necessity of investigating the failures originating from the faults in the components invisible to programmers.

A Taxonomy of Modern GPGPU Programming Methods: On the Benefits of a Unified Specification

Several Application Programming Interfaces (APIs) and frameworks have been proposed to simplify the development of General-Purpose GPU (GPGPU) applications. GPGPU application development typically involves specific customization for the target operating systems and hardware devices. The effort to port applications from one API to the other (or to develop multi-target applications) is complicated by the availability of a plethora of specifications, which in essence offers very similar underlying functionality. In this work we provide an in-depth study of six state-of-the-art GPGPU APIs. From these we derive a taxonomy of the common semantics and propose a unified specification. We describe a methodology to translate this unified specification into different target APIs. This simplifies cross-platform application development and provides a clean framework for benchmarking. Our proposed unified specification is called GUST (GPGPU Unified Specification and Translation) and it captures common functionality found in compute-only APIs (e.g., CUDA and OpenCL), in the compute pipeline of traditional graphic-oriented APIs (e.g., OpenGL and Direct3D11) and in last-generation bare-metal APIs (e.g., Vulkan and Direct3D12). The proposed translation methodology solves differences between specific APIs in a transparent manner, without hiding available tuning knobs for compute kernel optimizations and fostering best programming practices in a simple manner.