NVIDIA Graphics Processing Units Research Articles

A Comprehensive Analysis of Nvidia's Technological Innovations, Market Strategies, and Future Prospects

The article provides an in-depth analysis of Nvidia's technological evolution and its profound impact on Machine Learning, Big Data, and Artificial Intelligence (AI) on a global scale. Nvidia has emerged as a trailblazer, reshaping computational capabilities, and establishing itself as a prominent player in a fiercely competitive landscape. The examination meticulously scrutinizes the role of Nvidia's graphics processing unit (GPU) technologies in spearheading a transformative computing revolution, emphasizing the collaborative prowess inherent in Nvidia's developer ecosystem. The analysis extends to the dynamics of GPU innovation, its disruptive influence on the market, and the robust innovation engine ingrained within Nvidia's culture of calculated risk-taking. Internal and external factors contributing to Nvidia's remarkable success, and its consequential industry dominance are thoroughly investigated. Special attention is directed towards Nvidia's strategic development, technological advancements, influence on the industry, global footprint, and anticipated future implications.

Time Predictable Modeling Method for GPU Architecture with SIMT and Cache Miss Awareness

Graphics Processing Units (GPUs) are used to accelerate computing-intensive tasks, such as neural networks,data analysis, high-performance computing, etc. In the past decade or so, researchers have done a lot of work on GPU architecture and proposed a variety of theories and methods to study the microarchitectural characteristics of various GPUs. In this study, the GPU serves as a co-processor and works together with the CPU in an embedded real-time system to handle computationally intensive tasks. It models the architecture of the GPU and further considers it based on some excellent work. The SIMT mechanism and Cache-miss situation provide a more detailed analysis of the GPU architecture. In order to verify the GPU architecture model proposed in this article, 10 GPU kernel_task and an Nvidia GPU device were used to perform experiments. The experimental results showed that the minimum error between the kernel task execution time predicted by the GPU architecture model proposed in this article and the actual measured kernel task execution time was 3.80%, and the maximum error was 8.30%.

Plant Disease Identification Using Machine Learning Algorithms on Single-Board Computers in IoT Environments

This paper investigates the usage of machine learning (ML) algorithms on agricultural images with the aim of extracting information regarding the health of plants. More specifically, a custom convolutional neural network is trained on Google Colab using photos of healthy and unhealthy plants. The trained models are evaluated using various single-board computers (SBCs) that demonstrate different essential characteristics. Raspberry Pi 3 and Raspberry Pi 4 are the current mainstream SBCs that use their Central Processing Units (CPUs) for processing and are used for many applications for executing ML algorithms based on popular related libraries such as TensorFlow. NVIDIA Graphic Processing Units (GPUs) have a different rationale and base the execution of ML algorithms on a GPU that uses a different architecture than a CPU. GPUs can also implement high parallelization on the Compute Unified Device Architecture (CUDA) cores. Another current approach involves using a Tensor Processing Unit (TPU) processing unit carried by the Google Coral Dev TPU Board, which is an Application-Specific Integrated Circuit (ASIC) specialized for accelerating ML algorithms such as Convolutional Neural Networks (CNNs) via the usage of TensorFlow Lite. This study experiments with all of the above-mentioned devices and executes custom CNN models with the aim of identifying plant diseases. In this respect, several evaluation metrics are used, including knowledge extraction time, CPU utilization, Random Access Memory (RAM) usage, swap memory, temperature, current milli Amperes (mA), voltage (Volts), and power consumption milli Watts (mW).

ReARTSim: an ReRAM ARray Transient Simulator with GPU optimized runtime acceleration

The demand for computation driven by machine learning and deep learning applications has experienced exponential growth over the past five years (Sevilla et al 2022 2022 International Joint Conference on Neural Networks (IJCNN) (IEEE) pp 1-8), leading to a significant surge in computing hardware products. Meanwhile, this rapid increase has exacerbated the memory wall bottleneck within mainstream Von Neumann architectures (Hennessy and Patterson et al 2011 Computer architecture: a quantitative approach (Elsevier)). For instance, NVIDIA graphical processing units (GPUs) have gained nearly a 200x increase in fp32 computing power, transitioning from P100 to H100 in the last five years (NVIDIA Tesla P100 2023 (www.nvidia.com/en-us/data-center/tesla-p100/); NVIDIA H100 Tensor Core GPU 2023 (www.nvidia.com/en-us/data-center/h100/)), accompanied by a mere 8x scaling in memory bandwidth. Addressing the need to mitigate data movement challenges, process-in-memory designs, especially resistive random-access memory (ReRAM)-based solutions, have emerged as compelling candidates (Verma et al 2019 IEEE Solid-State Circuits Mag. 11 43–55; Sze et al 2017 Proc. IEEE 105 2295–329). However, this shift in hardware design poses distinct challenges at the design phase, given the limitations of existing hardware design tools. Popular design tools today can be used to characterize analog behavior via SPICE tools (PrimeSim HSPICE 2023 (www.synopsys.com/implementation-and-signoff/ams-simulation/primesim-hspice.html)), system and logical behavior using Verilog tools (VCS 2023 (www.synopsys.com/verification/simulation/vcs.html)), and mixed signal behavior through toolbox like CPPSIM (Meninger 2023 (www.cppsim.org/Tutorials/wideband_fracn_tutorial.pdf)). Nonetheless, the design of in-memory computing systems, especially those involving non-CMOS devices, presents a unique need for characterizing mixed-signal computing behavior across a large number of cells within a memory bank. This requirement falls beyond the scope of conventional design tools. In this paper, we bridge this gap by introducing the ReARTSim framework—a GPU-accelerated mixed-signal transient simulator for analyzing ReRAM crossbar array. This tool facilitates the characterization of analog circuit and device behavior on a large scale, while also providing enhanced simulation performance for complex algorithm analysis, sign-off, and verification.

Assessing the Impact of Compiler Optimizations on GPUs Reliability

Graphics Processing Units (GPUs) compilers have evolved in order to support general-purpose programming languages for multiple architectures. NVIDIA CUDA Compiler (NVCC) has many compilation levels before generating the machine code and applies complex optimizations to improve performance. These optimizations modify how the software is mapped in the underlying hardware; thus, as we show in this article, they can also affect GPU reliability. We evaluate the effects on the GPU error rate of the optimization flags applied at the NVCC Parallel Thread Execution (PTX) compiling phase by analyzing two NVIDIA GPU architectures (Kepler and Volta) and two compiler versions (NVCC 10.2 and 11.3). We compare and combine fault propagation analysis based on software fault injection, hardware utilization distribution obtained with application-level profiling, and machine instructions radiation-induced error rate measured with beam experiments. We consider eight different workloads and 144 combinations of compilation flags, and we show that optimizations can impact the GPUs’ error rate of up to an order of magnitude. Additionally, through accelerated neutron beam experiments on a NVIDIA Kepler GPU, we show that the error rate of the unoptimized GEMM (-O0 flag) is lower than the optimized GEMM’s (-O3 flag) error rate. When the performance is evaluated together with the error rate, we show that the most optimized versions (-O1 and -O3) always produce a higher amount of correct data than the unoptimized code (-O0).

Design and Development of a CCSDS 131.2-B Software-Defined Radio Receiver Based on Graphics Processing Unit Accelerators

In recent years, the number of Earth Observation missions has been exponentially increasing. Satellites dedicated to these missions usually embark with payloads that produce large amount of data and that need to be transmitted towards ground stations, in time-limited windows. Moreover, the noisy nature of the link between satellites and ground stations makes it hard to achieve reliable communication. To address these problems, a standard for a flexible advanced coding and modulation scheme for high-rate telemetry applications has been defined by the Consultative Committee for Space Data Systems (CCSDS). The defined standard, referred to as CCSDS 131.2-B, makes use of Serially Concatenated Convolutional Codes (SCCC) based on 27 ModCods to optimize transmission quality. A limiting factor in the adoption of this standard is represented by the complexity and the cost of the hardware required for developing high-performance receivers. In the last decade, the performance of software has grown due to the advancement of general-purpose processing hardware, leading to the development of many high-performance software systems even in the telecommunication sector. These are commonly referred to as Software-Defined Radio (SDR), indicating a radio communication system in which components that are usually implemented in hardware, by means of FPGAs or ASICs, are instead implemented in software, offering many advantages such as flexibility, modularity, extensibility, cheaper maintenance, and cost saving. This paper proposes the development of an SDR based on NVIDIA Graphics Processing Units (GPU) for implementing the receiver end of the CCSDS 131.2-B standard. At first, a brief description of the CCSDS 131.2-B standard is given, focusing on the architecture of the transmitter and receiver sides. Then, the receiver architecture is shown, giving an overview of its functional blocks and of the implementation choices made to optimize the processing of the signal, especially for the SCCC Decoder. Finally, the performance of the system is analyzed in terms of data-rate and error correction and compared with other SW systems to highlight the achieved improvements. The presented system has been demonstrated to be a perfect solution for CCSDS 131.2-B-compliant device testing and for its use in science missions, providing a valid low-cost alternative with respect to the state-of-the-art HW receivers.

GPU-accelerated non-dominated sorting genetic algorithm III for maximizing protein production

<abstract><p>Maximizing protein expression levels poses a major challenge in bioengineering. To increase protein expression levels, numerous factors, including codon bias, codon context bias, hidden stop codons, homologous recombination, suitable guanine-cytosine ratio, and hairpin loop structure, are crucial and quantified by six objective functions: CAI, CPB, HSC, HD, GC3, and SL. Optimizing these six objectives simultaneously constitutes a multi-objective optimization problem, aiming to identify the favorable Pareto solutions rather than a singular optimal solution. However, achieving satisfactory solutions requires numerous cycles and solutions, thus leading to a large number of functional evaluations. While there are frameworks for multi-objective optimization problems, they often lack efficient support for objective function computation in protein encoding. In this paper, we proposed a method to design a set of coding sequences (CDSs) based on non-dominated sorting genetic algorithm III (NSGA-III), accelerated using NVIDIA graphical processing units (GPUs). Experimental results indicated that our method is 15,454 times faster than the Pymoo framework and is evaluated using 100 solutions and 100 cycles. Since our GPU implementation facilitated the use of larger solutions and more cycles, we were able to design a superior set of CDSs by increasing solutions to 400 and cycles to 12,800. In addition, our NSGA-III-based method consistently surpassed the NSGA-II approach when the number of cycles exceeded 3200 by utilizing 100 solutions. Finally, we observed that a gradual reduction of the mutation probability as the number of cycles increased yielded better quality results than maintaining a fixed mutation probability.</p></abstract>

Run-3 Commissioning of CMS Online HLT reconstruction using GPUs

The software-based High-Level Trigger (HLT) of CMS reduces the data readout rate from 100 kHz (obtained from Level 1 trigger) to around 5 kHz. It makes use of all detector subsystems and runs a streamlined version of CMS reconstruction. Run-2 of the LHC saw the reconstruction algorithms run on a CPU farm. However, the need to have increased computational power as we approach the high luminosity phase of LHC demands the use of Graphical Processing Units (GPUs) to reign in the cost, size and power consumption of the HLT farm. Parallelization of the reconstruction algorithms, on top of the multi-threading functionality introduced in Run-2, allowed parts of the Hadronic Calorimeter (HCAL), Electromagnetic Calorimeter (ECAL) and Pixel Tracker reconstruction to be offloaded to NVIDIA GPUs. In order to ensure the reproducibility of physics results on any machine, the HLT configuration was designed to run seamlessly with and without GPUs, that is, the algorithms were automatically offloaded to a GPU when one was available and otherwise fell back to running on the CPU. This contribution will describe the development of GPU-based algorithms for the HLT and the challenges they presented, along with the comprehensive validation and commissioning activity undertaken by CMS to ensure the successful operations of the new HLT farm.

MRT-lattice Boltzmann simulation of MHD natural convection of Bingham nanofluid in a C-shaped enclosure with response surface analysis

This paper focuses on the natural convection of heat transfer using magnetohydrodynamic (MHD) Bingham nanofluid. Utilizing the multiple-relaxation-time (MRT) lattice Boltzmann method (LBM) within a C-shaped enclosure and the NVIDIA graphics processing unit (GPU)-based compute unified architecture (CUDA) C/C++ platform, the simulation is carried out numerically. Inside the cavity, the base fluid is water and the nanofluid is Al2O3. Boundary conditions are presented in accordance with the heated, cold, and adiabatic conditions present in the cavity's various walls. Several parameters including Bingham number (Bn=0,0.5,1,1.5,2), Rayleigh number (Ra=104,105,106), Hartmann number (Ha=0,10,20,30), and nanoparticle volume fraction (ϕ=0,0.01,0.02,0.03,0.04). The results of the numerical simulation are shown using streamlines and isotherms, velocity-temperature, Local Nusselt number, and average Nusselt number. From the obtained results it is found that for the variation of Bn, Ha, and ϕ with different Ra the rate of heat transfer decreases along the bottom wall and increases for the left and top walls. The average Nusselt number decreases while Bn, Ha rises. On the other contrary, the average Nu increases as ϕ increases. Response surface methodology (RSM) is added here to have better understanding of the effects of parameter used. RSM includes statistical table for the combination of data set and their graphs to understand the accuracy. Moreover, regression analysis shows how average Nu increases or decreases with the variation of different parameters. The C-shaped geometry provides an excellent option in heat exchanging or electronic cooling equipment the chip designing technology. This study is only for two-dimensional laminar flow.

Lightning: Leveraging DVFS-induced Transient Fault Injection to Attack Deep Learning Accelerator of GPUs

Graphics Processing Units (GPU) are widely used as deep learning accelerators because of its high performance and low power consumption. Additionally, it remains secure against hardware-induced transient fault injection attacks, a classic type of attacks that have been developed on other computing platforms. In this work, we demonstrate that well-trained machine learning models are robust against hardware fault injection attacks when the faults are generated randomly. However, we discover that these models have components, which we refer to as sensitive targets, that are vulnerable to faults. By exploiting this vulnerability, we propose the Lightning attack, which precisely strikes the model’s sensitive targets with hardware-induced transient faults based on the Dynamic Voltage and Frequency Scaling (DVFS). We design a sensitive targets search algorithm to find the most critical processing units of Deep Neural Network (DNN) models determining the inference results, and develop a genetic algorithm to automatically optimize the attack parameters for DVFS to induce faults. Experiments on three commodity Nvidia GPUs for four widely-used DNN models show that the proposed Lightning attack can reduce the inference accuracy by 69.1% on average for non-targeted attacks, and, more interestingly, achieve a success rate of 67.9% for targeted attacks.

Accelerating Dedispersion Using Many-core Architectures

Astrophysical radio signals are excellent probes of extreme physical processes that emit them. However, to reach Earth, electromagnetic radiation passes through the ionized interstellar medium, introducing a frequency-dependent time delay (dispersion) to the emitted signal. Removing dispersion enables searches for transient signals like fast radio bursts or repeating signals from isolated pulsars or those in orbit around other compact objects. The sheer volume and high resolution of data that next-generation radio telescopes will produce require high-performance computing solutions and algorithms to be used in time-domain data-processing pipelines to extract scientifically valuable results in real time. This paper presents a state-of-the-art implementation of brute force incoherent dedispersion on NVIDIA graphics-processing units and on Intel and AMD central-processing units. We show that our implementation is 4× faster (8-bit 8192 channels input) than other available solutions, and we demonstrate, using 11 existing telescopes, that our implementation is at least 20× faster than real time. This work is part of the AstroAccelerate package.

GPU-resident sparse direct linear solvers for alternating current optimal power flow analysis

Integrating renewable resources within the transmission grid at a wide scale poses significant challenges for economic dispatch as it requires analysis with more optimization parameters, constraints, and sources of uncertainty. This motivates the investigation of more efficient computational methods, especially those for solving the underlying linear systems, which typically take more than half of the overall computation time. In this paper, we present our work on sparse linear solvers that take advantage of hardware accelerators, such as graphical processing units (GPUs), and improve the overall performance when used within economic dispatch computations. We treat the problems as sparse, which allows for faster execution but also makes the implementation of numerical methods more challenging. We present the first GPU-native sparse direct solver that can execute on both AMD and NVIDIA GPUs. We demonstrate significant performance improvements when using high-performance linear solvers within alternating current optimal power flow (ACOPF) analysis. Furthermore, we demonstrate the feasibility of getting significant performance improvements by executing the entire computation on GPU-based hardware. Finally, we identify outstanding research issues and opportunities for even better utilization of heterogeneous systems, including those equipped with GPUs.

Speeding up the GENGA N-body integrator on consumer-grade graphics cards

Context. Graphics processing unit (GPU) computing has become popular due to the enormous calculation potential that can be harvested from a single card. The N-body integrator Gravitational ENcounters with GPU Acceleration (GENGA) is built to harvest the computing power from such cards, but it suffers a severe performance penalty on consumer-grade Nvidia GPUs due to their artificially truncated double precision performance. Aims. We aim to speed up GENGA on consumer-grade cards by harvesting their high single-precision performance. Methods. We modified GENGA to have the option to compute the mutual long-distance forces between bodies in single precision and tested this with five experiments. First, we ran a high number of simulations with similar initial conditions of on average 6600 fully self-gravitating planetesimals in both single and double precision to establish whether the outcomes were statistically different. These simulations were run on Tesla K20 cards. We supplemented this test with simulations that (i) began with a mixture of planetesimals and planetary embryos, (ii) planetesimal-driven giant planet migration, and (iii) terrestrial planet formation with a dissipating gas disc. All of these simulations served to determine the accuracy of energy and angular momentum conservation under various scenarios with single and double precision forces. Second, we ran the same simulation beginning with 40 000 self-gravitating planetesimals using both single and double precision forces on a variety of consumer-grade and Tesla GPUs to measure the performance boost of computing the long-range forces in single precision. Results. We find that there are no statistical differences when simulations are run with the gravitational forces in single or double precision that can be attributed to the force prescription rather than stochastic effects. The accumulations in uncertainty in energy are almost identical when running with single or double precision long-range forces. However, the uncertainty in the angular momentum using single rather than double precision long-range forces is about two orders of magnitude greater, but still very low. Running the simulations in single precision on consumer-grade cards decreases running time by a factor of three and becomes within a factor of three of a Tesla A100 GPU. Additional tuning speeds up the simulation by a factor of two across all types of cards. Conclusions. The option to compute the long-range forces in single precision in GENGA when using consumer-grade GPUs dramatically improves performance at a little penalty to accuracy. There is an additional environmental benefit because it reduces energy usage.

Efficient implementation of low-order-precision smoothed particle hydrodynamics

Smoothed particle hydrodynamics (SPH) method is widely accepted as a flexible numerical treatment for surface boundaries and interactions. High-resolution simulations of hydrodynamic events require high-performance computing (HPC). There is a need for an SPH code that runs efficiently on modern supercomputers involving accelerators such as NVIDIA or AMD graphics processing units. In this work, we applied half-precision, which is widely used in artificial intelligence, to the SPH method. However, improving HPC performance at such low-order precisions is a challenge. An as-is implementation with half-precision will have lower computational cost than that of float/double precision simulations, but also worsens the simulation accuracy. We propose a scaling and shifting method that maintains the simulation accuracy near the level of float/double precision. By examining the impact of half-precision on the simulation accuracy and time-to-solution, we demonstrated that the use of half-precision can improve the computational performance of SPH simulations for scientific purposes without sacrificing the accuracy. In addition, we demonstrated that the efficiency of half-precision depends on the architecture used.

IDEFIX: A versatile performance-portable Godunov code for astrophysical flows

Context. The exascale super-computers becoming available rely on hybrid energy-efficient architectures that involve an accelerator such as a graphics processing unit (GPU). Leveraging the computational power of these machines often means a significant rewrite of the numerical tools each time a new architecture becomes available. Aims. We present IDEFIX, a new code for astrophysical flows that relies on the KOKKOS meta-programming library to guarantee performance portability on a wide variety of architectures while keeping the code as simple as possible to the user. Methods. IDEFIX is based on a Godunov finite-volume method that solves the nonrelativistic hydrodynamical (HD) and magnetohy-drodynamical (MHD) equations on various grid geometries. IDEFIX includes a large choice of solvers and several additional modules (constrained transport, orbital advection, nonideal MHD), allowing users to address complex astrophysical problems. Results. IDEFIX has been successfully tested on Intel and AMD CPUs (up to 131 072 CPU cores on Irene-Rome at TGCC) as well as NVidia and AMD GPUs (up to 1024 GPUs on Adastra at CINES). IDEFIX achieves more than 108 cell s−1 in MHD on a single NVidia V100 GPU and 3 × 1011 cell s−1 on 256 Adastra nodes (1024 GPUs) with 95% parallelization efficiency (compared to single node). For the same problem, IDEFIX is up to six times more energy efficient on GPUs compared to Intel Cascade Lake CPUs. Conclusions. IDEFIX is now a mature exascale-ready open-source code that can be used on a large variety of astrophysical and fluid dynamics applications.

A Multi-GPU Aggregation-Based AMG Preconditioner for Iterative Linear Solvers

We present and release in open source format a sparse linear solver which efficiently exploits heterogeneous parallel computers. The solver can be easily integrated into scientific applications that need to solve large and sparse linear systems on modern parallel computers made of hybrid nodes hosting Nvidia Graphics Processing Unit (GPU) accelerators. The work extends previous efforts of some of the authors in the exploitation of a single GPU accelerator and proposes an implementation, based on the hybrid MPI-CUDA software environment, of a Krylov-type linear solver relying on an efficient Algebraic MultiGrid (AMG) preconditioner already available in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BootCMatchG</i> library. Our design for the hybrid implementation has been driven by the best practices for minimizing data communication overhead when multiple GPUs are employed, yet preserving the efficiency of the GPU kernels. Strong and weak scalability results of the new version of the library on well-known benchmark test cases are discussed. Comparisons with the Nvidia AmgX solution show a speedup, in the solve phase, up to 2.0x.

Program Analysis and Machine Learning–based Approach to Predict Power Consumption of CUDA Kernel

The General Purpose Graphics Processing Unit has secured a prominent position in the High-Performance Computing world due to its performance gain and programmability. Understanding the relationship between Graphics Processing Unit (GPU) power consumption and program features can aid developers in building energy-efficient sustainable applications. In this work, we propose a static analysis-based power model built using machine learning techniques. We have investigated six machine learning models across three NVIDIA GPU architectures: Kepler, Maxwell, and Volta with Random Forest, Extra Trees, Gradient Boosting, CatBoost, and XGBoost reporting favorable results. We observed that the XGBoost technique-based prediction model is the most efficient technique with an R 2 value of 0.9646 on Volta Architecture. The dataset used for these techniques includes kernels from different benchmarks suits, sizes, nature (e.g., compute-bound, memory-bound), and complexity (e.g., control divergence, memory access patterns). Experimental results suggest that the proposed solution can help developers precisely predict GPU applications power consumption using program analysis across GPU architectures. Developers can use this approach to refactor their code to build energy-efficient GPU applications.

Security Issues of GPUs and FPGAs for AI-powered near &amp; far Edge Services

Graphics Processing Units (GPUs) and Field Programmable Gate Arrays (FPGAs) are widely applied to cloud and embedded applications in which such devices are applied to near and far edge computing operations. This pool of available devices has a wide range of power/size specifications to support servers ranging from big data centres to small cloudlets, or even down to embedded systems and IoT boards. Overall, the most prominent devices and vendors in the market today are the following Xilinx for FPGA-based accelerators, Nvidia and AMD for GPUs, Intel for FPGA- /GPU-based accelerators. Decreasing the latency and increasing the throughput of Artificial Intelligence Functions (AIF), either for network automation or user applications, requires some sort of parallelization inside such purpose-built hardware acceleration. The AI@EDGE project is developing a Connect-Compute Platform (CCP) in which hardware accelerators (1 Nvidia GPU Tesla V100 (near edge device) and 1 Jetson AGX and 1 Jetson Nano (far edge devices), as well as 2 Xilinx FPGAs Alveo U280+U200 (near edge devices) and 1 Versal VCK190 and 2 Zynq ZCU104) are placed inside a server node and execute edge computing scenarios involving multiple nodes of diverse compute capabilities each, to test various integration approaches, to study orchestration techniques measure AIF deployment efficiency, all while developing certain FPGA/GPU code to accelerate representative AIFs of AI@EDGE. In this paper we compare the power/size/performance specifications of all accelerators and highlight the security issues associated with the cloud and embedded accelerators. This study presents the security issues announced by the vendors with the results of our tests and proposes tests and security functions (policies and objectives) which will be applied to the CCP to increase the security level of CCP. It also considers security issues related with the hardware set-up (accelerators inside server nodes) from the network point of view.

The Need for Speed and Energy Efficiency in Genome Analysis

The Need for Speed and Energy Efficiency in Genome Analysis

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains a challenge. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system side with a novel scheduling algorithm on the theory side. Through system and theory co-design, RTGPU achieves superior system throughput and real-time schedulability. In this paper, we start by building a model for the CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines, and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GTX1080Ti GPU systems. Our system side techniques can be applied on mainstream NVIDIA GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.