Tensor Cores Research Articles

Quantum Perturbation Theory Using Tensor Cores and a Deep Neural Network.

Time-independent quantum response calculations are performed using Tensor cores. This is achieved by mapping density matrix perturbation theory onto the computational structure of a deep neural network. The main computational cost of each deep layer is dominated by tensor contractions, i.e., dense matrix-matrix multiplications, in mixed-precision arithmetics, which achieves close to peak performance. Quantum response calculations are demonstrated and analyzed using self-consistent charge density-functional tight-binding theory as well as coupled-perturbed Hartree-Fock theory. For linear response calculations, a novel parameter-free convergence criterion is presented that is well-suited for numerically noisy low-precision floating point operations and we demonstrate a peak performance of almost 200 Tflops using the Tensor cores of two Nvidia A100 GPUs.

Recovering single precision accuracy from Tensor Cores while surpassing the FP32 theoretical peak performance

Tensor Core is a mixed-precision matrix–matrix multiplication unit on NVIDIA GPUs with a theoretical peak performance of more than 300 TFlop/s on Ampere architectures. Tensor Cores were developed in response to the high demand of dense matrix multiplication from machine learning. However, many applications in scientific computing such as preconditioners for iterative solvers and low-precision Fourier transforms can exploit these Tensor Cores. To compute a matrix multiplication on Tensor Cores, we need to convert input matrices to half-precision, which results in loss of accuracy. To avoid this, we can keep the mantissa loss in the conversion using additional half-precision variables and use them for correcting the accuracy of matrix–matrix multiplication. Even with this correction, the use of Tensor Cores yields higher throughput compared to FP32 SIMT Cores. Nevertheless, the correcting capability of this method alone is limited, and the resulting accuracy cannot match that of a matrix multiplication on FP32 SIMT Cores. We address this problem and develop a high accuracy, high performance, and low power consumption matrix–matrix multiplication implementation using Tensor Cores, which exactly matches the accuracy of FP32 SIMT Cores while achieving superior throughput. The implementation is based on NVIDIA’s CUTLASS. We found that the key to achieving this accuracy is how to deal with the rounding inside Tensor Cores and underflow probability during the correction computation. Our implementation achieves 51 TFlop/s for a limited exponent range using FP16 Tensor Cores and 33 TFlop/s for full exponent range of FP32 using TF32 Tensor Cores on NVIDIA A100 GPUs, which outperforms the theoretical FP32 SIMT Core peak performance of 19.5 TFlop/s.

A tensor intermediate representation for machine learning systems

With the wide deployment of various machine learning algorithms, highly energy-efficient customized machine learning systems have gained popularity. The machine learning compilers are crucial to machine learning systems. The intermediate representation is the key to programming and compilation environments, and it connects the high-level programming language and the lower-level instruction set architectures. The current state-of-the-art intermediate representations are either oriented to high-level algorithms or classical processors based on scalar processing, but they cannot be effectively implemented on tensor-based machine learning systems. To address this problem, we propose a tensor intermediate representation for machine learning systems to improve programming productivity and performance. Concretely, we define a series of tensor types, tensor operations, and tensor memories and optimize the tensor processing based on these definitions. To validate our proposal, we extend the proposed tensor intermediate representation to the low-level scalar intermediate representation of TVM and perform experiments with Tensor Core on a typical machine learning system. Experimental results show that we explore optimizations that are not discovered in the original intermediate representation and achieve 1.62$\times$$\sim$2.85$\times$ performance improvement. Besides, the tensor intermediate representation improves the efficiency of programming by 5.46$\times$ on average.

GPU-accelerated simulations of quantum annealing and the quantum approximate optimization algorithm

We study large-scale applications using a GPU-accelerated version of the massively parallel Jülich universal quantum computer simulator (JUQCS–G). First, we benchmark JUWELS Booster, a GPU cluster with 3744 NVIDIA A100 Tensor Core GPUs. Then, we use JUQCS–G to study the relation between quantum annealing (QA) and the quantum approximate optimization algorithm (QAOA). We find that a very coarsely discretized version of QA, termed approximate quantum annealing (AQA), performs surprisingly well in comparison to the QAOA. It can either be used to initialize the QAOA, or to avoid the costly optimization procedure altogether. Furthermore, we study the scaling of the success probability when using AQA for problems with 30 to 40 qubits. We find that the case with the largest discretization error scales most favorably, surpassing the best result obtained from the QAOA.

Very fast finite element Poisson solvers on lower precision accelerator hardware: A proof of concept study for Nvidia Tesla V100

Recently, accelerator hardware in the form of graphics cards including Tensor Cores, specialized for AI, has significantly gained importance in the domain of high-performance computing. For example, NVIDIA’s Tesla V100 promises a computing power of up to 125 TFLOP/s achieved by Tensor Cores, but only if half precision floating point format is used. We describe the difficulties and discrepancy between theoretical and actual computing power if one seeks to use such hardware for numerical simulations, that is, solving partial differential equations with a matrix-based finite element method, with numerical examples. If certain requirements, namely low condition numbers and many dense matrix operations, are met, the indicated high performance can be reached without an excessive loss of accuracy. A new method to solve linear systems arising from Poisson’s equation in 2D that meets these requirements, based on “prehandling” by means of hier-archical finite elements and an additional Schur complement approach, is presented and analyzed. We provide numerical results illustrating the computational performance of this method and compare it to a commonly used (geometric) multigrid solver on standard hardware. It turns out that we can exploit nearly the full computational power of Tensor Cores and achieve a significant speed-up compared to the standard methodology without losing accuracy.

Squeeze: Efficient compact fractals for tensor core GPUs

This work presents Squeeze, an efficient compact fractal processing scheme for tensor core GPUs. By combining discrete-space transformations between compact and expanded forms, one can do data-parallel computation on a fractal with neighborhood access without needing to expand the fractal in memory. The space transformations are formulated as two GPU tensor-core accelerated thread maps, λ(ω) and ν(ω), which act as compact-to-expanded and expanded-to-compact space functions, respectively. The cost of the maps is O(log2logs(n)) time, with n being the side of a n×n embedding for the fractal in its expanded form, and s the linear scaling factor. The proposed approach works for any fractal that belongs to the Non-overlapping-Bounding-Boxes (NBB) class of discrete fractals, and can be extended to three dimensions as well. Experimental results using a discrete Sierpinski Triangle as a case study shows up to ∼12× of speedup and a memory reduction factor of up to ∼315× with respect to a GPU-based expanded-space bounding box approach. These results show that the proposed compact approach will allow the scientific community to efficiently tackle problems that up to now could not fit into GPU memory.

Accelerated AI Inference at CNN-Based Machine Vision in ASICs: A Design Approach

Machine Vision (MV) has become an increasingly popular technology in Industry 4.0 for advanced manufacturing processes. Specially, assembly lines require fast MV of the machine objects, assembly-functions, and operations to improve accuracy and quality. In order to achieve this, MV deployed customized Application Specific Integrated Circuits (ASICs). These ASICs designed to run Convolution Neural Network (CNNs) and perform AI inference. The processing needs high performance computing (HPC) of a large number of tensors with least energy spending in the ASICs. In the paper, we propose a design approach for efficient tensor processing in tensor cores for accelerated AI inference at CNN based MV in ASICs. The design approach deploys the small DNN (Deep Learning Neural Network) model SqueezeNext along with application of quantization, fast arithmetic reduction, tensor cores aware tuning, pruning, and fusion for the efficient AI inference in ASICs. Successful implementation of proposed design can provide a competitive advantage to industries in terms of quality and cost of product along with time saving in manufacturing process.

A Deep Learning Framework Performance Evaluation to Use YOLO in Nvidia Jetson Platform

Deep learning-based object detection technology can efficiently infer results by utilizing graphics processing units (GPU). However, when using general deep learning frameworks in embedded systems and mobile devices, processing functionality is limited. This allows deep learning frameworks such as TensorFlow-Lite (TF-Lite) and TensorRT (TRT) to be optimized for different hardware. Therefore, this paper introduces a performance inference method that fuses the Jetson monitoring tool with TensorFlow and TRT source code on the Nvidia Jetson AGX Xavier platform. In addition, central processing unit (CPU) utilization, GPU utilization, object accuracy, latency, and power consumption of the deep learning framework were compared and analyzed. The model is You Look Only Once Version4 (YOLOv4), and the dataset uses Common Objects in Context (COCO) and PASCAL Visual Object Classes (VOC). We confirmed that using TensorFlow results in high latency. We also confirmed that TensorFlow-TensorRT (TF-TRT) and TRT using Tensor Cores provide the most efficiency. However, it was confirmed that TF-Lite showed the lowest performance because it utilizes a GPU limited to mobile devices. Through this paper, we think that when developing deep learning-related object detection technology on the Nvidia Jetson platform or desktop environment, services and research can be efficiently conducted through measurement results.

Massive-Parallel Trajectory Calculations version 2.2 (MPTRAC-2.2): Lagrangian transport simulations on graphics processing units (GPUs)

Abstract. Lagrangian models are fundamental tools to study atmospheric transport processes and for practical applications such as dispersion modeling for anthropogenic and natural emission sources. However, conducting large-scale Lagrangian transport simulations with millions of air parcels or more can become rather numerically costly. In this study, we assessed the potential of exploiting graphics processing units (GPUs) to accelerate Lagrangian transport simulations. We ported the Massive-Parallel Trajectory Calculations (MPTRAC) model to GPUs using the open accelerator (OpenACC) programming model. The trajectory calculations conducted within the MPTRAC model were fully ported to GPUs, i.e., except for feeding in the meteorological input data and for extracting the particle output data, the code operates entirely on the GPU devices without frequent data transfers between CPU and GPU memory. Model verification, performance analyses, and scaling tests of the Message Passing Interface (MPI) – Open Multi-Processing (OpenMP) – OpenACC hybrid parallelization of MPTRAC were conducted on the Jülich Wizard for European Leadership Science (JUWELS) Booster supercomputer operated by the Jülich Supercomputing Centre, Germany. The JUWELS Booster comprises 3744 NVIDIA A100 Tensor Core GPUs, providing a peak performance of 71.0 PFlop s−1. As of June 2021, it is the most powerful supercomputer in Europe and listed among the most energy-efficient systems internationally. For large-scale simulations comprising 108 particles driven by the European Centre for Medium-Range Weather Forecasts' fifth-generation reanalysis (ERA5), the performance evaluation showed a maximum speed-up of a factor of 16 due to the utilization of GPUs compared to CPU-only runs on the JUWELS Booster. In the large-scale GPU run, about 67 % of the runtime is spent on the physics calculations, conducted on the GPUs. Another 15 % of the runtime is required for file I/O, mostly to read the large ERA5 data set from disk. Meteorological data preprocessing on the CPUs also requires about 15 % of the runtime. Although this study identified potential for further improvements of the GPU code, we consider the MPTRAC model ready for production runs on the JUWELS Booster in its present form. The GPU code provides a much faster time to solution than the CPU code, which is particularly relevant for near-real-time applications of a Lagrangian transport model.

Nvidia Hopper GPU and Grace CPU Highlights

At GTC 2022, Nvidia announced a new product family that aims to cover from small enterprise workloads through exascale high performance computing (HPC) and trillion-parameter AI models. This column highlights the most interesting features of their new Hopper graphical processing unit (GPU) and Grace central processing unit (CPU) computer chips and the Hopper product family. We also discuss some of the history behind Nvidia technologies and their most useful features for computational scientists, such as the Hopper DPX dynamic programming (DP) instruction set, increased number of SMs, and FP 8 tensor core availability. Also included are descriptions of the new Hopper Clustered SMs architecture and updated NVSwitch technologies that integrate their new ARM-based Grace CPU.

Inference-Optimized AI and High Performance Computing for Gravitational Wave Detection at Scale.

We introduce an ensemble of artificial intelligence models for gravitational wave detection that we trained in the Summit supercomputer using 32 nodes, equivalent to 192 NVIDIA V100 GPUs, within 2 h. Once fully trained, we optimized these models for accelerated inference using NVIDIA TensorRT. We deployed our inference-optimized AI ensemble in the ThetaGPU supercomputer at Argonne Leadership Computer Facility to conduct distributed inference. Using the entire ThetaGPU supercomputer, consisting of 20 nodes each of which has 8 NVIDIA A100 Tensor Core GPUs and 2 AMD Rome CPUs, our NVIDIA TensorRT-optimized AI ensemble processed an entire month of advanced LIGO data (including Hanford and Livingston data streams) within 50 s. Our inference-optimized AI ensemble retains the same sensitivity of traditional AI models, namely, it identifies all known binary black hole mergers previously identified in this advanced LIGO dataset and reports no misclassifications, while also providing a 3X inference speedup compared to traditional artificial intelligence models. We used time slides to quantify the performance of our AI ensemble to process up to 5 years worth of advanced LIGO data. In this synthetically enhanced dataset, our AI ensemble reports an average of one misclassification for every month of searched advanced LIGO data. We also present the receiver operating characteristic curve of our AI ensemble using this 5 year long advanced LIGO dataset. This approach provides the required tools to conduct accelerated, AI-driven gravitational wave detection at scale.

Accelerating approximate matrix multiplication for near-sparse matrices on GPUs

Although the matrix multiplication plays a vital role in computational linear algebra, there are few efficient solutions for matrix multiplication of the near-sparse matrices. The Sparse Approximate Matrix Multiply (SpAMM) is one of the algorithms to fill the performance gap neglected by traditional optimizations for dense/sparse matrix multiplication. However, existing SpAMM algorithms fail to exploit the performance potential of GPUs for acceleration. In this paper, we present cuSpAMM, the first parallel SpAMM algorithm optimized for multiple GPUs. Several performance optimizations have been proposed, including algorithm re-design to adapt to the thread parallelism, blocking strategies for memory access optimization, and the acceleration with the tensor core. In addition, we scale cuSpAMM to run on multiple GPUs with an effective load balance scheme. We evaluate cuSpAMM on both synthesized and real-world datasets on multiple GPUs. The experiment results show that cuSpAMM achieves significant performance speedup compared to vendor optimized cuBLAS and cuSPARSE libraries.

Broadband photonic tensor core with integrated ultra-low crosstalk wavelength multiplexers.

The integration of artificial intelligence (AI) systems in the daily life greatly increases the amount of data generated and processed. In addition to the large computational power required, the hardware needs to be compact and energy efficient. One promising approach tofulfill those requirements is phase-change material basedphotonic neuromorphic computing that enables in-memory computation and a high degree of parallelization. In the following, we present an optimized layout of a photonic tensor core (PTC) which is designed to perform real valued matrix vector multiplications and operates at telecommunication wavelengths. We deploy the well-studied phase-change material Ge2Sb2Te5 (GST) as an optical attenuator to perform single positive valued multiplications. In order to generalize the multiplication to arbitrary real factors, we develop a novel symmetric multiplication unit which directly includes a reference-computation branch. The variable GST attenuator enables a modulation depth of 5dB over a wavelength range of 100nm with a wavelength dependency below 0.8dB. The passive photonic circuit itself ensures equal coupling to the main-computation and reference-computation branch over the complete wavelength range. For the first time, we integrate wavelength multiplexers (MUX) together with a photonic crossbar array on-chip, paving the way towards fully integrated systems. The MUX are crucial for the PTC since they enable multiple computational channels in a single photonic crossbar array. We minimize the crosstalk between the channels by designing Bragg scattering based MUX. By cascading, we achieve an extinction ratio larger than 61dB while the insertion loss is below 1dB.

Tensorox: Accelerating GPU Applications via Neural Approximation on Unused Tensor Cores

Driven by the demands of deep learning, many hardware accelerators, including GPUs, have begun to include specialized tensor processing units to accelerate matrix operations. However, general-purpose GPU applications that have little or no large dense matrix operations cannot benefit from these tensor units. This article proposes Tensorox, a framework that exploits the half-precision tensor cores available on recent GPUs for approximable, non deep learning applications. In essence, a shallow neural network is trained based on the input-output mapping of the function to be approximated. The key innovation in our implementation is the use of the small and dimension-restricted tensor operations in Nvidia GPUs to run multiple instances of the approximation neural network in parallel. With the proper scaling and training methods, our approximation yielded an overall accuracy that is higher than naïvely running the original programs with half-precision. Furthermore, Tensorox allows for the runtime adjustment of the degree of approximation. For the 10 benchmarks we tested, we achieved speedups from 2× to 112× compared to the original in single precision floating point, while maintaining the error caused by the approximation to below 10 percent in most applications.

Interpretable AI forecasting for numerical relativity waveforms of quasicircular, spinning, nonprecessing binary black hole mergers

We present a deep-learning artificial intelligence model (AI) that is capable of learning and forecasting the late-inspiral, merger and ringdown of numerical relativity waveforms that describe quasicircular, spinning, nonprecessing binary black hole mergers. We used theNRHybSur3dq8 surrogate model to produce train, validation and test sets of $\ensuremath{\ell}=|m|=2$ waveforms that cover the parameter space of binary black hole mergers with mass ratios $q\ensuremath{\le}8$ and individual spins $|{s}_{{1,2}}^{z}|\ensuremath{\le}0.8$. These waveforms cover the time range $t\ensuremath{\in}[\ensuremath{-}5000\text{ }\text{ }\mathrm{M},130\text{ }\text{ }\mathrm{M}]$, where $t=0M$ marks the merger event, defined as the maximum value of the waveform amplitude. We harnessed the ThetaGPU supercomputer at the Argonne Leadership Computing Facility to train our AI model using a training set of 1.5 million waveforms. We used 16 NVIDIA DGX A100 nodes, each consisting of 8 NVIDIA A100 Tensor Core GPUs and 2 AMD Rome CPUs, to fully train our model within 3.5 h. Our findings show that artificial intelligence can accurately forecast the dynamical evolution of numerical relativity waveforms in the time range $t\ensuremath{\in}[\ensuremath{-}100\text{ }\text{ }\mathrm{M},130\text{ }\text{ }\mathrm{M}]$. Sampling a test set of 190,000 waveforms, we find that the average overlap between target and predicted waveforms is $\ensuremath{\gtrsim}99%$ over the entire parameter space under consideration. We also combined scientific visualization and accelerated computing to identify what components of our model take in knowledge from the early and late-time waveform evolution to accurately forecast the latter part of numerical relativity waveforms. This work aims to accelerate the creation of scalable, computationally efficient and interpretable artificial intelligence models for gravitational wave astrophysics.

TensorCrypto: High Throughput Acceleration of Lattice-Based Cryptography Using Tensor Core on GPU

Tensor core is a newly introduced hardware unit in NVIDIA GPU chips that allows matrix multiplication to be computed much faster than in the integer and floating-point units. In this paper, we show that for the first time, tensor core can be used to accelerate state-of-the-art lattice-based cryptosystems. We employed tensor core to speed up polynomial convolution, which is the most time consuming operation in lattice-based cryptosystems. Towards that aim, several parallel algorithms are proposed to allow the tensor core to handle flexible matrix sizes and ephemeral key pairs. Experimental results show that the polynomial convolution computed using the tensor core is at least <inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> faster than the version implemented with conventional integer units of the NVIDIA GPU. The proposed tensor-core-based polynomial convolution technique was applied to NTRU, one of the finalists in NIST post-quantum cryptography (PQC) standardization. It achieved <inline-formula> <tex-math notation="LaTeX">$2.02\times /1.98\times $ </tex-math></inline-formula> (encapsulation) and <inline-formula> <tex-math notation="LaTeX">$1.56\times /1.90\times $ </tex-math></inline-formula> (decapsulation) higher throughput on two parameter sets (ntruhps2048509 and ntruhps2048677), compared to the conventional integer-based implementations on a GPU. In particular, the proposed implementation techniques achieved throughput up to 793651 key encapsulations per second and 505051 decapsulations per second on a RTX2060 GPU. To demonstrate the flexibility of the proposed technique, we extend the implementation to other lattice-based cryptosystems that have a small modulus: LAC and two variant parameter sets in FrodoKEM. Considering that the IoT gateway devices and cloud servers need to handle massive connections from the sensor nodes, the proposed high throughput implementation on GPU is very useful in securing the IoT communication.

Future Scaling of Memory Hierarchy for Tensor Cores and Eliminating Redundant Shared Memory Traffic Using Inter-Warp Multicasting

The CUDA core of NVIDIA GPUs had been one of the most efficient computation units for parallel computing. However, recent rapid developments in deep neural networks demand an even higher level of computational performance. To meet this requirement, NVIDIA has introduced the Tensor core in recent generations. However, their impressive enhancements in computational performance have newly brought high pressure on the memory hierarchy. In this paper, first we identify the required memory bandwidth in the memory hierarchy as the computational performance increases in actual GPU hardware. Through a comparison of the CUDA core and the Tensor core in V100, we find that the tremendous performance increase of the Tensor core requires much higher memory bandwidth than that in the CUDA core. Moreover, we thoroughly investigate memory bandwidth requirement over Tensor core generations of V100, RTX TITAN, and A100. Lastly, we analyze a hypothetical next-generation Tensor core introduced by NVIDIA through a GPU simulation, through which we propose an inter-warp multicasting microarchitecture that reduces redundant shared memory (SMEM) traffic during the GEMM process. Our evaluation shows that inter-warp multicasting reduces the SMEM bandwidth pressure by 33% and improves the performance by 19% on average in all layers of ResNet-152 and BERT-Large.

Generative Modeling Via Tensor Train Sketching

In this paper we introduce a sketching algorithm for constructing a tensor train representation of a probability density from its samples. Our method deviates from the standard recursive SVD-based procedure for constructing a tensor train. Instead we formulate and solve a sequence of small linear systems for the individual tensor train cores. This approach can avoid the curse of dimensionality that threatens both the algorithmic and sample complexities of the recovery problem. Specifically, for Markov models, we prove that the tensor cores can be recovered with a sample complexity that is constant with respect to the dimension. Finally, we illustrate the performance of the method with several numerical experiments.

A Proof of Concept for Very Fast Finite Element Poisson Solvers on Accelerator Hardware

AbstractIt is demonstrated that modern accelerator hardware specialized in AI, e.g., “next gen GPUs” equipped with Tensor Cores, can be profitably used in finite element simulations by means of a new hardware‐oriented method to solve linear systems arising from Poisson's equation in 2D. We consider the NVIDIA Tesla V100 Tensor Core GPU with a peak performance of 125 TFLOP/s, that is only achievable in half precision and if operations with high arithmetic intensity, such as dense matrix multiplications, are executed, though. Its computing power can be exploited to a great extent by the new method based on “prehandling” without loss of accuracy. We obtain a significant reduction of computing time compared to a standard geometric multigrid solver on standard x64 hardware.

The Tensor-Core Correlator

Correlators are key components of radio telescopes as they combine the data from all receivers. They are characterized by high data rates, impressive compute loads, and the necessity to process data in real time. Often, these instruments use graphics processing units (GPUs) to correlate the incoming data streams as GPUs are fast, energy efficient, flexible, and programmable with a reasonable programming effort. Recent GPUs are equipped with a new technology, called “tensor cores”, to perform specific (typically artificial intelligence) workloads an order of magnitude more efficiently than regular GPU cores. This paper introduces the Tensor-Core Correlator, a GPU library that exploits this technology for signal processing and allows a GPU to correlate signals five to ten times faster and more energy efficiently than traditional state-of-the-art GPU correlators. The library hides the complexity of the use of tensor cores and can be easily integrated into the GPU pipelines of existing and future instruments, leading to a significant reduction in costs and energy consumption.