Tensor Cores Research Articles

RT-GNN: Accelerating Sparse Graph Neural Networks by Tensor-CUDA Kernel Fusion

Graph Neural Networks (GNNs) have achieved remarkable successes in various graph-based learning tasks, thanks to their ability to leverage advanced GPUs. However, GNNs currently face challenges arising from the concurrent use of advanced Tensor Cores (TCs) and CUDA Cores (CDs) in GPUs. These challenges are further exacerbated due to repeated, inefficient, and redundant aggregations in GNN that result from the high sparsity and irregular non-zero distribution of real-world graphs. We propose RT-GNN, a GNN framework based on the fusion of advanced TC and CD units, to eliminate the aforementioned redundancies by exploiting the properties of an adjacency matrix. First, a novel GNN representation technique, hierarchical embedding graph (HEG) is proposed to manage the intermediate aggregation results hierarchically, which can further avoid redundancy in intermediate aggregations elegantly. Next, to address the inherent sparsity of graphs, RT-GNN places the blocks (a.k.a tiles) in HEG onto TCs and CDs according to their sparsity by a new block-based row-wise multiplication approach, which assembles TCs and CDs to work concurrently. Experimental results demonstrate that HEG outperforms HAG by an average speedup of 19.3 × for redundancy elimination performance, especially up to 72 × speedup on the dataset of ARXIV. Moreover, for overall performance, RT-GNN outperforms state-of-the-art GNN frameworks (including DGL, HAG, GNNAdvisor, and TC-GNN) by an average factor of 3.1 × while maintaining or even improving the task accuracy.

GPU Parallel Computing Architectures : Unlocking the Power of Parallelism for High-Performance Applications

Graphics Processing Units (GPUs) have evolved from specialized graphics rendering hardware to become powerful parallel computing architectures, revolutionizing high-performance computing across diverse domains. This comprehensive article explores the fundamental principles of GPU parallel computing architectures, their design, and their impact on modern computational challenges. We begin by examining the multi-core structure, memory hierarchy, and data processing capabilities of GPUs, including the SIMD execution model and thread organization. The article then delves into prominent programming models like CUDA and OpenCL, discussing their features and comparative advantages. We explore how GPUs are leveraged for general-purpose computing in scientific simulations, machine learning, and big data analytics, while also addressing the challenges inherent in GPU parallel computing, such as data transfer bottlenecks and load balancing. Recent technological advancements, including tensor cores, unified memory architecture, and ray tracing acceleration, are analyzed for their transformative potential. The article concludes by examining future directions in GPU technology, including integration with emerging technologies like quantum computing, advancements in energy efficiency, and the potential impact on solving complex global challenges. Through this comprehensive analysis, we illustrate the pivotal role of GPU parallel computing architectures in shaping the future of high-performance computing and their potential to address some of the world's most pressing computational problems.

120 GOPS Photonic tensor core in thin-film lithium niobate for inference and in situ training

Photonics offers a transformative approach to artificial intelligence (AI) and neuromorphic computing by enabling low-latency, high-speed, and energy-efficient computations. However, conventional photonic tensor cores face significant challenges in constructing large-scale photonic neuromorphic networks. Here, we propose a fully integrated photonic tensor core, consisting of only two thin-film lithium niobate (TFLN) modulators, a III-V laser, and a charge-integration photoreceiver. Despite its simple architecture, it is capable of implementing an entire layer of a neural network with a computational speed of 120 GOPS, while also allowing flexible adjustment of the number of inputs (fan-in) and outputs (fan-out). Our tensor core supports rapid in-situ training with a weight update speed of 60 GHz. Furthermore, it successfully classifies (supervised learning) and clusters (unsupervised learning) 112 × 112-pixel images through in-situ training. To enable in-situ training for clustering AI tasks, we offer a solution for performing multiplications between two negative numbers.

Tensor Network State Algorithms on AI Accelerators.

We introduce novel algorithmic solutions for hybrid CPU-multiGPU tensor network state algorithms utilizing non-Abelian symmetries building on AI-motivated state-of-the-art hardware and software technologies. The presented numerical simulations on the FeMo cofactor, which plays a crucial role in converting atmospheric nitrogen to ammonia, are far beyond the scope of traditional approaches. Our large-scale SU(2) spin adapted density matrix renormalization group calculations up to bond dimension D = 216 on complete active space (CAS) size of 18 electrons in 18 orbitals [CAS(18, 18)] demonstrate that the current limit of exact solution, i.e. full-CI limit, can be achieved in fraction of time. Furthermore, benchmarks up to CAS(113, 76) demonstrate the utilization of NVIDIA's highly specialized AI accelerators via NVIDIA Tensor Cores, leading to performance around 115 TFLOPS on a single node supplied with eight NVIDIA A100 devices. As a consequence of reaching 71% of the full capacity of the hardware, the cubic scaling of computational time with bond dimension can be reduced to a linear form for a broad range of D values; thus, breaking the current computational limits of small CAS spaces in ab initio quantum chemistry and material science is becoming a reality. In comparison to strict U(1) implementations with matching accuracy, our solution has an estimated effective performance of 300-500 TFLOPS, which emphasizes the mutual need for both algorithmic and technological developments to push current frontiers on classical computation.

Acceleration of Tensor-Product Operations with Tensor Cores

In this paper, we explore the acceleration of tensor product operations in finite element methods, leveraging the computational power of the NVIDIA A100 GPU Tensor Cores. We provide an accessible overview of the necessary mathematical background and discuss our implementation strategies. Our study focuses on two common programming approaches for NVIDIA Tensor Cores: the C++ Warp Matrix Functions in nvcuda::wmma and the inline Parallel Thread Execution (PTX) instructions mma.sync.aligned . A significant focus is placed on the adoption of the versatile inline PTX instructions combined with a conflict-free shared memory access pattern, a key to unlocking superior performance. When benchmarked against traditional CUDA Cores, our approach yields a remarkable 2.3-fold increase in double precision performance, achieving 8 TFLOPS/s—45% of the theoretical maximum. Furthermore, in half-precision computations, numerical experiments demonstrate a fourfold enhancement in solving the Poisson equation using the flexible GMRES (FGMRES) method, preconditioned by a multigrid method in 3D. This is achieved while maintaining the same discretization error as observed in double precision computations. These results highlight the considerable benefits of using Tensor Cores for finite element operators with tensor products, achieving an optimal balance between computational speed and precision.

Investigating Register Cache Behavior: Implications for CUDA and Tensor Core Workloads on GPUs

Investigating Register Cache Behavior: Implications for CUDA and Tensor Core Workloads on GPUs

Efficient Mixed-Precision Matrix Factorization of the Inverse Overlap Matrix in Electronic Structure Calculations with AI-Hardware and GPUs.

In recent years, a new kind of accelerated hardware has gained popularity in the artificial intelligence (AI) community which enables extremely high-performance tensor contractions in reduced precision for deep neural network calculations. In this article, we exploit Nvidia Tensor cores, a prototypical example of such AI-hardware, to develop a mixed precision approach for computing a dense matrix factorization of the inverse overlap matrix in electronic structure theory, S-1. This factorization of S-1, written as ZZT = S-1, is used to transform the general matrix eigenvalue problem into a standard matrix eigenvalue problem. Here we present a mixed precision iterative refinement algorithm where Z is given recursively using matrix-matrix multiplications and can be computed with high performance on Tensor cores. To understand the performance and accuracy of Tensor cores, comparisons are made to GPU-only implementations in single and double precision. Additionally, we propose a nonparametric stopping criteria which is robust in the face of lower precision floating point operations. The algorithm is particularly useful when we have a good initial guess to Z, for example, from previous time steps in quantum-mechanical molecular dynamics simulations or from a previous iteration in a geometry optimization.

Interactive Volume Visualization via Multi-Resolution Hash Encoding Based Neural Representation.

Implicit neural networks have demonstrated immense potential in compressing volume data for visualization. However, despite their advantages, the high costs of training and inference have thus far limited their application to offline data processing and non-interactive rendering. In this article, we present a novel solution that leverages modern GPU tensor cores, a well-implemented CUDA machine learning framework, an optimized global-illumination-capable volume rendering algorithm, and a suitable acceleration data structure to enable real-time direct ray tracing of volumetric neural representations. Our approach produces high-fidelity neural representations with a peak signal-to-noise ratio (PSNR) exceeding 30 dB, while reducing their size by up to three orders of magnitude. Remarkably, we show that the entire training step can fit within a rendering loop, bypassing the need for pre-training. Additionally, we introduce an efficient out-of-core training strategy to support extreme-scale volume data, making it possible for our volumetric neural representation training to scale up to terascale on a workstation with an NVIDIA RTX 3090 GPU. Our method significantly outperforms state-of-the-art techniques in terms of training time, reconstruction quality, and rendering performance, making it an ideal choice for applications where fast and accurate visualization of large-scale volume data is paramount.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1–5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6–8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9–11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson’s disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Sparse Tensor Decomposition of Multi-Sensory Data for Fault Localization in Rotating Machinery Health Monitoring

Multi-sensor monitoring is prevalent in modern structural health monitoring (SHM) practice. As the number of sensors and sampling requirements increase, a monitoring sensor network can generate substantial data which are high-volume and high-dimensional, especially for large structure and machinery. In condition-based monitoring (CBM) of rotating machines, e.g., gas turbine, rotor fault diagnosis serves a significant role in the system reliability, safety, and efficiency, which helps reduce potential damage to the on-rotor structures and avoid catastrophic failures. For multi-channel vibration measurement, classical rotor diagnosis approaches typically involve data fusion techniques based on cross spectral analysis for identifying spectral correlation, e.g., cross power spectral density, and/or matrix analysis tool for dimensionality reduction, e.g., principal component analysis. The operation on vectors or matrices may limit their effectiveness for higher-order array or tensor data. To circumvent this limitation, the third order vibrational spectral tensor is generated by representing the multi-channel acceleration spectra as the three-way array. Through nonnegative Tucker decomposition (NTD), the spectral tensor is decomposed into multiple principal factors: the spectral factor, segmental factor, channel-wise factor, and the dense tensor core. The correlations across the characteristic spectral contents and the sensor channels are revealed by the factorization, which enables the diagnosis of rotor fault and facilitates fault localization by identifying the dominant channels of the characteristic spectral factor. The method is validated on an experimental rotor testbed where the faulty channel with crack, rub, or misalignment fault, is effectively localized via 4-channel vibration measurements, which presents a promising approach for multi-sensor fusion and fault diagnosis in rotating machinery health monitoring.

Tricking AI chips into simulating the human brain: A detailed performance analysis

In recent years, significant strides in Artificial Intelligence (AI) have led to various practical applications, primarily centered around training and deployment of deep neural networks (DNNs). These applications, however, require considerable computational resources, predominantly reliant on modern Graphics-Processing Units (GPUs). Yet, the quest for larger and faster DNNs has spurred the creation of specialized AI chips and efficient Machine-Learning (ML) software tools like TensorFlow and PyTorch have been developed for striking a balance between usability and performance. Simultaneously, the field of computational neuroscience shares a similar quest for increased computational power to simulate more extensive and detailed brain models, while also keeping usability high. Although GPUs have also entered this field, programming complexity remains high, resulting in cumbersome simulations. Inspired by AI progress, we introduce a workflow for easily accelerating brain simulations using TensorFlow and evaluate the performance of various, cutting-edge AI chips – including the Graphcore Intelligence-Processing Unit (IPU), GroqChip, Nvidia GPU with Tensor Cores, and Google Tensor-Processing Unit (TPU) – when simulating a biologically detailed as well as simpler brain models. Our model simulations explore the architectural tradeoffs of a modern-day CPU and these four AI platforms by varying computational density, memory requirements and floating-point numerical accuracy. Results show that the GroqChip achieves the best performance for small networks, yet is unable to simulate large-scale networks. At the scale of mammalian brains, the GPU, IPU and TPU achieve speedups ranging from 29x to 1,208x times over CPU runtimes. Remarkably, the TPU sets a new record for the largest, real-time simulation of the inferior-olivary nucleus in the brain. Reduced-accuracy floating-point implementations make some simulation results unreliable for brain research, notably for the GroqChip. Consequently, this work underscores the potential of ML libraries for accelerating brain simulations as well as the critical role of AI-chip numerical accuracy for biophysically realistic brain models.

Enhancing RAG Systems: A Survey of Optimization Strategies for Performance and Scalability

Retrieval Augmented Generation (RAG) systems offer significant advancements in natural language processing by combining large language models (LLMs) with external knowledge sources to improve factual accuracy and contextual relevance. However, the computational complexity of RAG pipelines presents challenges in terms of efficiency and scalability. This research paper conducts a comprehensive survey of optimization techniques across four key areas: tokenizer performance, encoder performance, vector database search strategies, and LLM agent integra- tion. The study explores approaches to accelerate tokenization using Rust-based implementations and investigates hardware optimizations like CUDA and Ten- sor cores to boost encoder efficiency. Additionally, it delves into algorithms and indexing strategies for efficient vector database searches and examines meth- ods for optimising the interaction between retrieved knowledge and LLM agents. By analysing recent research and evaluating various optimization strategies, this paper aims to provide valuable insights into enhancing the performance and practicality of RAG systems for real-world applications. Keywords: Retrieval-Augmented Generation, Performance optimization, Tokenization, WordPiece, Rust, FastTokenizer,Encoding, GPU, CUDA, Tensor Cores, Parallel Processing, Vector Databases, Indexing, HNSW, FAISS, DiskANN, LLM Agents, Prompt Engineering, Retriever

Generating Multi-Depth 3D Holograms Using a Fully Convolutional Neural Network.

Efficiently generating 3D holograms is one of the most challenging research topics in the field of holography. This work introduces a method for generating multi-depth phase-only holograms using a fully convolutional neural network (FCN). The method primarily involves a forward-backward-diffraction framework to compute multi-depth diffraction fields, along with a layer-by-layer replacement method (L2RM) to handle occlusion relationships. The diffraction fields computed by the former are fed into the carefully designed FCN, which leverages its powerful non-linear fitting capability to generate multi-depth holograms of 3D scenes. The latter can smooth the boundaries of different layers in scene reconstruction by complementing information of occluded objects, thus enhancing the reconstruction quality of holograms. The proposed method can generate a multi-depth 3D hologram with a PSNR of 31.8dB in just 90ms for a resolution of 2160×3840 on the NVIDIA Tesla A100 40G tensor core GPU. Additionally, numerical and experimental results indicate that the generated holograms accurately reconstruct clear 3D scenes with correct occlusion relationships and provide excellent depth focusing.

Accelerating range minimum queries with ray tracing cores

Over the past decade, GPU technology has undergone a notable transformation, evolving from pure general-purpose computation to the integration of application-specific integrated circuits (ASICs), including Tensor Cores and Ray Tracing (RT) cores. While these specialized GPU cores were initially developed to enhance specific domains like AI and real-time rendering, recent research has successfully harnessed their capabilities to expedite other tasks traditionally reliant on conventional GPU computing. One GPU task that is still yet to find its way into RT cores is the processing of range minimum queries (RMQs) in parallel, which is fundamental in fields such as information retrieval or pattern matching, among others. In this context, accelerating RMQs with RT cores would impact many of the applications that heavily rely on this task. In this work we present RTXRMQ, a new approach that can compute RMQs with RT cores. The main contribution is the proposal of a geometric solution for RMQ, where elements become triangles that are placed and shaped according to the element’s value and position in the array, respectively, such that the closest hit of a ray launched from a point given by the query parameters corresponds to the result of that query. Experimental results show that RTXRMQ is currently best suited for small query ranges relative to the input size, achieving up to 5× and 2.3× of speedup over parallel state of the art CPU and GPU approaches, respectively. For medium and large query ranges RTXRMQ is still slower than the state of the art GPU approach, but still competitive by being 2.5× and 4× faster than a state of the art CPU method running in parallel as well. Furthermore, performance scaling experiments across the latest RTX GPU architectures show that if the current RT core scaling trend continues, then RTXRMQ’s performance would scale at a higher rate than the other compared approaches, making it an attractive tool for future high performance applications that employ many batches of RMQs.

POAS: a framework for exploiting accelerator level parallelism in heterogeneous environments

In the era of heterogeneous computing, a new paradigm called accelerator level parallelism (ALP) has emerged. In ALP, accelerators are used concurrently to provide unprecedented levels of performance and energy efficiency. To reach that there are many problems to be solved, one of the most challenging being co-execution. In this paper, we present a new scheduling framework called POAS, a general method for providing co-execution to applications. Our proposal consists of four steps: predict, optimize, adapt and schedule. With POAS, an unseen application can be executed concurrently in ALP with little effort. We evaluate POAS on a heterogeneous environment consisting of CPUs, GPUs (CUDA cores), and XPUs (Tensor cores) on two different fields, namely linear algebra (matrix multiplication benchmark) and deep learning (convolution benchmark). Our experiments prove that POAS provides excellent performance and completes the tasks within a time very close to the optimal time for the hardware and applications used, with a negligible execution time overhead. Moreover, the POAS predictor performed exceptionally well, achieving very low RMSE values for both use cases. Therefore, POAS can be a valuable tool for fully exploiting ALP and improving overall performance over offloading in heterogeneous settings.

DGEMM on integer matrix multiplication unit

Deep learning hardware achieves high throughput and low power consumption by reducing computing precision and specializing in matrix multiplication. For machine learning inference, fixed-point value computation is commonplace, where the input and output values and the model parameters are quantized. Thus, many processors are now equipped with fast integer matrix multiplication units (IMMU). It is of significant interest to find a way to harness these IMMUs to improve the performance of HPC applications while maintaining accuracy. We focus on computing double-precision equivalent matrix multiplication using the Ozaki scheme, which computes a high-precision matrix multiplication by using lower-precision computing units, and show the advantages and disadvantages of using IMMU. The experiment using integer Tensor Cores shows that we can compute double-precision matrix multiplication faster than cuBLAS and an existing Ozaki scheme implementation on FP16 Tensor Cores on NVIDIA consumer GPUs. Furthermore, we demonstrate accelerating a quantum circuit simulation by up to 4.85 while maintaining the FP64 accuracy.

ConvKyber: Unleashing the Power of AI Accelerators for Faster Kyber with Novel Iteration-based Approaches

The remarkable performance capabilities of AI accelerators offer promising opportunities for accelerating cryptographic algorithms, particularly in the context of lattice-based cryptography. However, current approaches to leveraging AI accelerators often remain at a rudimentary level of implementation, overlooking the intricate internal mechanisms of these devices. Consequently, a significant number of computational resources is underutilized.In this paper, we present a comprehensive exploration of NVIDIA Tensor Cores and introduce a novel framework tailored specifically for Kyber. Firstly, we propose two innovative approaches that efficiently break down Kyber’s NTT into iterative matrix multiplications, resulting in approximately a 75% reduction in costs compared to the state-of-the-art scanning-based methods. Secondly, by reversing the internal mechanisms, we precisely manipulate the internal resources of Tensor Cores using assembly-level code instead of inefficient standard interfaces, eliminating memory accesses and redundant function calls. Finally, building upon our highly optimized NTT, we provide a complete implementation for all parameter sets of Kyber. Our implementation surpasses the state-of-the-art Tensor Core based work, achieving remarkable speed-ups of 1.93x, 1.65x, 1.22x and 3.55x for polyvec_ntt, KeyGen, Enc and Dec in Kyber-1024, respectively. Even when considering execution latency, our throughput-oriented full Kyber implementation maintains an acceptable execution latency. For instance, the execution latency ranges from 1.02 to 5.68 milliseconds for Kyber-1024 on R3080 when achieving the peak throughput.

ReARTSim: an ReRAM ARray Transient Simulator with GPU optimized runtime acceleration

The demand for computation driven by machine learning and deep learning applications has experienced exponential growth over the past five years (Sevilla et al 2022 2022 International Joint Conference on Neural Networks (IJCNN) (IEEE) pp 1-8), leading to a significant surge in computing hardware products. Meanwhile, this rapid increase has exacerbated the memory wall bottleneck within mainstream Von Neumann architectures (Hennessy and Patterson et al 2011 Computer architecture: a quantitative approach (Elsevier)). For instance, NVIDIA graphical processing units (GPUs) have gained nearly a 200x increase in fp32 computing power, transitioning from P100 to H100 in the last five years (NVIDIA Tesla P100 2023 (www.nvidia.com/en-us/data-center/tesla-p100/); NVIDIA H100 Tensor Core GPU 2023 (www.nvidia.com/en-us/data-center/h100/)), accompanied by a mere 8x scaling in memory bandwidth. Addressing the need to mitigate data movement challenges, process-in-memory designs, especially resistive random-access memory (ReRAM)-based solutions, have emerged as compelling candidates (Verma et al 2019 IEEE Solid-State Circuits Mag. 11 43–55; Sze et al 2017 Proc. IEEE 105 2295–329). However, this shift in hardware design poses distinct challenges at the design phase, given the limitations of existing hardware design tools. Popular design tools today can be used to characterize analog behavior via SPICE tools (PrimeSim HSPICE 2023 (www.synopsys.com/implementation-and-signoff/ams-simulation/primesim-hspice.html)), system and logical behavior using Verilog tools (VCS 2023 (www.synopsys.com/verification/simulation/vcs.html)), and mixed signal behavior through toolbox like CPPSIM (Meninger 2023 (www.cppsim.org/Tutorials/wideband_fracn_tutorial.pdf)). Nonetheless, the design of in-memory computing systems, especially those involving non-CMOS devices, presents a unique need for characterizing mixed-signal computing behavior across a large number of cells within a memory bank. This requirement falls beyond the scope of conventional design tools. In this paper, we bridge this gap by introducing the ReARTSim framework—a GPU-accelerated mixed-signal transient simulator for analyzing ReRAM crossbar array. This tool facilitates the characterization of analog circuit and device behavior on a large scale, while also providing enhanced simulation performance for complex algorithm analysis, sign-off, and verification.

GTCO: Graph and Tensor Co-Design for Transformer-Based Image Recognition on Tensor Cores

GTCO: Graph and Tensor Co-Design for Transformer-Based Image Recognition on Tensor Cores

Atomic Structure and Dynamics of Unusual and Wide‐Gap Phase‐Change Chalcogenides: A GeTe2 Case

Brain‐inspired computing, reconfigurable optical metamaterials, photonic tensor cores, and many other advanced applications require next‐generation phase change materials (PCMs) with better energy efficiency and a wider thermal and spectral range for reliable operations. Germanium ditelluride (GeTe2), with higher thermal stability and a larger bandgap compared to current benchmark PCMs, appears promising for THz metasurfaces and the controlled crystallization of atomically thin 2D materials. Using high‐energy X‐ray diffraction supported by first‐principles simulation, we investigate the atomic structure in semiconducting pulsed laser deposition films and metallic high‐temperature liquids. The results suggest that the structural and chemical metastability of GeTe2, leading to disproportionation into GeTe and Te, is related to high internal pressure during a semiconductor‐metal transition, presumably occurring in the supercooled melt. Similar phenomena are expected for canonical GeS2 and GeSe2 under high temperatures and pressures.This article is protected by copyright. All rights reserved.