Tensor Contraction Research Articles

Tensor-Network Decoding Beyond 2D

Decoding algorithms based on approximate tensor-network (TN) contraction have proven tremendously successful in decoding two-dimensional (2D) local quantum codes such as toric or surface codes and color codes, effectively achieving optimal decoding accuracy. In this work, we introduce several techniques to generalize TN decoding to higher dimensions so that it can be applied to three-dimensional (3D) codes as well as 2D codes with noisy syndrome measurements (phenomenological noise or circuit-level noise). The 3D case is significantly more challenging than 2D, as the involved approximate tensor contraction is dramatically less well behaved than its 2D counterpart. Nonetheless, we numerically demonstrate that the decoding accuracy of our approach outperforms state-of-the-art decoders on the 3D surface code, both in the point and loop sectors, as well as for depolarizing noise. Our techniques could prove useful in near-term experimental demonstrations of quantum error correction, when decoding is to be performed off line and accuracy is of the utmost importance. To this end, we show how TN decoding can be applied to circuit-level noise and demonstrate that it outperforms the matching decoder on the rotated surface code. Published by the American Physical Society 2024

SparseAuto: An Auto-scheduler for Sparse Tensor Computations using Recursive Loop Nest Restructuring

Automated code generation and performance enhancements for sparse tensor algebra have become essential in many real-world applications, such as quantum computing, physical simulations, computational chemistry, and machine learning. General sparse tensor algebra compilers are not always versatile enough to generate asymptotically optimal code for sparse tensor contractions. This paper shows how to generate asymptotically better schedules for complex sparse tensor expressions using kernel fission and fusion. We present generalized loop restructuring transformations to reduce asymptotic time complexity and memory footprint. Furthermore, we present an auto-scheduler that uses a partially ordered set (poset)-based cost model that uses both time and auxiliary memory complexities to prune the search space of schedules. In addition, we highlight the use of Satisfiability Module Theory (SMT) solvers in sparse auto-schedulers to approximate the Pareto frontier of better schedules to the smallest number of possible schedules, with user-defined constraints available at compile-time. Finally, we show that our auto-scheduler can select better-performing schedules and generate code for them. Our results show that the auto-scheduler provided schedules achieve orders-of-magnitude speedup compared to the code generated by the Tensor Algebra Compiler (TACO) for several computations on different real-world tensors.

Efficient Mixed-Precision Matrix Factorization of the Inverse Overlap Matrix in Electronic Structure Calculations with AI-Hardware and GPUs.

In recent years, a new kind of accelerated hardware has gained popularity in the artificial intelligence (AI) community which enables extremely high-performance tensor contractions in reduced precision for deep neural network calculations. In this article, we exploit Nvidia Tensor cores, a prototypical example of such AI-hardware, to develop a mixed precision approach for computing a dense matrix factorization of the inverse overlap matrix in electronic structure theory, S-1. This factorization of S-1, written as ZZT = S-1, is used to transform the general matrix eigenvalue problem into a standard matrix eigenvalue problem. Here we present a mixed precision iterative refinement algorithm where Z is given recursively using matrix-matrix multiplications and can be computed with high performance on Tensor cores. To understand the performance and accuracy of Tensor cores, comparisons are made to GPU-only implementations in single and double precision. Additionally, we propose a nonparametric stopping criteria which is robust in the face of lower precision floating point operations. The algorithm is particularly useful when we have a good initial guess to Z, for example, from previous time steps in quantum-mechanical molecular dynamics simulations or from a previous iteration in a geometry optimization.

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Complete Contraction of Tensor and Its Application

This article presents a mathematical model for tensor complete contraction utilizing permutation group theory. The model helps in identifying the index class that dictates the complete contraction of even-order tensors. Additionally, the concept of a complete contraction closed loop is introduced along with a method for calculating the number of completely contracted images of any even-order tensor under any λ type permutation. The model is applied to investigate the algebraic structure of global conformal invariants on three-dimensional hypersurfaces. Moreover, it is shown that the generalized Willmore functional is the sole global conformal invariant in three-dimensional hypersurfaces when the difference is a constant multiple.

Encoding a Many-Body Potential Energy Surface into a Grid-Based Matrix Product Operator.

An efficient algorithm for compressing a given many-body potential energy surface (PES) of molecular systems into a grid-based matrix product operator (MPO) is proposed. The PES is once represented by a full-dimensional or truncated many-body expansion form, which is obtained by ab initio calculations at each grid mesh point, and then all terms in the expansion are compressed and merged into a single MPO while maintaining the bond dimension of the MPO as small as possible. It was shown that the ab initio PES of the H2CO was compressed by more than 2 orders of magnitude in the size of the site operators without loss of accuracy. By the use of grid basis, the tensor rank of the site operators of the MPO is reduced from four to three due to the diagonal nature of the position-dependent operators on grid basis, which significantly reduces the computational cost of the tensor contractions required in the real and imaginary time evolution of the matrix product state (MPS) wave functions with the grid-based MPO (Grid-MPO) Hamiltonian. Similar to other grid-based methods, Grid-MPO is easily applicable to any kinds of potentials of molecular systems, such as analytical empirical model potentials expressed by position operators and ab initio potentials, if the values at the grid points are available. Using the Grid-MPO combined with the MPS, we calculated the time correlation function of the Eigen cation to predict the infrared spectrum and compared with the experimental and the previous theoretical studies. The actual scaling with the size of systems was examined for the multidimensional Henon-Heiles Hamiltonian. It was shown that the method is considerably accelerated by the graphic processing unit (GPU) because the sizes of site operators were kept small and all tensors were able to be stored on the VRAM of a GPU.

Optimal evaluation of symmetry-adapted n-correlations via recursive contraction of sparse symmetric tensors

Abstract We present a comprehensive analysis of an algorithm for evaluating high-dimensional polynomials that are invariant (or equi-variant) under permutations and rotations. This task arises in the evaluation linear models as well as equivariant neural network models of many-particle systems. The theoretical bottleneck is the contraction of a high-dimensional symmetric and sparse tensor with a specific sparsity pattern that is directly related to the symmetries imposed on the polynomial. The sparsity of this tensor makes it challenging to construct a highly efficient evaluation scheme. The references [10, 11] introduced a recursive evaluation strategy that relied on a number of heuristics, but performed well in tests. In the present work, we propose an explicit construction of such a recursive evaluation strategy and show that it is in fact optimal in the limit of infinite polynomial degree.

Exploring Data Layout for Sparse Tensor Times Dense Matrix on GPUs

An important sparse tensor computation is sparse-tensor-dense-matrix multiplication (SpTM), which is used in tensor decomposition and applications. SpTM is a multi-dimensional analog to sparse-matrix-dense-matrix multiplication (SpMM). In this article, we employ a hierarchical tensor data layout that can unfold a multidimensional tensor to derive a 2D matrix, making it possible to compute SpTM using SpMM kernel implementations for GPUs. We compare two SpMM implementations to the state-of-the-art PASTA sparse tensor contraction implementation using: (1) SpMM with hierarchical tensor data layout; and, (2) unfolding followed by an invocation of cuSPARSE’s SpMM. Results show that SpMM can outperform PASTA 70.9% of the time, but none of the three approaches is best overall. Therefore, we use a decision tree classifier to identify the best performing sparse tensor contraction kernel based on precomputed properties of the sparse tensor.

Kropina Metrics with Isotropic Scalar Curvature via Navigation Data

Through an interesting physical perspective and a certain contraction of the Ricci curvature tensor in Finsler geometry, Akbar-Zadeh introduced the concept of scalar curvature for the Finsler metric. In this paper, we show that the Kropina metric is of isotropic scalar curvature if and only if F is an Einstein metric according to the navigation data. Moreover, we obtain the three-dimensional rigidity theorem for an Einstein–Kropina metric.

Accelerating Pythonic Coupled-Cluster Implementations: A Comparison Between CPUs and GPUs.

In this work, we benchmark several Python routines for time and memory requirements to identify the optimal choice of the tensor contraction operations available. We scrutinize how to accelerate the bottleneck tensor operations of Pythonic coupled-cluster implementations in the Cholesky linear algebra domain, utilizing a NVIDIA Tesla V100S PCIe 32GB (rev 1a) graphics processing unit (GPU). The NVIDIA compute unified device architecture API interacts with CuPy, an open-source library for Python, designed as a NumPy drop-in replacement for GPUs. Due to the limitations of video memory, the GPU calculations must be performed batch-wise. Timing results of some contractions containing large tensors are presented. The CuPy implementation leads to a factor of 10-16 speed-up of the bottleneck tensor contractions compared to computations on 36 central processing unit (CPU) cores. Finally, we compare example CCSD and pCCD-LCCSD calculations performed solely on CPUs to their CPU-GPU hybrid implementation, which leads to a speed-up of a factor of 3-4 compared to the CPU-only variant.

Code generation in ORCA: progress, efficiency and tight integration.

An improved version of ORCA's automated generator environment (ORCA-AGE II) is presented. The algorithmic improvements and the move to C++ as the programming language lead to a performance gain of up to two orders of magnitude compared to the previously developed PYTHON toolchain. Additionally, the restructured modular design allows for far more complex code engines to be implemented readily. Importantly, we have realised an extremely tight integration with the ORCA host program. This allows for a workflow in which only the wavefunction Ansatz is part of the source code repository while all actual high-level code is generated automatically, inserted at the appropriate place in the host program before it is compiled and linked together with the hand written code parts. This construction ensures longevity and uniform code quality. Furthermore the new developments allow ORCA-AGE II to generate parallelised production-level code for highly complex theories, such as fully internally contracted multireference coupled-cluster theory (fic-MRCC) with an enormous number of contributing tensor contractions. We also discuss the automated implementation of nuclear gradients for arbitrary theories. All these improvements enable the implementation of theories that are too complex for the human mind and also reduce development times by orders of magnitude. We hope that this work enables researchers to concentrate on the intellectual content of the theories they develop rather than be concerned with technical details of the implementation.

PyBEST: Improved functionality and enhanced performance

PyBEST: Improved functionality and enhanced performance

Gauging tensor networks with belief propagation

Effectively compressing and optimizing tensor networks requires reliable methods for fixing the latent degrees of freedom of the tensors, known as the gauge. Here we introduce a new algorithm for gauging tensor networks using belief propagation, a method that was originally formulated for performing statistical inference on graphical models and has recently found applications in tensor network algorithms. We show that this method is closely related to known tensor network gauging methods. It has the practical advantage, however, that existing belief propagation implementations can be repurposed for tensor network gauging, and that belief propagation is a very simple algorithm based on just tensor contractions so it can be easier to implement, optimize, and generalize. We present numerical evidence and scaling arguments that this algorithm is faster than existing gauging algorithms, demonstrating its usage on structured, unstructured, and infinite tensor networks. Additionally, we apply this method to improve the accuracy of the widely used simple update gate evolution algorithm.

Accelerating Coupled-Cluster Calculations with GPUs: An Implementation of the Density-Fitted CCSD(T) Approach for Heterogeneous Computing Architectures Using OpenMP Directives.

An algorithm is presented for the coupled-cluster singles, doubles, and perturbative triples correction [CCSD(T)] method based on the density fitting or the resolution-of-the-identity (RI) approximation for performing calculations on heterogeneous computing platforms composed of multicore CPUs and graphics processing units (GPUs). The directive-based approach to GPU offloading offered by the OpenMP application programming interface has been employed to adapt the most compute-intensive terms in the RI-CCSD amplitude equations with computational costs scaling as , , and (where NO and NV denote the numbers of correlated occupied and virtual orbitals, respectively) and the perturbative triples correction to execute on GPU architectures. The pertinent tensor contractions are performed using an accelerated math library such as cuBLAS or hipBLAS. Optimal strategies are discussed for splitting large data arrays into tiles to fit them into the relatively small memory space of the GPUs, while also minimizing the low-bandwidth CPU-GPU data transfers. The performance of the hybrid CPU-GPU RI-CCSD(T) code is demonstrated on pre-exascale supercomputers composed of heterogeneous nodes equipped with NVIDIA Tesla V100 and A100 GPUs and on the world's first exascale supercomputer named "Frontier", the nodes of which consist of AMD MI250X GPUs. Speedups within the range 4-8× relative to the recently reported CPU-only algorithm are obtained for the GPU-offloaded terms in the RI-CCSD amplitude equations. Applications to polycyclic aromatic hydrocarbons containing 16-66 carbon atoms demonstrate that the acceleration of the hybrid CPU-GPU code for the perturbative triples correction relative to the CPU-only code increases with the molecule size, attaining a speedup of 5.7× for the largest circumovalene molecule (C66H20). The GPU-offloaded code enables the computation of the perturbative triples correction for the C60 molecule using the cc-pVDZ/aug-cc-pVTZ-RI basis sets in 7 min on Frontier when using 12,288 AMD GPUs with a parallel efficiency of 83.1%.

A modified IHB method for nonlinear dynamic and thermal coupling analysis of rotor-bearing systems

This paper presents a modified incremental harmonic balance (IHB) method to analyze the nonlinear dynamic and thermal coupling characteristics of rotor-bearing systems after thermal balance. The IHB method is modified by tensor contraction and fast Fourier transform (FFT) to efficiently calculate the residual vector and Jacobian matrix. Nonlinear dynamic and thermal full coupling is implemented through the nonlinear part of the Jacobian matrix. The presented method achieves the simultaneous and efficient calculation of the motion and heat balance equations in the frequency domain according to the improvement measures. The effectiveness of the presented method is tested using the vibration response and temperature variation analysis for the coupled thermo-mechanical model of a rotor-ball bearing system. Furthermore, the performance comparisons show the good consistency of results between the presented method and the step-by-step (S-S) method, which is a partitioned iterative solution method for dealing with fully coupled problems in the time domain. The presented method has greater computational efficiency, more accurate temperature prediction, and better recognition of nonlinear phenomena in the system rather than the S-S method. Consequently, the presented method has a broad application prospect in solving fully coupled thermo-mechanical problems of more complex and higher dimensional rotor-bearing systems.

TAMM: Tensor algebra for many-body methods.

Tensor algebra operations such as contractions in computational chemistry consume a significant fraction of the computing time on large-scale computing platforms. The widespread use of tensor contractions between large multi-dimensional tensors in describing electronic structure theory has motivated the development of multiple tensor algebra frameworks targeting heterogeneous computing platforms. In this paper, we present Tensor Algebra for Many-body Methods (TAMM), a framework for productive and performance-portable development of scalable computational chemistry methods. TAMM decouples the specification of the computation from the execution of these operations on available high-performance computing systems. With this design choice, the scientific application developers (domain scientists) can focus on the algorithmic requirements using the tensor algebra interface provided by TAMM, whereas high-performance computing developers can direct their attention to various optimizations on the underlying constructs, such as efficient data distribution, optimized scheduling algorithms, and efficient use of intra-node resources (e.g., graphics processing units). The modular structure of TAMM allows it to support different hardware architectures and incorporate new algorithmic advances. We describe the TAMM framework and our approach to the sustainable development of scalable ground- and excited-state electronic structure methods. We present case studies highlighting the ease of use, including the performance and productivity gains compared to other frameworks.

Coupled cluster theory on modern heterogeneous supercomputers.

This study examines the computational challenges in elucidating intricate chemical systems, particularly through ab-initio methodologies. This work highlights the Divide-Expand-Consolidate (DEC) approach for coupled cluster (CC) theory-a linear-scaling, massively parallel framework-as a viable solution. Detailed scrutiny of the DEC framework reveals its extensive applicability for large chemical systems, yet it also acknowledges inherent limitations. To mitigate these constraints, the cluster perturbation theory is presented as an effective remedy. Attention is then directed towards the CPS (D-3) model, explicitly derived from a CC singles parent and a doubles auxiliary excitation space, for computing excitation energies. The reviewed new algorithms for the CPS (D-3) method efficiently capitalize on multiple nodes and graphical processing units, expediting heavy tensor contractions. As a result, CPS (D-3) emerges as a scalable, rapid, and precise solution for computing molecular properties in large molecular systems, marking it an efficient contender to conventional CC models.

Analytical differentiation of the articulated-body algorithm: a geometric multilinear approach

This paper presents a new geometric algorithm to efficiently compute the analytical differentiation of the Articulated-body Algorithm (ABA) with respect to the state variables. Despite the fact that ABA solves the forward-dynamics problem in linear time for branched-multibody systems, its explicit differentiation is not straightforward, and computationally demanding tensor products and contractions appear. The proposed algorithm makes use of Lie algebraic and multilinear operations to recursively exploit the underlying sparsity of the linearization problem. As a result, the arithmetic complexity is dramatically reduced without affecting the analytical solution. Since the linealization of forward dynamics is of great importance in robot-trajectory optimization, a differential dynamic programming solver is employed to demonstrate the performance of our algorithm with humanoid robot models such as NAO and HRP-2. In addition, we provide the computational cost with different robots using an optimized C++ implementation that can be found at https://github.com/garechav/geombd_crtp.git .

High-performance strategies for the recent MRSF-TDDFT in GAMESS.

Multiple ERI (Electron Repulsion Integral) tensor contractions (METC) with several matrices are ubiquitous in quantum chemistry. In response theories, the contraction operation, rather than ERI computations, can be the major bottleneck, as its computational demands are proportional to the multiplicatively combined contributions of the number of excited states and the kernel pre-factors. This paper presents several high-performance strategies for METC. Optimal approaches involve either the data layout reformations of interim density and Fock matrices, the introduction of intermediate ERI quartet buffer, and loop-reordering optimization for a higher cache hit rate. The combined strategies remarkably improve the performance of the MRSF (mixed reference spin flip)-TDDFT (time-dependent density functional theory) by nearly 300%. The results of this study are not limited to the MRSF-TDDFT method and can be applied to other METC scenarios.

Sparse tensor based nuclear gradients for periodic Hartree-Fock and low-scaling correlated wave function methods in the CP2K software package: A massively parallel and GPU accelerated implementation.

The development of novel double-hybrid density functionals offers new levels of accuracy and is leading to fresh insights into the fundamental properties of matter. Hartree-Fock exact exchange and correlated wave function methods, such as second-order Møller-Plesset (MP2) and direct random phase approximation (dRPA), are usually required to build such functionals. Their high computational cost is a concern, and their application to large and periodic systems is, therefore, limited. In this work, low-scaling methods for Hartree-Fock exchange (HFX), SOS-MP2, and direct RPA energy gradients are developed and implemented in the CP2K software package. The use of the resolution-of-the-identity approximation with a short range metric and atom-centered basis functions leads to sparsity, allowing for sparse tensor contractions to take place. These operations are efficiently performed with the newly developed Distributed Block-sparse Tensors (DBT) and Distributed Block-sparse Matrices (DBM) libraries, which scale to hundreds of graphics processing unit (GPU) nodes. The resulting methods, resolution-of-the-identity (RI)-HFX, SOS-MP2, and dRPA, were benchmarked on large supercomputers. They exhibit favorable sub-cubic scaling with system size, good strong scaling performance, and GPU acceleration up to a factor of 3. These developments will allow for double-hybrid level calculations of large and periodic condensed phase systems to take place on a more regular basis.