Quantized Tensor Train Research Articles

TetraFEM: Numerical Solution of Partial Differential Equations Using Tensor Train Finite Element Method

In this paper, we present a methodology for the numerical solving of partial differential equations in 2D geometries with piecewise smooth boundaries via finite element method (FEM) using a Quantized Tensor Train (QTT) format. During the calculations, all the operators and data are assembled and represented in a compressed tensor format. We introduce an efficient assembly procedure of FEM matrices in the QTT format for curvilinear domains. The features of our approach include efficiency in terms of memory consumption and potential expansion to quantum computers. We demonstrate the correctness and advantages of the method by solving a number of problems, including nonlinear incompressible Navier–Stokes flow, in differently shaped domains.

Estimates for the Quantized Tensor Train Ranks for the Power Functions

Estimates for the Quantized Tensor Train Ranks for the Power Functions

Tensor networks for solving the time-independent Boltzmann neutron transport equation

Tensor network techniques, known for their low-rank approximation ability that breaks the curse of dimensionality, are emerging as a foundation of new mathematical methods for ultra-fast numerical solutions of high-dimensional Partial Differential Equations (PDEs). Here, we present a mixed Tensor Train (TT)/Quantized Tensor Train (QTT) approach for the numerical solution of time-independent Boltzmann Neutron Transport equations (BNTEs) in Cartesian geometry. Discretizing a realistic three-dimensional (3D) BNTE by (i) diamond differencing in space, (ii) multigroup-in-energy, and (iii) discrete ordinates collocation in angle leads to large generalized eigenvalue problems that generally require a matrix-free approach and large computer clusters. Starting from this discretization, we construct a TT representation of the PDE fields and discrete operators, followed by a QTT representation of the TT cores. We then solve the tensorized generalized eigenvalue problem using a fixed-point scheme with tensor network optimization techniques. We validate our approach by applying the method to two examples of 3D neutron transport problems, currently solved by the Los Alamos National Laboratory PARallel TIme-dependent SN (PARTISN) solver.1 We demonstrate that our TT/QTT method, executed on a standard desktop computer, leads to large compression. This allows for the storage of terrabyte-sized neutron angular flux eigenvectors in megabytes. Additionally, we create megabyte-sized full access TT representations of yottabyte-sized transport matrix operators. By leveraging the TT operators and solution methods, we obtain a 7500 times speedup when compared to the PARTISN solution time with an error of less than 10−5.

Editorial: Tensor numerical methods and their application in scientific computing and data science

Editorial: Tensor numerical methods and their application in scientific computing and data science

Denoising convolution algorithms and applications to SAR signal processing

Convolutions are one of the most important operations in signal processing. They often involve large arrays and require significant computing time. Moreover, in practice, the signal data to be processed by convolution may be corrupted by noise. In this paper, we introduce a new method for computing the convolutions in the quantized tensor train (QTT) format and removing noise from data using the QTT decomposition. We demonstrate the performance of our method using a common mathematical model for synthetic aperture radar (SAR) processing that involves a sinc kernel and present the entire cost of decomposing the original data array, computing the convolutions, and then reformatting the data back into full arrays.

A Quantized Tensor Train Method for High-Frequency Scattering Problems Involving Heterogeneous Dielectric Layers

We present a new numerical scheme to solve efficiently scattering problems involving an elongated and flat heterogeneous dielectric material assumed to be invariant along a direction of space. The technique consists of compressing the integral operators of an integro-differential formulation with a so-called quantized tensor train (QTT) algorithm whose use is rather original in this context. We show that it allows to compute and store operators with a notably small memory footprint while having at the same time a fast matrix-vector product (MVP) leading to a competitive method compared with the more classical <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {H}$ </tex-math></inline-formula> -matrix approach.

Tensor rank bounds and explicit QTT representations for the inverses of circulant matrices

AbstractIn this paper, we are concerned with the inversion of circulant matrices and their quantized tensor‐train (QTT) structure. In particular, we show that the inverse of a complex circulant matrix , generated by the first column of the form admits a QTT representation with the QTT ranks bounded by . Under certain assumptions on the entries of , we also derive an explicit QTT representation of . The latter can be used, for instance, to overcome stability issues arising when numerically solving differential equations with periodic boundary conditions in the QTT format.

Tensor rank bounds for point singularities in ℝ3

We analyze rates of approximation by quantized, tensor-structured representations of functions with isolated point singularities in ℝ3. We consider functions in countably normed Sobolev spaces with radial weights and analytic- or Gevrey-type control of weighted semi-norms. Several classes of boundary value and eigenvalue problems from science and engineering are discussed whose solutions belong to the countably normed spaces. It is shown that quantized, tensor-structured approximations of functions in these classes exhibit tensor ranks bounded polylogarithmically with respect to the accuracy ε ∈ (0,1) in the Sobolev space H1. We prove exponential convergence rates of three specific types of quantized tensor decompositions: quantized tensor train (QTT), transposed QTT and Tucker QTT. In addition, the bounds for the patchwise decompositions are uniform with respect to the position of the point singularity. An auxiliary result of independent interest is the proof of exponential convergence of hp-finite element approximations for Gevrey-regular functions with point singularities in the unit cube Q = (0,1)3. Numerical examples of function approximations and of Schrödinger-type eigenvalue problems illustrate the theoretical results.

Low-rank tensor approximation of singularly perturbed boundary value problems in one dimension

We derive rank bounds on the quantized tensor train (QTT) compressed approximation of singularly perturbed reaction diffusion boundary value problems in one dimension. Specifically, we show that, independently of the scale of the singular perturbation parameter, a numerical solution with accuracy 0<varepsilon <1 can be represented in the QTT format with a number of parameters that depends only polylogarithmically on varepsilon . In other words, QTT-compressed solutions converge exponentially fast to the exact solution, with respect to a root of the number of parameters. We also verify the rank bound estimates numerically and overcome known stability issues of the QTT-based solution of partial differential equations (PDEs) by adapting a preconditioning strategy to obtain stable schemes at all scales. We find, therefore, that the QTT-based strategy is a rapidly converging algorithm for the solution of singularly perturbed PDEs, which does not require prior knowledge on the scale of the singular perturbation and on the shape of the boundary layers.

QTTNet: Quantized tensor train neural networks for 3D object and video recognition

Relying on the rapidly increasing capacity of computing clusters and hardware, convolutional neural networks (CNNs) have been successfully applied in various fields and achieved state-of-the-art results. Despite these exciting developments, the huge memory cost is still involved in training and inferring a large-scale CNN model and makes it hard to be widely used in resource-limited portable devices. To address this problem, we establish a training framework for three-dimensional convolutional neural networks (3DCNNs) named QTTNet that combines tensor train (TT) decomposition and data quantization together for further shrinking the model size and decreasing the memory and time cost. Through this framework, we can fully explore the superiority of TT in reducing the number of trainable parameters and the advantage of quantization in decreasing the bit-width of data, particularly compressing 3DCNN model greatly with little accuracy degradation. In addition, due to the low bit quantization to all parameters during the inference process including TT-cores, activations, and batch normalizations, the proposed method naturally takes advantage in memory and time cost. Experimental results of compressing 3DCNNs for 3D object and video recognition on ModelNet40, UCF11, and UCF50 datasets verify the effectiveness of the proposed method. The best compression ratio we have obtained is up to nearly 180× with competitive performance compared with other state-of-the-art researches. Moreover, the total bytes of our QTTNet models on ModelNet40 and UCF11 datasets can be 1000× lower than some typical practices such as MVCNN.

Building Z-Permuted Matrices in the QTT Format

The paper presents a method for building matrices in the QTT format, the columns and rows of which are reordered in a special way, by z-permutation. To obtain a matrix in this permutation, a new operation in the QTT (Quantized Tensor Train) format, z-kron, is introduced. This reordering allows one to reduce the QTT ranks of the approximation of the stiffness matrix, which makes it possible to accelerate the convergence of the numerical solution of the system. For example, when solving the Dirichlet problem for Poisson’s equation by the finite element method (FEM), where the QTT format is used to store the coefficient matrix, reordering the rows and columns in a coefficients matrix with dimensions $$n \times n$$ , where $$n = {{4}^{d}}$$ , makes it possible to prevent the exponential in $$d$$ growth of ranks.

An adaptive solution to the chemical master equation using quantized tensor trains with sliding windows

To cope with an extremely large or even infinite state space when solving the chemical master equation in biological problems, a potent strategy is to restrict to a finite state projection (FSP) and represent the transition matrix and probability vector in quantized tensor train (QTT) format, leading to savings in storage while retaining accuracy. In an earlier adaptive FSP–QTT algorithm, the multidimensional state space was downsized and kept in the form of a hyper rectangle that was updated when needed by selectively doubling some of its side dimensions. However, this could result in a much larger state space than necessary, with the effect of hampering both the execution time and stepping scheme. In this work, we improve the algorithm by enabling sliding windows that can dynamically slide, shrink or expand, with updates driven by a number of stochastic simulation algorithm trajectories. The ensuing state space is a considerably reduced hyper rectangle containing only the most probable states at each time step. Three numerical experiments of varying difficulty are performed to compare our approach with the original adaptive FSP–QTT algorithm.

QTT-isogeometric solver in two dimensions

Elliptical PDEs are at the core of many computational problems. Sometimes it is necessary to solve them on fine meshes, which entail huge memory footprints and low computational speeds. We provide a method to solve elliptical PDEs on fine grids with lowered memory consumption and improved convergence.This paper considers one typical elliptical PDE – the Poisson equation on various polygonal domains with Dirichlet boundary conditions. The Finite Element Method (FEM) is used for numerical solution. FEM approximates a two-dimensional PDE as a system of linear equations Au=f. For an n-by-n mesh grid the sparse representation of A has the size of O(n2). We replace the sparse matrix representation with a Quantized Tensor Train (QTT) representation to obtain O((log⁡n)α) time and memory complexity to construct both A and f. This is ensured by constant-bounded QTT approximation ranks. AMEn solver is used on the final linear system, and its iterations are faster for low rank QTT approximation. To avoid rank growth caused by the intrinsic structure of A we introduce a new operation z-kron which constructs a matrix with rows and columns permuted into so called z-order. An algorithm to construct A in z-order directly in QTT with O(1) ranks w.r.t. n is provided.The proposed method is used to solve the Poisson equation on two different polygonal domains. Experiments show that our approach significantly improves memory consumption and speed over classical sparse-matrix based partial differential equation solvers like FEniCS.

Tensor Train Accelerated Solvers for Nonsmooth Rigid Body Dynamics

Abstract In the last two decades, increased need for high-fidelity simulations of the time evolution and propagation of forces in granular media has spurred a renewed interest in the discrete element method (DEM) modeling of frictional contact. Force penalty methods, while economic and widely accessible, introduce artificial stiffness, requiring small time steps to retain numerical stability. Optimization-based methods, which enforce contacts geometrically through complementarity constraints leading to a differential variational inequality problem (DVI), allow for the use of larger time steps at the expense of solving a nonlinear complementarity problem (NCP) each time-step. We review the latest efforts to produce solvers for this NCP, focusing on its relaxation to a cone complementarity problem (CCP) and solution via an equivalent quadratic optimization problem with conic constraints. We distinguish between first-order methods, which use only gradient information and are thus linearly convergent and second-order methods, which rely on a Newton type step to gain quadratic convergence and are typically more robust and problem-independent. However, they require the approximate solution of large sparse linear systems, thus losing their competitive advantages in large scale problems due to computational cost. In this work, we propose a novel acceleration for the solution of Newton step linear systems in second-order methods using low-rank compression based fast direct solvers, leveraging on recent direct solver techniques for structured linear systems arising from differential and integral equations. We employ the quantized tensor train (QTT) decomposition to produce efficient approximate representations of the system matrix and its inverse. This provides a versatile and robust framework to accelerate its solution using this inverse in a direct or a preconditioned iterative method. We demonstrate compressibility of the Newton step matrices in primal dual interior point (PDIP) methods as applied to the multibody dynamics problem. Using a number of numerical tests, we demonstrate that this approach displays sublinear scaling of precomputation costs, may be efficiently updated across Newton iterations as well as across simulation time steps, and leads to a fast, optimal complexity solution of the Newton step. This allows our method to gain an order of magnitude speedups over state-of-the-art preconditioning techniques for moderate to large-scale systems, hence mitigating the computational bottleneck of second-order methods.

Computing the density of states for optical spectra of molecules by low-rank and QTT tensor approximation

In this paper, we introduce a new interpolation scheme to approximate the density of states (DOS) for a class of rank-structured matrices with application to the Tamm–Dancoff approximation (TDA) of the Bethe–Salpeter equation (BSE). The presented approach for approximating the DOS is based on two main techniques. First, we propose an economical method for calculating the traces of parametric matrix resolvents at interpolation points by taking advantage of the block-diagonal plus low-rank matrix structure described in [6,3] for the BSE/TDA problem. This allows us to overcome the computational difficulties of the traditional schemes since we avoid the construction of the matrix inverse and hence the need of stochastic sampling. Second, we show that a regularized or smoothed DOS discretized on a fine grid of size N can be accurately represented in a low rank quantized tensor train (QTT) format that can be determined through a least squares fitting procedure. The QTT tensor provides good approximation properties for strictly oscillating DOS functions with multiple gaps, in contrast to interpolation by problem independent functions like polynomials, trigonometric functions, etc. Moreover, the QTT approximant requires asymptotically much fewer (e.g., O(log⁡N)) functional calls compared with the full grid size N. Numerical tests indicate that the QTT approach yields accurate recovery of DOS associated with problems that contain relatively large spectral gaps. The QTT tensor rank only weakly depends on the size of a molecular system that paves the way for treatment of large-scale spectral problems.

Fast Low-Rank Solution of the Multidimensional Hyperbolic Problems

In this paper, the numerical solution of multidimensional hyperbolic problems is discussed with the quantized tensor train (QTT)-approximation methods. Three schemes are proposed. First, an improved implicit time iteration scheme is presented by using the two-site density matrix renormalization group (DMRG) algorithm to solve a linear system at each time step. Second, the time is considered as an independent dimension, and a discretization of the whole differential equation is introduced with all spatial and time dimensions connected in one big global linear system. Then the problem is solved in the QTT-format. The third scheme is to solve the global system by splitting the global time interval into several subintervals. The numerical experiments, with these three schemes applied to the wave equation, show that the complexity of the first scheme is linear while that of the second and third schemes is log-linear in both time and spatial grid points.

On compact vector formats in the solution of the chemical master equation with backward differentiation

SummaryA stochastic chemical system with multiple types of molecules interacting through reaction channels can be modeled as a continuous‐time Markov chain with a countably infinite multidimensional state space. Starting from an initial probability distribution, the time evolution of the probability distribution associated with this continuous‐time Markov chain is described by a system of ordinary differential equations, known as the chemical master equation (CME). This paper shows how one can solve the CME using backward differentiation. In doing this, a novel approach to truncate the state space at each time step using a prediction vector is proposed. The infinitesimal generator matrix associated with the truncated state space is represented compactly, and exactly, using a sum of Kronecker products of matrices associated with molecules. This exact representation is already compact and does not require a low‐rank approximation in the hierarchical Tucker decomposition (HTD) format. During transient analysis, compact solution vectors in HTD format are employed with the exact, compact, and truncated generated matrices in Kronecker form, and the linear systems are solved with the Jacobi method using fixed or adaptive rank control strategies on the compact vectors. Results of simulation on benchmark models are compared with those of the proposed solver and another version, which works with compact vectors and highly accurate low‐rank approximations of the truncated generator matrices in quantized tensor train format and solves the linear systems with the density matrix renormalization group method. Results indicate that there is a reason to solve the CME numerically, and adaptive rank control strategies on compact vectors in HTD format improve time and memory requirements significantly.

Direct tensor-product solution of one-dimensional elliptic equations with parameter-dependent coefficients

We consider a one-dimensional second-order elliptic equation with a high-dimensional parameter in a hypercube as a parametric domain. Such a problem arises, for example, from the Karhunen–Loève expansion of a stochastic PDE posed in a one-dimensional physical domain. For the discretization in the parametric domain we use the collocation on a tensor-product grid. The paper is focused on the tensor-structured solution of the resulting multiparametric problem, which allows to avoid the curse of dimensionality owing to the use of the separation of parametric variables in the tensor train and quantized tensor train formats.We suggest an efficient tensor-structured preconditioning of the entire multiparametric family of one-dimensional elliptic problems and arrive at a direct solution formula. We compare this method to a tensor-structured preconditioned GMRES solver in a series of numerical experiments.

An adaptive solution to the chemical master equation using tensors.

Solving the chemical master equation directly is difficult due to the curse of dimensionality. We tackle that challenge by a numerical scheme based on the quantized tensor train (QTT) format, which enables us to represent the solution in a compressed form that scales linearly with the dimension. We recast the finite state projection in this QTT framework and allow it to expand adaptively based on proven error criteria. The end result is a QTT-formatted matrix exponential that we evaluate through a combination of the inexact uniformization technique and the alternating minimal energy algorithm. Our method can detect when the equilibrium distribution is reached with an inexpensive test that exploits the structure of the tensor format. We successfully perform numerical tests on high-dimensional problems that had been out of reach for classical approaches.

A Tensor-Train accelerated solver for integral equations in complex geometries

We present a framework using the Quantized Tensor Train (qtt) decomposition to accurately and efficiently solve volume and boundary integral equations in three dimensions. We describe how the qtt decomposition can be used as a hierarchical compression and inversion scheme for matrices arising from the discretization of integral equations. For a broad range of problems, computational and storage costs of the inversion scheme are extremely modest O(log⁡N) and once the inverse is computed, it can be applied in O(Nlog⁡N).We analyze the qtt ranks for hierarchically low rank matrices and discuss its relationship to commonly used hierarchical compression techniques such as fmm and hss. We prove that the qtt ranks are bounded for translation-invariant systems and argue that this behavior extends to non-translation invariant volume and boundary integrals.For volume integrals, the qtt decomposition provides an efficient direct solver requiring significantly less memory compared to other fast direct solvers. We present results demonstrating the remarkable performance of the qtt-based solver when applied to both translation and non-translation invariant volume integrals in 3d.For boundary integral equations, we demonstrate that using a qtt decomposition to construct preconditioners for a Krylov subspace method leads to an efficient and robust solver with a small memory footprint. We test the qtt preconditioners in the iterative solution of an exterior elliptic boundary value problem (Laplace) formulated as a boundary integral equation in complex, multiply connected geometries.