Tensor Train Format Research Articles

TetraFEM: Numerical Solution of Partial Differential Equations Using Tensor Train Finite Element Method

In this paper, we present a methodology for the numerical solving of partial differential equations in 2D geometries with piecewise smooth boundaries via finite element method (FEM) using a Quantized Tensor Train (QTT) format. During the calculations, all the operators and data are assembled and represented in a compressed tensor format. We introduce an efficient assembly procedure of FEM matrices in the QTT format for curvilinear domains. The features of our approach include efficiency in terms of memory consumption and potential expansion to quantum computers. We demonstrate the correctness and advantages of the method by solving a number of problems, including nonlinear incompressible Navier–Stokes flow, in differently shaped domains.

Tensor-Train Format Hierarchical Equations of Motion Formalism: Charge Transfer in Organic Semiconductors via Dissipative Holstein Models.

Hierarchical Equations of Motion (HEOM) in the Tensor-Train (TT) representation is applied to study the charge-transfer dynamics in organic semiconductors (OSCs). The theoretical formulation as well as the basic computational aspects of HEOM-TT are discussed in detail. Charge transfer in OSCs is modeled using dissipative polaronic models that incorporate the effects of both high- and low-frequency molecular vibrations, and it is simulated in a fully quantum and nonperturbative manner, which has not been studied intensively. The capability of treating complex electron-vibrational systems is examined by analyzing and comparing the numerical behavior of the time-dependent variational approach and the time-Alternating Minimal Energy methods and by calculating the current autocorrelation function and diffusivity across various models. Our results indicate that the HEOM-TT framework offers a robust tool for the detailed analysis of complex polaronic systems, suggesting its potential for broader applications.

Extraction of nonlinearity in neural networks with Koopman operator

Nonlinearity plays a crucial role in deep neural networks. In this paper, we investigate the degree to which the nonlinearity of the neural network is essential. For this purpose, we employ the Koopman operator, extended dynamic mode decomposition, and the tensor-train format. The Koopman operator approach has been recently developed in physics and nonlinear sciences; the Koopman operator deals with the time evolution in the observable space instead of the state space. Since we can replace the nonlinearity in the state space with the linearity in the observable space, it is a hopeful candidate for understanding complex behavior in nonlinear systems. Here, we analyze learned neural networks for the classification problems. As a result, the replacement of the nonlinear middle layers with the Koopman matrix yields enough accuracy in numerical experiments. In addition, we confirm that the pruning of the Koopman matrix gives sufficient accuracy even at high compression ratios. These results indicate the possibility of extracting some features in the neural networks with the Koopman operator approach.

Tensor-Based Data-Driven Identification of Partial Differential Equations

Abstract We present a tensor-based method for model selection which identifies the unknown partial differential equation that governs a dynamical system using only spatiotemporal measurements. The method circumvents a disadvantage of standard matrix-based methods which typically have large storage consumption. Using a recently developed multidimensional approximation of nonlinear dynamical systems (MANDy), we collect the nonlinear and partial derivative terms of the measured data and construct a low-rank dictionary tensor in the tensor-train (TT) format. A tensor-based linear regression problem is then built, which balances the learning accuracy, model complexity, and computational efficiency. An algebraic expression of the unknown equations can be extracted. Numerical results are demonstrated on datasets generated by the wave equation, the Burgers' equation, and a few parametric partial differential equations (PDEs).

Approximation in the extended functional tensor train format

This work proposes the extended functional tensor train (EFTT) format for compressing and working with multivariate functions on tensor product domains. Our compression algorithm combines tensorized Chebyshev interpolation with a low-rank approximation algorithm that is entirely based on function evaluations. Compared to existing methods based on the functional tensor train format, the adaptivity of our approach often results in reducing the required storage, sometimes considerably, while achieving the same accuracy. In particular, we reduce the number of function evaluations required to achieve a prescribed accuracy by up to over 96%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$96\\%$$\\end{document} compared to the algorithm from Gorodetsky et al. (Comput. Methods Appl. Mech. Eng. 347, 59–84 2019).

Tensor shape search for efficient compression of tensorized data and neural networks

Compressing big data and model parameters via tensor decomposition such as the tensor train (TT) format has gained great success in recent years. The application of tensor compression methods requires the data be high dimensional. However, not all the real-world data primarily are high-dimensional, and sometimes reshaping is necessary before the application of tensor compression methods. Meantime, reordering and reshaping data may affect the efficiency of the compression. This work utilizes tensor reshaping to improve the efficiency of tensor compression using the TT format. An optimization model is proposed that maximizes the space-saving of tensor compression with respect to the shape of a given tensor while the compression error is bounded. The study is narrowed down to the TT decomposition and the TT-SVD algorithm is linked with a genetic algorithm (GA) to find an optimal tensor shape. The proposed method is applied to compress RGB images and a neural network to exemplify its capability. The results of the proposed tensor shape search using the GA are also compared with a purely random search. The results demonstrate that the proposed tensor shape search method significantly improves the space-saving and compression ratio of the data compression and enhances the efficiency of tensorized neural networks using the TT decomposition.

A novel recursive least-squares adaptive method for streaming tensor-train decomposition with incomplete observations

Tensor tracking which is referred to as online (adaptive) decomposition of streaming tensors has recently gained much attention in the signal processing community due to the fact that many modern applications generate a huge number of multidimensional data streams over time. In this paper, we propose an effective tensor tracking method via the tensor-train format for decomposing high-order incomplete streaming tensors. On the arrival of new data, the proposed algorithm minimizes a weighted least-squares objective function accounting for both missing values and time-variation constraints on the underlying tensor-train cores, thanks to the recursive least-squares filtering technique and the block coordinate descent framework. Our algorithm is fully capable of tensor tracking from noisy, incomplete, and high-dimensional observations in both static and time-varying environments. Its tracking ability is validated with several experiments on both synthetic and real data.

Black Box Approximation in the Tensor Train Format Initialized by ANOVA Decomposition

Black Box Approximation in the Tensor Train Format Initialized by ANOVA Decomposition

Survey of the hierarchical equations of motion in tensor-train format for non-Markovian quantum dynamics

Survey of the hierarchical equations of motion in tensor-train format for non-Markovian quantum dynamics

Outlier detection toward high-dimensional industrial data using extreme tensor-train learning machine with compression

Outlier detection in a high-dimensional dataset is a significant but challenging task in a number of applications. Extreme learning machine (ELM) is a powerful modeling tool for identifying outlier in an underlying dataset. However, when dealing with outliers in high-dimensional industry data, ELM brings huge storage and computational cost. To address this issue, we propose ELM based on a tensor-train format (ETFLM). Specifically, a tensor-train layer is builded with tensor-train decomposition. The fully connected layers of a neural network are replaced with tensor-train layers. Based on tensor-train layers and ELM, ETFLM is proposed in this study and its training algorithm is further presented. The experimental results show that ETFLM achieves high compression rate on low-dimensional data, and detection accuracy is slightly decreased. However, on high-dimensional data, ETFLM achieves more than 60%, whereas traditional algorithms achieve less than 40%.

WaveTrain: A Python package for numerical quantum mechanics of chain-like systems based on tensor trains.

WaveTrain is an open-source software for numerical simulations of chain-like quantum systems with nearest-neighbor (NN) interactions only. The Python package is centered around tensor train (TT, or matrix product) format representations of Hamiltonian operators and (stationary or time-evolving) state vectors. It builds on the Python tensor train toolbox Scikit_tt, which provides efficient construction methods and storage schemes for the TT format. Its solvers for eigenvalue problems and linear differential equations are used in WaveTrain for the time-independent and time-dependent Schrödinger equations, respectively. Employing efficient decompositions to construct low-rank representations, the tensor-train ranks of state vectors are often found to depend only marginally on the chain length N. This results in the computational effort growing only slightly more than linearly with N, thus mitigating the curse of dimensionality. As a complement to the classes for full quantum mechanics, WaveTrain also contains classes for fully classical and mixed quantum-classical (Ehrenfest or mean field) dynamics of bipartite systems. The graphical capabilities allow visualization of quantum dynamics "on the fly," with a choice of several different representations based on reduced density matrices. Even though developed for treating quasi-one-dimensional excitonic energy transport in molecular solids or conjugated organic polymers, including coupling to phonons, WaveTrain can be used for any kind of chain-like quantum systems, with or without periodic boundary conditions and with NN interactions only. The present work describes version 1.0 of our WaveTrain software, based on version 1.2 of scikit_tt, both of which are freely available from the GitHub platform where they will also be further developed. Moreover, WaveTrain is mirrored at SourceForge, within the framework of the WavePacket project for numerical quantum dynamics. Worked-out demonstration examples with complete input and output, including animated graphics, are available.

Pricing High-Dimensional Bermudan Options with Hierarchical Tensor Formats

An efficient compression technique based on hierarchical tensors for popular option pricing methods is presented. It is shown that the “curse of dimensionality” can be alleviated for the computation of Bermudan option prices with the Monte Carlo least-squares approach as well as the dual martingale method, both using high-dimensional tensorized polynomial expansions. Complexity estimates are provided as well as a description of the optimization procedures in the tensor train format. Numerical experiments illustrate the favorable accuracy of the proposed methods. The dynamical programming method yields results comparable to recent neural network based methods.

Scalable conditional deep inverse Rosenblatt transports using tensor trains and gradient-based dimension reduction

We present a novel offline-online method to mitigate the computational burden of the characterization of posterior random variables in statistical learning. In the offline phase, the proposed method learns the joint law of the parameter random variables and the observable random variables in the tensor-train (TT) format. In the online phase, the resulting order-preserving conditional transport can characterize the posterior random variables given newly observed data in real time. Compared with the state-of-the-art normalizing flow techniques, the proposed method relies on function approximation and is equipped with a thorough performance analysis. The function approximation perspective also allows us to further extend the capability of transport maps in challenging problems with high-dimensional observations and high-dimensional parameters. On the one hand, we present novel heuristics to reorder and/or reparametrize the variables to enhance the approximation power of TT. On the other hand, we integrate the TT-based transport maps and the parameter reordering/reparametrization into layered compositions to further improve the performance of the resulting transport maps. We demonstrate the efficiency of the proposed method on various statistical learning tasks in ordinary differential equations (ODEs) and partial differential equations (PDEs).

The tensor-train mimetic finite difference method for three-dimensional Maxwell’s wave propagation equations

Coupling the mimetic finite difference method with the tensor-train format results in a very effective method for low-rank numerical approximations of the solutions of the time-dependent Maxwell wave propagation equations in three dimensions. To this end, we discretize the curl operators on the primal/dual tensor product grid complex and we couple the space discretization with a staggered-in-time second-order accurate time-marching scheme. The resulting solver is accurate to the second order in time and space, and still compatible, so that the approximation of the magnetic flux field has zero discrete divergence with a discrepancy close to the machine precision level. Our approach is not limited to the second-order of accuracy. We can devise higher-order formulations in space through suitable extensions of the tensor-train stencil to compute the derivatives of the mimetic differential operators. Employing the tensor-train format improves the solver performance by orders of magnitude in terms of CPU time and memory storage. A final set of numerical experiments confirms this expectation.

Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions

We study the approximation by tensor networks (TNs) of functions from classical smoothness classes. The considered approximation tool combines a tensorization of functions in $L^p([0,1))$, which allows to identify a univariate function with a multivariate function (or tensor), and the use of tree tensor networks (the tensor train format) for exploiting low-rank structures of multivariate functions. The resulting tool can be interpreted as a feed-forward neural network, with first layers implementing the tensorization, interpreted as a particular featuring step, followed by a sum-product network with sparse architecture. In part I of this work, we presented several approximation classes associated with different measures of complexity of tensor networks and studied their properties. In this work (part II), we show how classical approximation tools, such as polynomials or splines (with fixed or free knots), can be encoded as a tensor network with controlled complexity. We use this to derive direct (Jackson) inequalities for the approximation spaces of tensor networks. This is then utilized to show that Besov spaces are continuously embedded into these approximation spaces. In other words, we show that arbitrary Besov functions can be approximated with optimal or near to optimal rate. We also show that an arbitrary function in the approximation class possesses no Besov smoothness, unless one limits the depth of the tensor network.

Solving phase‐field fracture problems in the tensor train format

AbstractPhase‐field models for the quasi‐static simulation of brittle fracture where the crack is approximated by a damage phase‐field are limited by the necessary memory and computation time. In this contribution, we study the applicability of low‐rank methods to phase‐field fracture models, specifically the tensor train (TT) format. To this end, we investigate the low‐rank structure of the crack phase‐field. Additionally, we present an implementation of an alternating minimization scheme to solve the coupled displacement and damage problem in the TT format. We show the evolution of the TT ranks of the displacement and damage fields for a specific example.

Low-rank nonnegative tensor approximation via alternating projections and sketching

We show how to construct nonnegative low-rank approximations of nonnegative tensors in Tucker and tensor train formats. We use alternating projections between the nonnegative orthant and the set of low-rank tensors, using STHOSVD and TTSVD algorithms, respectively, and further accelerate the alternating projections using randomized sketching. The numerical experiments on both synthetic data and hyperspectral images show the decay of the negative elements and that the error of the resulting approximation is close to the initial error obtained with STHOSVD and TTSVD. The proposed method for the Tucker case is superior to the previous ones in terms of computational complexity and decay of negative elements. The tensor train case, to the best of our knowledge, has not been studied before.

Randomized Algorithms for Rounding in the Tensor-Train Format

The tensor-train (TT) format is a highly compact low-rank representation for high-dimensional tensors. TT is particularly useful when representing approximations to the solutions of certain types of parametrized partial differential equations. For many of these problems, computing the solution explicitly would require an infeasible amount of memory and computational time. While the TT format makes these problems tractable, iterative techniques for solving the PDEs must be adapted to perform arithmetic while maintaining the implicit structure. The fundamental operation used to maintain feasible memory and computational time is called rounding, which truncates the internal ranks of a tensor already in TT format. We propose several randomized algorithms for this task that are generalizations of randomized low-rank matrix approximation algorithms and provide significant reduction in computation compared to deterministic TT-rounding algorithms. Randomization is particularly effective in the case of rounding a sum of TT-tensors (where we observe speedup), which is the bottleneck computation in the adaptation of GMRES to vectors in TT format. We present the randomized algorithms and compare their empirical accuracy and computational time with deterministic alternatives.

Challenging the Curse of Dimensionality in Multidimensional Numerical Integration by Using a Low-Rank Tensor-Train Format

Numerical integration is a basic step in the implementation of more complex numerical algorithms suitable, for example, to solve ordinary and partial differential equations. The straightforward extension of a one-dimensional integration rule to a multidimensional grid by the tensor product of the spatial directions is deemed to be practically infeasible beyond a relatively small number of dimensions, e.g., three or four. In fact, the computational burden in terms of storage and floating point operations scales exponentially with the number of dimensions. This phenomenon is known as the curse of dimensionality and motivated the development of alternative methods such as the Monte Carlo method. The tensor product approach can be very effective for high-dimensional numerical integration if we can resort to an accurate low-rank tensor-train representation of the integrand function. In this work, we discuss this approach and present numerical evidence showing that it is very competitive with the Monte Carlo method in terms of accuracy and computational costs up to several hundredths of dimensions if the integrand function is regular enough and a sufficiently accurate low-rank approximation is available.

Tensor Train Random Projection

This work proposes a Tensor Train Random Projection (TTRP) method for dimension reduction, where pairwise distances can be approximately preserved. Our TTRP is systematically constructed through a Tensor Train (TT) representation with TT-ranks equal to one. Based on the tensor train format, this random projection method can speed up the dimension reduction procedure for high-dimensional datasets and requires fewer storage costs with little loss in accuracy, compared with existing methods. We provide a theoretical analysis of the bias and the variance of TTRP, which shows that this approach is an expected isometric projection with bounded variance, and we show that the scaling Rademacher variable is an optimal choice for generating the corresponding TT-cores. Detailed numerical experiments with synthetic datasets and the MNIST dataset are conducted to demonstrate the efficiency of TTRP.