Tensor Train Research Articles

Solving differential Riccati equations: A nonlinear space-time method using tensor trains

Differential Riccati equations are at the heart of many applications in control theory. They are time-dependent, matrix-valued, and in particular nonlinear equations that require special methods for their solution. Low-rank methods have been used heavily for computing a low-rank solution at every step of a time-discretization. We propose the use of an all-at-once space-time solution leading to a large nonlinear space-time problem for which we propose the use of a Newton–Kleinman iteration. Approximating the space-time problem in a higher-dimensional low-rank tensor form requires fewer degrees of freedom in the solution and in the operator, and gives a faster numerical method. Numerical experiments demonstrate a storage reduction of up to a factor of 100.

Scalable Tensor-Train-Based Tensor Computations for Cyber–Physical–Social Big Data

Tensor-based big data analysis approaches are effectively exploited to handle multisource and heterogeneous cyber-physical-social big data generated from diverse spaces. However, the curse of dimensionality seriously restricts their widespread exploitation, especially under edge/fog computing environments. To alleviate the dilemma, we attempt to present a set of tensor-train (TT)-based tensor operations with their scalable computations and then propose a novel TT-based big data processing framework under edge/fog computing environments. Specifically, in this article, we first summarize and present a set of TT-based tensor operations by converting the original high-order tensor operation to a series of low-order (second- or third-order) TTcore-based operations. Then, we propose a two-layer scalable TT-based computation architecture, including inter-TTcore and intra-TTcore scalable models. Afterward, according to various scalable models, a series of scalable TT-based tensor computations (STT-TCs) with their complexity analysis are proposed in detail. Finally, we propose a novel TT-based big data processing framework to adapt to edge/fog computing environments. We conduct extensive experiments based on both random data sets and real-world ubiquitous bus traffic data sets. Experimental results demonstrate that the proposed STT-TCs can significantly improve computation efficiency and are suitable for edge/fog computing environments.

DecomVQANet: Decomposing visual question answering deep network via tensor decomposition and regression

The model we developed is a novel comprehensive solution to compress and accelerate the Visual Question Answering systems. In our algorithm Convolutional Neural Network is compressed with Long Short Term Memory to accelerate processing simultaneously. We propose to conduct various decomposition methods and regression strategies on different layers, including Canonical Polyadic, Tucker, and Tensor Train to decompose Fully Connected layers in CNN and LSTM. The Flattening Layer and Fully Connected layer at the end of the model are replaced with Tensor Regression layers. In order to compress the parameter further, the feature flow between the layers is compressed by Tensor Contraction layer. The proposed tensor decomposition model was evaluated on VQA 2.0 dataset with Pythia as baseline model. Our proposed model achieved from 77% to 91% of compression ratio, and only from 1% to 5% accuracy drop.

Adaptive stochastic Galerkin FEM for lognormal coefficients in hierarchical tensor representations

Stochastic Galerkin methods for non-affine coefficient representations are known to cause major difficulties from theoretical and numerical points of view. In this work, an adaptive Galerkin FE method for linear parametric PDEs with lognormal coefficients discretized in Hermite chaos polynomials is derived. It employs problem-adapted function spaces to ensure solvability of the variational formulation. The inherently high computational complexity of the parametric operator is made tractable by using hierarchical tensor representations. For this, a new tensor train format of the lognormal coefficient is derived and verified numerically. The central novelty is the derivation of a reliable residual-based a posteriori error estimator. This can be regarded as a unique feature of stochastic Galerkin methods. It allows for an adaptive algorithm to steer the refinements of the physical mesh and the anisotropic Wiener chaos polynomial degrees. For the evaluation of the error estimator to become feasible, a numerically efficient tensor format discretization is developed. Benchmark examples with unbounded lognormal coefficient fields illustrate the performance of the proposed Galerkin discretization and the fully adaptive algorithm.

Guaranteed a posteriori error bounds for low-rank tensor approximate solutions

Abstract We propose a guaranteed and fully computable upper bound on the energy norm of the error in low-rank tensor train (TT) approximate solutions of (possibly) high-dimensional reaction–diffusion problems. The error bound is obtained from Euler–Lagrange equations for a complementary flux reconstruction problem, which are solved in the low-rank TT representation using the block alternating linear scheme. This bound is guaranteed to be above the energy norm of the total error, including the discretization error, the tensor approximation error and the error in the solver of linear algebraic equations, although quadrature errors, in general, can pollute its evaluation. Numerical examples with the Poisson equation and the Schrödinger equation with the Henon–Heiles potential in up to 40 dimensions are presented to illustrate the efficiency of this approach.

The probabilistic tensor decomposition toolbox

This article introduces the probabilistic tensor decomposition toolbox - a MATLAB toolbox for tensor decomposition using Variational Bayesian inference and Gibbs sampling. An introduction and overview of probabilistic tensor decomposition and its connection with classical tensor decomposition methods based on maximum likelihood is provided. We subsequently describe the probabilistic tensor decomposition toolbox which encompasses the Canonical Polyadic, Tucker, and Tensor Train decomposition models. Currently, unconstrained, non-negative, orthogonal, and sparse factors are supported. Bayesian inference forms a principled way of incorporating prior knowledge, prediction of held-out data, and estimating posterior probabilities. Furthermore, it facilitates automatic model order determination, automatic regularization on factors (e.g. sparsity), and inherently penalizes model complexity which is beneficial when inferring hierarchical models, such as heteroscedastic noise modelling. The toolbox allows researchers to easily apply Bayesian tensor decomposition methods without the need to derive or implement these methods themselves. Furthermore, it serves as a reference implementation for comparing existing and new tensor decomposition methods. The software is available from https://github.com/JesperLH/prob-tensor-toolbox/.

Privacy Preserving High-Order Bi-Lanczos in Cloud–Fog Computing for Industrial Applications

Industrial cyber–physical–social systems (CPSSs), a prominent data-driven paradigm, tightly couple and coordinate social space into cyber–physical systems (CPSs) within industrial environments. With the proliferation of cloud–fog computing, cloud–fog computing becomes the most prominent computing paradigm used to implement industrial data analysis. However, the open environment of cloud–fog computing and the limited control of industrial CPSSs users make industrial data analysis without compromising users’ privacy one great research challenge in practical cloud–fog-based industrial applications. High-order Bi-Lanczos (HOBI-Lanczos) approach has shown remarkable success in heterogeneous data analysis in industrial applications. In this article, a novel privacy preserving HOBI-Lanczos approach using tensor train in cloud–fog computing is proposed for industrial data applications. Specifically, a privacy preserving industrial data analysis model using cloud–fog computing and tensor train is firstly proposed. The proposed model enables fogs and clouds to securely carry out industrial data analysis for large-scale tensors given in a tensor train format. In addition, by using this model, a privacy preserving HOBI-Lanczos approach is provided. Last but not least, by using a brain-controlled robot system case study, the proposed approach is theoretically and empirically analyzed. Our proposed approach is proven to be secure. A series of experiments corroborate the superiority of the proposed approach in cloud–fog computing for industrial applications.

Tensor train-Karhunen-Loève expansion: New theoretical and algorithmic frameworks for representing general non-Gaussian random fields

The goals of this work are two-fold: firstly, to propose a new theoretical framework for representing random fields on a large class of multidimensional physical domain in the tensor train format; secondly, to develop a new algorithmic framework for accurately computing the modes and describing the correlation structure of the latent factors beyond second order within moderate time. Central to the algorithmic framework is the tensor train decomposition of cumulant functions. This decomposition is accurately computed with a novel rank-revealing algorithm. For computing the modes, compared with existing Galerkin-type and collocation-type methods, the proposed computational procedure totally removes the need of selecting the basis functions or collocation points and the quadrature points, which not only greatly enhances adaptivity, but also avoids solving large-scale eigenvalue problems. Moreover, by computing with higher-order cumulant functions, the new theoretical and algorithmic frameworks show great potential for representing general non-Gaussian non-homogeneous random fields. Three numerical examples, including a three-dimensional random field discretization problem, illustrate the efficiency and accuracy of the proposed algorithmic framework.

Characterization Of sampling patterns for low-tt-rank tensor retrieval

In this paper, we analyze the fundamental conditions for low-rank tensor completion given the separation or tensor-train (TT) rank, i.e., ranks of TT unfoldings. We exploit the algebraic structure of the TT decomposition to obtain the deterministic necessary and sufficient conditions on the locations of the samples to ensure finite completability. Specifically, we propose an algebraic geometric analysis on the TT manifold that can incorporate the whole rank vector simultaneously in contrast to the existing approach based on the Grassmannian manifold that can only incorporate one rank component. Our proposed technique characterizes the algebraic independence of a set of polynomials defined based on the sampling pattern and the TT decomposition, which is instrumental to obtaining the deterministic condition on the sampling pattern for finite completability. In addition, based on the proposed analysis, assuming that the entries of the tensor are sampled independently with probability p, we derive a lower bound on the sampling probability p, or equivalently, the number of sampled entries that ensures finite completability with high probability. Moreover, we also provide the deterministic and probabilistic conditions for unique completability.

Tensor trains and moment conservation for multivariate aggregation in population balance modeling

We consider the numerical solution of the multivariate aggregation population balance equation on a uniform tensor grid. This class of equations is numerically challenging to solve - the computational complexity of “straightforward” algorithms grows exponentially with respect to the number of internal coordinates describing particle properties.Here, we develop algorithms which reduce the storage and computational complexity to almost linear order, O(dn) and O(dnlog⁡(n)), respectively, where d denotes the number of internal coordinates and n the number of pivots per internal coordinate. In particular, we develop fast algorithms in tensor train format to evaluate the multidimensional aggregation integral exploiting fast Fourier transformation for the underlying convolution. A further significant result lies in the conservation of the first 2d moments for our proposed method. Numerical tests confirm the favorable theoretical results concerning computational complexity and conservation of moments.

FFT-based homogenisation accelerated by low-rank tensor approximations

Fast Fourier transform (FFT) based methods have turned out to be an effective computational approach for numerical homogenisation. In particular, Fourier–Galerkin methods are computational methods for partial differential equations that are discretised with trigonometric polynomials. Their computational effectiveness benefits from efficient FFT based algorithms as well as a favourable condition number. Here these kinds of methods are accelerated by low-rank tensor approximation techniques for a solution field using canonical polyadic, Tucker, and tensor train formats. This reduced order model also allows to efficiently compute suboptimal global basis functions without solving the full problem. It significantly reduces computational and memory requirements for problems with a material coefficient field that admits a moderate rank approximation. The advantages of this approach against those using full material tensors are demonstrated using numerical examples for the model homogenisation problem that consists of a scalar linear elliptic variational problem defined in two and three dimensional settings with continuous and discontinuous heterogeneous material coefficients. This approach opens up the potential of an efficient reduced order modelling of large scale engineering problems with heterogeneous material.

On algorithms for and computing with the tensor ring decomposition

AbstractTensor decompositions such as the canonical format and the tensor train format have been widely utilized to reduce storage costs and operational complexities for high‐dimensional data, achieving linear scaling with the input dimension instead of exponential scaling. In this paper, we investigate even lower storage‐cost representations in the tensor ring format, which is an extension of the tensor train format with variable end‐ranks. Firstly, we introduce two algorithms for converting a tensor in full format to tensor ring format with low storage cost. Secondly, we detail a rounding operation for tensor rings and show how this requires new definitions of common linear algebra operations in the format to obtain storage‐cost savings. Lastly, we introduce algorithms for transforming the graph structure of graph‐based tensor formats, with orders of magnitude lower complexity than existing literature. The efficiency of all algorithms is demonstrated on a number of numerical examples, and in certain cases, we demonstrate significantly higher compression ratios when compared to previous approaches to using the tensor ring format.

Tensor-Train decomposition for image recognition

We explore the potential of Tensor-Train (TT) decompositions in the context of multi-feature face or object recognition strategies. We devise a new recognition algorithm that can handle three or more way tensors in the TT format, and propose a truncation strategy to limit memory usage. Numerical comparisons with other related methods—including the well established recognition algorithm based on high-order SVD—illustrate the features of the various strategies on benchmark datasets.

Privacy-Preserving Tucker Train Decomposition Over Blockchain-Based Encrypted Industrial IoT Data

Tucker decomposition has been widely used to extract meaningful and underlying data from heterogeneous data generated by different kinds of devices in a wide range of industrial Internet of Things (IIoT) applications. IIoT data uploaded to the cloud contain personal and sensitive information; thus, there is a growing concern about data privacy. Current existing data analysis solutions, however, assume that the data are reliably and securely collected from different IIoT data providers, an assumption that is not always true in the real world. To address the issues, in this article we propose a privacy-preserving tucker train decomposition based on gradient descent over blockchain-based encrypted IIoT data. Specifically, we use blockchain techniques to enable IIoT data providers to reliably and securely share their data by encrypting them locally before recording them in the blockchain. We use tensor train (TT) theory to build an efficient TT-based tucker decomposition based on gradient descent that tremendously reduces the number of elements to be updated during the tucker decomposition. We utilize the massive resources of fogs and clouds to implement an efficient privacy-preserving tucker train decomposition scheme. We use homomorphic encryption to build our scheme that does complete tucker train decomposition without the involvement of users. Results from a series of extensive experiments on synthetic datasets and real-world datasets demonstrate that our proposed scheme is efficient.

Stability of Low-Rank Tensor Representations and Structured Multilevel Preconditioning for Elliptic PDEs

Folding grid value vectors of size 2^L into Lth-order tensors of mode size 2times cdots times 2, combined with low-rank representation in the tensor train format, has been shown to result in highly efficient approximations for various classes of functions. These include solutions of elliptic PDEs on nonsmooth domains or with oscillatory data. This tensor-structured approach is attractive because it leads to highly compressed, adaptive approximations based on simple discretizations. Standard choices of the underlying bases, such as piecewise multilinear finite elements on uniform tensor product grids, entail the well-known matrix ill-conditioning of discrete operators. We demonstrate that, for low-rank representations, the use of tensor structure itself additionally introduces representation ill-conditioning, a new effect specific to computations in tensor networks. We analyze the tensor structure of a BPX preconditioner for a second-order linear elliptic operator and construct an explicit tensor-structured representation of the preconditioner, with ranks independent of the number L of discretization levels. The straightforward application of the preconditioner yields discrete operators whose matrix conditioning is uniform with respect to the discretization parameter, but in decompositions that suffer from representation ill-conditioning. By additionally eliminating certain redundancies in the representations of the preconditioned discrete operators, we obtain reduced-rank decompositions that are free of both matrix and representation ill-conditioning. For an iterative solver based on soft thresholding of low-rank tensors, we obtain convergence and complexity estimates and demonstrate its reliability and efficiency for discretizations with up to 2^{50} nodes in each dimension.

Tensor train rank minimization with hybrid smoothness regularization for visual data recovery

Recently, the tensor train (TT) rank has received much attention for tensor completion, due to its ability to explore the global low-rankness of tensors. However, existing methods still leave room for improvement, since the low-rankness itself is generally not sufficient to recover the underlying data. Inspired by this, we consider a novel tensor completion model by simultaneously exploiting the global low-rankness and local smoothness of visual data. In particular, we use low-rank matrix factorization to characterize the global TT low-rankness, and framelet and total variation regularization to enhance the local smoothness. We develop an efficient proximal alternating minimization algorithm to solve the proposed new model with guaranteed convergence. Extensive experiments on various data demonstrated that the proposed method outperforms compared methods in terms of visual and quantitative measures.

Tensor train representation of MIMO channels using the JIRAFE method

MIMO technology has been subject of increasing interest in both academia and industry for future wireless standards. However, its performance benefits strongly depend on the accuracy of the channel at the base station. In a recent work, a fourth-order channel tensor model was proposed for MIMO systems. In this paper, we extend this model by exploiting additional spatial diversity at the receiver, which induces a fifth order tensor model for the channel. For such high orders, there is a crucial need to break the initial high-dimensional optimization problem into a collection of smaller coupled optimization sub-problems. This paper exploits new results on the equivalence between the canonical polyadic decomposition (CPD) and the tensor train (TT) decomposition for the multi-path scenario. Specifically, we propose a Joint dImensionality Reduction And Factor rEtrieval (JIRAFE) method to find the transmit and receive spatial signatures as well as the complex path gains (which also capture the polarization effects). Monte Carlo simulations show that our proposed TT-based representation of the channel is more robust to noise and computationally more attractive than available competing tensor-based methods, for physical parameters estimation.

Fast and Accurate Tensor Completion With Total Variation Regularized Tensor Trains

We propose a new tensor completion method based on tensor trains. The to-be-completed tensor is modeled as a low-rank tensor train, where we use the known tensor entries and their coordinates to update the tensor train. A novel tensor train initialization procedure is proposed specifically for image and video completion, which is demonstrated to ensure fast convergence of the completion algorithm. The tensor train framework is also shown to easily accommodate Total Variation and Tikhonov regularization due to their low-rank tensor train representations. Image and video inpainting experiments verify the superiority of the proposed scheme in terms of both speed and scalability, where a speedup of up to 155× is observed compared to state-of-the-art tensor completion methods at a similar accuracy. Moreover, we demonstrate the proposed scheme is especially advantageous over existing algorithms when only tiny portions (say, 1%) of the to-be-completed images/videos are known.

Tensor Denoising Using Low-Rank Tensor Train Decomposition

Exploiting the latent low-rankness of tensors is crucial in tensor denoising. Classically, many methods use the Tucker model to find the low-rank structure of a tensor. Recently, the tensor train (TT) model has drawn wide attention owing to its powerful representation ability, and well-balanced matricization scheme for a tensor, and it has been successfully applied to various problems in signal processing, and machine learning applications. In this letter, we propose a tensor denoising method using the TT singular value decomposition, and information criteria, where we leverage the minimum description length to automatically estimate the TT rank. Furthermore, we establish the relationship between Tucker decomposition, and TT decomposition. In specific, the low Tucker rank of a tensor is the sufficient but unnecessary condition to the low TT rank. It unveils in theory the potential advantages of the TT model in characterizing the latent low-rankness of tensor. Denoising experiments on both synthetic data, and real HSI dataset demonstrate its superiority against Tucker-based methods.

Tensor Train Construction From Tensor Actions, With Application to Compression of Large High Order Derivative Tensors

We present a method for converting tensors into the tensor train format based on actions of the tensor as a vector-valued multilinear function. Existing methods for constructing tensor trains requi...