Tensor Train Format Research Articles

Tensor-train approximation of the chemical master equation and its application for parameter inference.

In this work, we perform Bayesian inference tasks for the chemical master equation in the tensor-train format. The tensor-train approximation has been proven to be very efficient in representing high-dimensional data arising from the explicit representation of the chemical master equation solution. An additional advantage of representing the probability mass function in the tensor-train format is that parametric dependency can be easily incorporated by introducing a tensor product basis expansion in the parameter space. Time is treated as an additional dimension of the tensor and a linear system is derived to solve the chemical master equation in time. We exemplify the tensor-train method by performing inference tasks such as smoothing and parameter inference using the tensor-train framework. A very high compression ratio is observed for storing the probability mass function of the solution. Since all linear algebra operations are performed in the tensor-train format, a significant reduction in the computational time is observed as well.

Supervised Learning for Nonsequential Data: A Canonical Polyadic Decomposition Approach.

Efficient modeling of feature interactions underpins supervised learning for nonsequential tasks, characterized by a lack of inherent ordering of features (variables). The brute force approach of learning a parameter for each interaction of every order comes at an exponential computational and memory cost (curse of dimensionality). To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor, the order of which is equal to the number of features; for efficiency, it can be further factorized into a compact tensor train (TT) format. However, both TT and other tensor networks (TNs), such as tensor ring and hierarchical Tucker, are sensitive to the ordering of their indices (and hence to the features). To establish the desired invariance to feature ordering, we propose to represent the weight tensor through the canonical polyadic (CP) decomposition (CPD) and introduce the associated inference and learning algorithms, including suitable regularization and initialization schemes. It is demonstrated that the proposed CP-based predictor significantly outperforms other TN-based predictors on sparse data while exhibiting comparable performance on dense nonsequential tasks. Furthermore, for enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors. In conjunction with feature vector normalization, this is shown to yield dramatic improvements in performance for dense nonsequential tasks, matching models such as fully connected neural networks.

Finite volume approximation with ADI scheme and low-rank solver for high dimensional spatial distributed-order fractional diffusion equations

High dimensional conservative spatial distributed-order fractional diffusion equation is discretized by midpoint quadrature rule, Crank–Nicolson method, and a finite volume approximation, with alternating direction implicit scheme. The resulting scheme is shown to be consistent and unconditionally stable, hence convergent with order 3−α, where α is the maximum of the involving fractional orders. Moreover, if the initial condition and source term possess Tensor-Train format (TT-format) with low TT-ranks, the scheme can be solved in TT-format, such that higher dimensional cases can be considered. Perturbation analysis ensures that the accumulated errors due to data recompression do not affect the overall convergence order. Numerical examples with low TT-ranks initial conditions and source terms, and with dimensions up to 20 are tested.

Tensor-Train-Based High-Order Dominant Eigen Decomposition for Multimodal Prediction Services

By leveraging neoteric analytical techniques associated with big data, numerous new data-focused computation and service models have flourished in service computing systems. Accurate future predictions based on tensor-based multivariate Markov models can vigorously support enterprise decisions. However, the computation efficiency and quick response of tensor-based multimodal prediction approach are seriously restricted by the curse of dimensionality arising from high-order tensor. Therefore, to alleviate the problem, this paper focuses on proposing a tensor-train (TT)-based computation approach with its scalable implementation for high-order dominant eigen decomposition (HODED) in multivariate Markov models. First, we present a TT-based Einstein product directly based on decomposed TT cores and guarantee that the result remains TT format. Then, we put forward a scalable implementation for TT-based Einstein product in a distributed or parallel manner. Afterwards, we propose a scalable TT-based HODED (TT-HODED) algorithm and a multimodal accurate prediction algorithm. Furthermore, a TT-based big data processing and services framework is presented to provide accurate proactive services. Experimental results based on real-world GPS trajectory dataset demonstrate that TT-HODED algorithm can significantly improve the computation efficiency and reduce the running memory on the premise of guaranteeing the almost consistent prediction accuracy compared to the original HODED algorithm.

Application of Tensor Train Decomposition in S2VT Model for Sign Language Recognition

Sign language recognition is a conversion of sign language into text or speech, bridging the communication between the hearing and society. Recently, sequence-to-sequence video to text (S2VT) models has been employed in the field of sign language recognition as an effective method. However, more than 20 million parameters trained in S2VT models will result in a huge consumption in memory and computational resources, making it hard to be applied in mobile devices. In order to overcome this issue, we proposed to employ tensor-train decomposition in S2VT models to reduce the parameters. First, the impact of parameters of tensor-train factorization on the model performance was investigated systematically. After that, we applied tensor-train decomposition in different layers of a S2VT model to establish 6 tensor-train S2VT models for Chinese sign language recognition. The experimental results demonstrated that when the fully-connected layer and the first LSTM layer in S2VT was represented with tensor-train format, the model could obtain the best performance, remaining high accuracy and reducing parameters and memory significantly. The proposed tensor-train S2VT models can also be applied in other sequence-to-sequence problems to improve the performance.

Efficient Construction of Tensor Ring Representations from Sampling

In this paper we propose an efficient method to compress a high dimensional function into a tensor ring format, based on alternating least squares (ALS). Since the function has size exponential in $d$, where $d$ is the number of dimensions, we propose an efficient sampling scheme to obtain $O(d)$ important samples in order to learn the tensor ring. Furthermore, we devise an initialization method for ALS that allows fast convergence in practice. Numerical examples show that to approximate a function with similar accuracy, the tensor ring format provided by the proposed method has fewer parameters than the tensor-train format and also better respects the structure of the original function.

Differentially Private Tensor Train Deep Computation for Internet of Multimedia Things

The significant growth of the Internet of Things (IoT) takes a key and active role in healthcare, smart homes, smart manufacturing, and wearable gadgets. Due to complexness and difficulty in processing multimedia data, the IoT based scheme, namely Internet of Multimedia Things (IoMT) exists that is specialized for services and applications based on multimedia data. However, IoMT generated data are facing major processing and privacy issues. Therefore, tensor-based deep computation models proved a better platform to process IoMT generated data. A differentially private deep computation method working in the tensor space can attest to its efficacy for IoMT. Nevertheless, the deep computation model comprises a multitude of parameters; thus, it requires large units of memory and expensive computing units with higher performance levels, which hinders its performance for IoMT. Motivated by this, therefore, the paper proposes a deep private tensor train autoencoder (dPTTAE) technique to deal with IoMT generated data. Notably, the compression of weight tensors to manageable tensor train format is achieved through Tensor Train (TT) network. Moreover, TT format parameters are trained through higher-order back-propagation and gradient descent. We applied dPTTAE on three representative datasets. Comprehensive experimental evaluations and theoretical analysis show that dPTTAE enhances training time efficiency, and greatly improve memory utilization efficiency, attesting its potential for IoMT.

An adaptive solution to the chemical master equation using quantized tensor trains with sliding windows

To cope with an extremely large or even infinite state space when solving the chemical master equation in biological problems, a potent strategy is to restrict to a finite state projection (FSP) and represent the transition matrix and probability vector in quantized tensor train (QTT) format, leading to savings in storage while retaining accuracy. In an earlier adaptive FSP–QTT algorithm, the multidimensional state space was downsized and kept in the form of a hyper rectangle that was updated when needed by selectively doubling some of its side dimensions. However, this could result in a much larger state space than necessary, with the effect of hampering both the execution time and stepping scheme. In this work, we improve the algorithm by enabling sliding windows that can dynamically slide, shrink or expand, with updates driven by a number of stochastic simulation algorithm trajectories. The ensuing state space is a considerably reduced hyper rectangle containing only the most probable states at each time step. Three numerical experiments of varying difficulty are performed to compare our approach with the original adaptive FSP–QTT algorithm.

Hybrid tensor decomposition in neural network compression

Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current demand of DNNs for computational resources especially the storage consumption is growing due to that the increasing sizes of models are being required for more and more complicated applications. To address this problem, several tensor decomposition methods including tensor-train (TT) and tensor-ring (TR) have been applied to compress DNNs and shown considerable compression effectiveness. In this work, we introduce the hierarchical Tucker (HT), a classical but rarely-used tensor decomposition method, to investigate its capability in neural network compression. We convert the weight matrices and convolutional kernels to both HT and TT formats for comparative study, since the latter is the most widely used decomposition method and the variant of HT. We further theoretically and experimentally discover that the HT format has better performance on compressing weight matrices, while the TT format is more suited for compressing convolutional kernels. Based on this phenomenon we propose a strategy of hybrid tensor decomposition by combining TT and HT together to compress convolutional and fully connected parts separately and attain better accuracy than only using the TT or HT format on convolutional neural networks (CNNs). Our work illuminates the prospects of hybrid tensor decomposition for neural network compression.

Compressing 3DCNNs based on tensor train decomposition

Three-dimensional convolutional neural networks (3DCNNs) have been applied in many tasks, e.g., video and 3D point cloud recognition. However, due to the higher dimension of convolutional kernels, the space complexity of 3DCNNs is generally larger than that of traditional two-dimensional convolutional neural networks (2DCNNs). To miniaturize 3DCNNs for the deployment in confining environments such as embedded devices, neural network compression is a promising approach. In this work, we adopt the tensor train (TT) decomposition, a straightforward and simple in situ training compression method, to shrink the 3DCNN models. Through proposing tensorizing 3D convolutional kernels in TT format, we investigate how to select appropriate TT ranks for achieving higher compression ratio. We have also discussed the redundancy of 3D convolutional kernels for compression, core significance and future directions of this work, as well as the theoretical computation complexity versus practical executing time of convolution in TT. In the light of multiple contrast experiments based on VIVA challenge, UCF11, UCF101, and ModelNet40 datasets, we conclude that TT decomposition can compress 3DCNNs by around one hundred times without significant accuracy loss, which will enable its applications in extensive real world scenarios.

A two-step accelerated Levenberg–Marquardt method for solving multilinear systems in tensor-train format

Recently, several iterative algorithms have been proposed for the solution of multilinear system Axm−1=b in the sense that the tensor A of order m and dimension n is a structured one. In this paper, we continue to address this multilinear system in which the coefficient tensor A is an unstructured one given in tensor-train format, and present a new iterative method for it. The method we propose here is an accelerated version of the modified Levenberg–Marquardt method for nonlinear equations (Fan, 2012), which contains two-step line searches corresponding to the LM step and the approximate LM step equipped with a novel LM parameter. It is shown that this algorithm has cubic convergence under the local error bound condition which is weaker than nonsingularity, and the computational complexity of which is free from the so-called curse-of-dimensionality. The performed numerical examples show that our method is promising.

Adaptive stochastic Galerkin FEM for lognormal coefficients in hierarchical tensor representations

Stochastic Galerkin methods for non-affine coefficient representations are known to cause major difficulties from theoretical and numerical points of view. In this work, an adaptive Galerkin FE method for linear parametric PDEs with lognormal coefficients discretized in Hermite chaos polynomials is derived. It employs problem-adapted function spaces to ensure solvability of the variational formulation. The inherently high computational complexity of the parametric operator is made tractable by using hierarchical tensor representations. For this, a new tensor train format of the lognormal coefficient is derived and verified numerically. The central novelty is the derivation of a reliable residual-based a posteriori error estimator. This can be regarded as a unique feature of stochastic Galerkin methods. It allows for an adaptive algorithm to steer the refinements of the physical mesh and the anisotropic Wiener chaos polynomial degrees. For the evaluation of the error estimator to become feasible, a numerically efficient tensor format discretization is developed. Benchmark examples with unbounded lognormal coefficient fields illustrate the performance of the proposed Galerkin discretization and the fully adaptive algorithm.

Privacy Preserving High-Order Bi-Lanczos in Cloud–Fog Computing for Industrial Applications

Industrial cyber–physical–social systems (CPSSs), a prominent data-driven paradigm, tightly couple and coordinate social space into cyber–physical systems (CPSs) within industrial environments. With the proliferation of cloud–fog computing, cloud–fog computing becomes the most prominent computing paradigm used to implement industrial data analysis. However, the open environment of cloud–fog computing and the limited control of industrial CPSSs users make industrial data analysis without compromising users’ privacy one great research challenge in practical cloud–fog-based industrial applications. High-order Bi-Lanczos (HOBI-Lanczos) approach has shown remarkable success in heterogeneous data analysis in industrial applications. In this article, a novel privacy preserving HOBI-Lanczos approach using tensor train in cloud–fog computing is proposed for industrial data applications. Specifically, a privacy preserving industrial data analysis model using cloud–fog computing and tensor train is firstly proposed. The proposed model enables fogs and clouds to securely carry out industrial data analysis for large-scale tensors given in a tensor train format. In addition, by using this model, a privacy preserving HOBI-Lanczos approach is provided. Last but not least, by using a brain-controlled robot system case study, the proposed approach is theoretically and empirically analyzed. Our proposed approach is proven to be secure. A series of experiments corroborate the superiority of the proposed approach in cloud–fog computing for industrial applications.

Tensor train-Karhunen-Loève expansion: New theoretical and algorithmic frameworks for representing general non-Gaussian random fields

The goals of this work are two-fold: firstly, to propose a new theoretical framework for representing random fields on a large class of multidimensional physical domain in the tensor train format; secondly, to develop a new algorithmic framework for accurately computing the modes and describing the correlation structure of the latent factors beyond second order within moderate time. Central to the algorithmic framework is the tensor train decomposition of cumulant functions. This decomposition is accurately computed with a novel rank-revealing algorithm. For computing the modes, compared with existing Galerkin-type and collocation-type methods, the proposed computational procedure totally removes the need of selecting the basis functions or collocation points and the quadrature points, which not only greatly enhances adaptivity, but also avoids solving large-scale eigenvalue problems. Moreover, by computing with higher-order cumulant functions, the new theoretical and algorithmic frameworks show great potential for representing general non-Gaussian non-homogeneous random fields. Three numerical examples, including a three-dimensional random field discretization problem, illustrate the efficiency and accuracy of the proposed algorithmic framework.

Tensor trains and moment conservation for multivariate aggregation in population balance modeling

We consider the numerical solution of the multivariate aggregation population balance equation on a uniform tensor grid. This class of equations is numerically challenging to solve - the computational complexity of “straightforward” algorithms grows exponentially with respect to the number of internal coordinates describing particle properties.Here, we develop algorithms which reduce the storage and computational complexity to almost linear order, O(dn) and O(dnlog⁡(n)), respectively, where d denotes the number of internal coordinates and n the number of pivots per internal coordinate. In particular, we develop fast algorithms in tensor train format to evaluate the multidimensional aggregation integral exploiting fast Fourier transformation for the underlying convolution. A further significant result lies in the conservation of the first 2d moments for our proposed method. Numerical tests confirm the favorable theoretical results concerning computational complexity and conservation of moments.

Fast BEM Solution for 2-D Scattering Problems Using Quantized Tensor-Train Format

It is common to accelerate the boundary element method (BEM) by compression techniques [fast multipole method (FMM), H-matrix/adaptive cross approximation (ACA)] that enable a more accurate solution or a solution in higher frequency. In this article, we present a compression method based on a transformation of the linear system into the tensor-train format by the quantization technique. The method is applied to a scattering problem on a canonical object with a regular mesh and improves the performance obtained from existing methods.

FFT-based homogenisation accelerated by low-rank tensor approximations

Fast Fourier transform (FFT) based methods have turned out to be an effective computational approach for numerical homogenisation. In particular, Fourier–Galerkin methods are computational methods for partial differential equations that are discretised with trigonometric polynomials. Their computational effectiveness benefits from efficient FFT based algorithms as well as a favourable condition number. Here these kinds of methods are accelerated by low-rank tensor approximation techniques for a solution field using canonical polyadic, Tucker, and tensor train formats. This reduced order model also allows to efficiently compute suboptimal global basis functions without solving the full problem. It significantly reduces computational and memory requirements for problems with a material coefficient field that admits a moderate rank approximation. The advantages of this approach against those using full material tensors are demonstrated using numerical examples for the model homogenisation problem that consists of a scalar linear elliptic variational problem defined in two and three dimensional settings with continuous and discontinuous heterogeneous material coefficients. This approach opens up the potential of an efficient reduced order modelling of large scale engineering problems with heterogeneous material.

On algorithms for and computing with the tensor ring decomposition

AbstractTensor decompositions such as the canonical format and the tensor train format have been widely utilized to reduce storage costs and operational complexities for high‐dimensional data, achieving linear scaling with the input dimension instead of exponential scaling. In this paper, we investigate even lower storage‐cost representations in the tensor ring format, which is an extension of the tensor train format with variable end‐ranks. Firstly, we introduce two algorithms for converting a tensor in full format to tensor ring format with low storage cost. Secondly, we detail a rounding operation for tensor rings and show how this requires new definitions of common linear algebra operations in the format to obtain storage‐cost savings. Lastly, we introduce algorithms for transforming the graph structure of graph‐based tensor formats, with orders of magnitude lower complexity than existing literature. The efficiency of all algorithms is demonstrated on a number of numerical examples, and in certain cases, we demonstrate significantly higher compression ratios when compared to previous approaches to using the tensor ring format.

Tensor-Train decomposition for image recognition

We explore the potential of Tensor-Train (TT) decompositions in the context of multi-feature face or object recognition strategies. We devise a new recognition algorithm that can handle three or more way tensors in the TT format, and propose a truncation strategy to limit memory usage. Numerical comparisons with other related methods—including the well established recognition algorithm based on high-order SVD—illustrate the features of the various strategies on benchmark datasets.

Stability of Low-Rank Tensor Representations and Structured Multilevel Preconditioning for Elliptic PDEs

Folding grid value vectors of size 2^L into Lth-order tensors of mode size 2times cdots times 2, combined with low-rank representation in the tensor train format, has been shown to result in highly efficient approximations for various classes of functions. These include solutions of elliptic PDEs on nonsmooth domains or with oscillatory data. This tensor-structured approach is attractive because it leads to highly compressed, adaptive approximations based on simple discretizations. Standard choices of the underlying bases, such as piecewise multilinear finite elements on uniform tensor product grids, entail the well-known matrix ill-conditioning of discrete operators. We demonstrate that, for low-rank representations, the use of tensor structure itself additionally introduces representation ill-conditioning, a new effect specific to computations in tensor networks. We analyze the tensor structure of a BPX preconditioner for a second-order linear elliptic operator and construct an explicit tensor-structured representation of the preconditioner, with ranks independent of the number L of discretization levels. The straightforward application of the preconditioner yields discrete operators whose matrix conditioning is uniform with respect to the discretization parameter, but in decompositions that suffer from representation ill-conditioning. By additionally eliminating certain redundancies in the representations of the preconditioned discrete operators, we obtain reduced-rank decompositions that are free of both matrix and representation ill-conditioning. For an iterative solver based on soft thresholding of low-rank tensors, we obtain convergence and complexity estimates and demonstrate its reliability and efficiency for discretizations with up to 2^{50} nodes in each dimension.