Tucker Tensor Research Articles

Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms

SummaryWe present Nesterov‐type acceleration techniques for alternating least squares (ALS) methods applied to canonical tensor decomposition. While Nesterov acceleration turns gradient descent into an optimal first‐order method for convex problems by adding a momentum term with a specific weight sequence, a direct application of this method and weight sequence to ALS results in erratic convergence behavior. This is so because ALS is accelerated instead of gradient descent for our nonconvex problem. Instead, we consider various restart mechanisms and suitable choices of momentum weights that enable effective acceleration. Our extensive empirical results show that the Nesterov‐accelerated ALS methods with restart can be dramatically more efficient than the stand‐alone ALS or Nesterov's accelerated gradient methods, when problems are ill‐conditioned or accurate solutions are desired. The resulting methods perform competitively with or superior to existing acceleration methods for ALS, including ALS acceleration by nonlinear conjugate gradient, nonlinear generalized minimal residual method, or limited‐memory Broyden‐Fletcher‐Goldfarb‐Shanno, and additionally enjoy the benefit of being much easier to implement. We also compare with Nesterov‐type updates where the momentum weight is determined by a line search (LS), which are equivalent or closely related to existing LS methods for ALS. On a large and ill‐conditioned 71×1,000×900 tensor consisting of readings from chemical sensors to track hazardous gases, the restarted Nesterov‐ALS method shows desirable robustness properties and outperforms any of the existing methods we compare with by a large factor. There is clear potential for extending our Nesterov‐type acceleration approach to accelerating other optimization algorithms than ALS applied to other nonconvex problems, such as Tucker tensor decomposition.

Tensor alternating least squares grey model and its application to short-term traffic flows

Traffic flow data, as an important data source for the research and development of intelligent transportation systems, contain abundant multi-mode features. In this paper, a high-dimensional multi-mode tensor is used to represent traffic flow data. The Tucker tensor decomposition least squares algorithm is used to establish the tensor alternating least squares GM (1,1) model by combining the modelling mechanism of the grey classical model GM (1,1) with the algorithm, and the modelling steps are obtained. To demonstrate the effectiveness of the new model, first, the multi-mode traffic flow data are represented by the tensor model, and the correlation of the traffic flow data is analysed. Second, two short-term traffic flow prediction cases are analysed, and the results show that the performance of the GM (1, 1) model based on the tensor alternating least squares algorithm is obviously better than that of the other models. Finally, the original tensor data and the approximate tensor data during the peak period from 8:00 to 8:30 a.m. for six consecutive Mondays are selected as the experimental data, and the effect of the new model is much better than that of the GM (1,1) model of the original tensor data.

Time integration of symmetric and anti-symmetric low-rank matrices and Tucker tensors

A numerical integrator is presented that computes a symmetric or skew-symmetric low-rank approximation to large symmetric or skew-symmetric time-dependent matrices that are either given explicitly or are the unknown solution to a matrix differential equation. A related algorithm is given for the approximation of symmetric or anti-symmetric time-dependent tensors by symmetric or anti-symmetric Tucker tensors of low multilinear rank. The proposed symmetric or anti-symmetric low-rank integrator is different from recently proposed projector-splitting integrators for dynamical low-rank approximation, which do not preserve symmetry or anti-symmetry. However, it is shown that the (anti-)symmetric low-rank integrators retain favourable properties of the projector-splitting integrators: given low-rank time-dependent matrices and tensors are reproduced exactly, and the error behaviour is robust to the presence of small singular values, in contrast to standard integration methods applied to the differential equations of dynamical low-rank approximation. Numerical experiments illustrate the behaviour of the proposed integrators.

Dynamically orthogonal tensor methods for high-dimensional nonlinear PDEs

We develop new dynamically orthogonal tensor methods to approximate multivariate functions and the solution of high-dimensional time-dependent nonlinear partial differential equations (PDEs). The key idea relies on a hierarchical decomposition of the approximation space obtained by splitting the independent variables of the problem into disjoint subsets. This process, which can be conveniently visualized in terms of binary trees, yields series expansions analogous to the classical Tensor-Train and Hierarchical Tucker tensor formats. By enforcing dynamic orthogonality conditions at each level of the binary tree, we obtain coupled evolution equations for the modes spanning each subspace within the hierarchical decomposition. This allows us to effectively compute the solution to high-dimensional time-dependent nonlinear PDEs on tensor manifolds of constant rank, with no need for rank reduction methods. We also propose new algorithms for dynamic addition and removal of modes within each subspace. Numerical examples are presented and discussed for high-dimensional hyperbolic and parabolic PDEs in bounded domains.

Unmanned Aerial System multispectral mapping for low and variable solar irradiance conditions: Potential of tensor decomposition

Unlike satellite earth observation, multispectral images acquired by Unmanned Aerial Systems (UAS) provide great opportunities to monitor land surface conditions also in cloudy or overcast weather conditions. This is especially relevant for high latitudes where overcast and cloudy days are common. However, multispectral imagery acquired by miniaturized UAS sensors under such conditions tend to present low brightness and dynamic ranges, and high noise levels. Additionally, cloud shadows over space (within one image) and time (across images) are frequent in UAS imagery collected under variable irradiance and result in sensor radiance changes unrelated to the biophysical dynamics at the surface. To exploit the potential of UAS for vegetation mapping, this study proposes methods to obtain robust and repeatable reflectance time series under variable and low irradiance conditions. To improve sensor sensitivity to low irradiance, a radiometric pixel-wise calibration was conducted with a six-channel multispectral camera (mini-MCA6, Tetracam) using an integrating sphere simulating the varying low illumination typical of outdoor conditions at 55oN latitude. The sensor sensitivity was increased by using individual settings for independent channels, obtaining higher signal-to-noise ratios compared to the uniform setting for all image channels. To remove cloud shadows, a multivariate statistical procedure, Tucker tensor decomposition, was applied to reconstruct images using a four-way factorization scheme that takes advantage of spatial, spectral and temporal information simultaneously. The comparison between reconstructed (with Tucker) and original images showed an improvement in cloud shadow removal. Outdoor vicarious reflectance validation showed that with these methods, the multispectral imagery can provide reliable reflectance at sunny conditions with root mean square deviations of around 3%. The proposed methods could be useful for operational multispectral mapping with UAS under low and variable irradiance weather conditions as those prevalent in northern latitudes.

Greedy low-rank approximation in Tucker format of solutions of tensor linear systems

We propose a method for the approximation of tensors, given either explicitly or implicitly as the solution of tensor linear systems, in the Tucker tensor format. It is an iterative method that greedily constructs a suitable tensor product subspace in which to approximate the tensor by means of successive rank one approximations. An approximation to the target tensor is then obtained by solving a suitable projection of the linear system into the constructed tensor subspace. The proposed method has applications in tensor-structured discretization methods for partial differential equations, such as the Finite Difference Method and Isogeometric Analysis. In several numerical experiments, we compare the method to the greedy rank one update (or Proper Generalized Decomposition) approach, as well as to quasi-optimal approximations obtained by a truncated higher-order singular value decomposition. The proposed method outperforms the rank one update method significantly, both in terms of error per iteration and error per unit of computation time.

An optimization approach for dynamical Tucker tensor approximation

An optimization-based approach for Tucker tensor approximation of parameter-dependent data tensors and solutions of tensor differential equations with low Tucker rank is presented. The problem of updating the tensor decomposition is reformulated as a fitting problem subject to the tangent space without relying on an orthogonality gauge condition. A discrete Euler scheme is established in an alternating least squares framework, where the quadratic subproblems are reduced to trace optimization problems, that are shown to be explicitly solvable and accessible using SVD of small size. In the presence of small singular values, instability for larger ranks is reduced, since the method does not need the (pseudo) inverse of matricizations of the core tensor. Regularization of Tikhonov type can be used to compensate for the lack of uniqueness in the tangent space. The method is validated numerically and shown to be stable also for larger ranks in the case of small singular values of the core unfoldings. Higher order explicit integrators of Runge-Kutta type can be composed.

Classification and Prediction of Erythemato-Squamous Diseases Through Tensor-Based Learning

The paper proposes a classification algorithm based on support tensor machines which finds the maximum margin between the tensor spaces. The proposed algorithm has been deployed to classify erythemato-squamous diseases (ESDs) with the help of its features. Features are derived from the skin lesion images of ESDs, and it has been represented as second-order tensors, i.e., $$ \varvec{X} \in \varvec{ }{\mathbb{R}}^{\varvec{n}} $$ can be transformed into $$ \varvec{X} \in \,\varvec{ }{\mathbf{\Re }}^{{\varvec{n}_{1} }} \,\varvec{ } \otimes \,{\mathbf{\Re }}^{{\varvec{n}_{2} }} $$ where $$ n_{1} \times n_{2} \cong n $$. After deriving the features from the skin lesion images, dominant features are extracted using Tucker tensor decomposition method. Most of the existing machine learning algorithms depend on the vector-based learning models, and these algorithms suffer from the data overfitting problem. To resolve this problem, in this paper, tensor-based learning is implemented for classification. Proposed algorithm is evaluated with the real-time dataset (Xie et al. in: He, Liu, Krupinski, Xu (eds) Health information science, Springer, Berlin, 2012), and higher classification accuracy of 99.93–100% is achieved. The acquired results are compared with the existing machine learning algorithms, and it drives home the point that the proposed algorithm provides higher classification accuracy.

High performance hardware architecture for singular spectrum analysis of Hankel tensors

This paper presents a hardware architecture for singular spectrum analysis of Hankel tensors, including computation of tucker decomposition, tensor reconstruction and final Hankelization. In the proposed design, we explore two level of optimization. First, in algorithm level, we optimize the calculation process by exploiting the Hankel property to reduce the computation complexity and on-chip BRAM resource usage. Secondly, in hardware level, parallelism is explored for acceleration. Resource sharing is applied to reduce look-up tables (LUTs) usage. To enable flexibility, the number of processing elements (PEs) can be changed through parameter setting. Our proposed design is implemented on Field-Programmable Gate Arrays (FPGAs) to process third order tensors. Experiment results show that our design achieve a speed-up from 172 to 1004 compared with CPU implementation via Intel MKL and 5 to 40 compared with GPU implementation.

Nonlinearly preconditioned L‐BFGS as an acceleration mechanism for alternating least squares with application to tensor decomposition

SummaryWe derive nonlinear acceleration methods based on the limited‐memory Broyden–Fletcher–Goldfarb–Shanno (L‐BFGS) update formula for accelerating iterative optimization methods of alternating least squares (ALS) type applied to canonical polyadic and Tucker tensor decompositions. Our approach starts from linear preconditioning ideas that use linear transformations encoded by matrix multiplications and extends these ideas to the case of genuinely nonlinear preconditioning, where the preconditioning operation involves fully nonlinear transformations. As such, the ALS‐type iterations are used as fully nonlinear preconditioners for L‐BFGS, or equivalently, L‐BFGS is used as a nonlinear accelerator for ALS. Numerical results show that the resulting methods perform much better than either stand‐alone L‐BFGS or stand‐alone ALS, offering substantial improvements in terms of time to solution and robustness over state‐of‐the‐art methods for large and noisy tensor problems, including previously described acceleration methods based on nonlinear conjugate gradients and the nonlinear generalized minimal residual method. Our approach provides a general L‐BFGS‐based acceleration mechanism for nonlinear optimization.

Tucker Tensor Analysis of Matérn Functions in Spatial Statistics

Abstract In this work, we describe advanced numerical tools for working with multivariate functions and for the analysis of large data sets. These tools will drastically reduce the required computing time and the storage cost, and, therefore, will allow us to consider much larger data sets or finer meshes. Covariance matrices are crucial in spatio-temporal statistical tasks, but are often very expensive to compute and store, especially in three dimensions. Therefore, we approximate covariance functions by cheap surrogates in a low-rank tensor format. We apply the Tucker and canonical tensor decompositions to a family of Matérn- and Slater-type functions with varying parameters and demonstrate numerically that their approximations exhibit exponentially fast convergence. We prove the exponential convergence of the Tucker and canonical approximations in tensor rank parameters. Several statistical operations are performed in this low-rank tensor format, including evaluating the conditional covariance matrix, spatially averaged estimation variance, computing a quadratic form, determinant, trace, loglikelihood, inverse, and Cholesky decomposition of a large covariance matrix. Low-rank tensor approximations reduce the computing and storage costs essentially. For example, the storage cost is reduced from an exponential 𝒪 ⁢ ( n d ) {\mathcal{O}(n^{d})} to a linear scaling 𝒪 ⁢ ( d ⁢ r ⁢ n ) {\mathcal{O}(drn)} , where d is the spatial dimension, n is the number of mesh points in one direction, and r is the tensor rank. Prerequisites for applicability of the proposed techniques are the assumptions that the data, locations, and measurements lie on a tensor (axes-parallel) grid and that the covariance function depends on a distance, ∥ x - y ∥ {\|x-y\|} .

Tucker tensor decomposition‐based tracking and Gaussian mixture model for anomaly localisation and detection in surveillance videos

The anomaly detection and localisation (ADL) gains remarkable interest as dealing with the complex surveillance videos for detecting the abnormal behaviour is tedious. The human effort in monitoring and classifying the abnormal object is inaccurate and time‐consuming; therefore, the method is proposed using the Tucker tensor decomposition (TTD) and classification of the objects using Gaussian mixture model (GMM). Initially, the object is detected in the frames for easy recognition using simple background subtraction. The TTD decomposes the tensor as core tensor and factor matrices and the two decomposed tensors are compared using the cosine similarity measure that determines the location of the object in the frame. Finally, the features including shape and speed of the object are extracted that is used for classification using the GMM that follows the maximum posterior probability principle to detect and locate the anomaly in the video. The experimentation for anomaly detection proves that the proposed TTD and TTD‐GMM method attains a higher rate of multiple object tracking precision, accuracy, sensitivity, and specificity at 0.96375, 0.975, 1, and 1, respectively.

Image Representation and Learning With Graph-Laplacian Tucker Tensor Decomposition.

Tucker tensor decomposition (TD) is widely used for image representation, reconstruction, and learning tasks. Compared to principal component analysis (PCA) models, tensor models retain more 2-D characteristics of images whereas PCA models linearize images. However, traditional TD involves attribute information only and thus does not consider the pairwise similarity information between images. In this paper, we propose a graph-Laplacian tucker tensor decomposition (GLTD) which explores both attributes and pairwise similarity information simultaneously. Generally, GLTD has three main benefits: 1) GLTD reconstruction shows clear robustness against image occlusions/outliers. We provide analysis to show that Laplacian regularization is mainly responsible to this robustness via an out-of-sample GLTD model. To the best of our knowledge, this Laplacian regularization induced robustness of TD has not been studied or emphasized before; 2) GLTD representation performs more regularity, which improves both unsupervised and supervised learning results; and 3) an effective algorithm is derived to solve GLTD problem. Although GLTD is a noncovex problem, the proposed algorithm is shown experimentally to provide a stable/unique solution starting from different random initializations. Experimental results on image reconstruction, data clustering, and classification tasks show the benefits of GLTD.

Time Integration of Rank-Constrained Tucker Tensors

Dynamical low-rank approximation in the Tucker tensor format of given large time-dependent tensors and of tensor differential equations is the subject of this paper. In particular, a discrete time integration method for rank-constrained Tucker tensors is presented and analyzed. It extends the known projector-splitting integrator for dynamical low-rank approximation of matrices to Tucker tensors and is shown to inherit the same favorable properties. The integrator is based on iteratively applying the matrix projector-splitting integrator to tensor unfoldings but with inexact solution in a substep. It has the property that it reconstructs time-dependent Tucker tensors of the given rank exactly. The integrator is also shown to be robust to the presence of small singular values in the tensor unfoldings. Numerical examples with time-dependent problems from quantum physics and tensor optimization methods illustrate our theoretical results.

A novel method for classification of matrix data using Twin Multiple Rank SMMs

It is known that most high-order tensor data are linear inseparable and all of them can be transformed into matrix data through tucker tensor decomposition and the matrices involved are always multiple rank. Hence, how to effectively and efficiently classify matrix data becomes an important research topic. However, up to now most known classifiers for matrix data are linear and a few nonlinear classifiers are only for rank-one matrices. In order to classify multiple rank matrix data, in this paper, a novel supervised classification method called Linear Twin Multiple Rank Support Matrix Machine (LTMRSMM) is developed, which is also an improvement of TSVM and an extension of MRMLSVM. For linear inseparable matrix data, a nonlinear classification method named Nonlinear Twin Multiple Rank Support Matrix Machine (NTMRSMM) is provided by means of a new matrix kernel function introduced recently by authors of the paper. Experimental results show that the presented methods are effective and competitive classification methods for multiple rank matrix data.

Nested Tucker tensor decomposition with application to MIMO relay systems using tensor space–time coding (TSTC)

The aim of this paper is twofold. In a first part, we present a new tensor decomposition that we call Tucker train decomposition or nested Tucker decomposition (NTD). NTD can be viewed as a particular case of tensor-train decomposition recently proposed for representing and approximating high-dimensional tensors in a compact way. NTD of a fourth-order tensor is more specially analysed in terms of parameter estimation and uniqueness issue. In a second part, we show that the use of a tensor space–time coding (TSTC) structure at both the source node and the relay node of a one-way two-hop multi-input multi-output (MIMO) relay communication system leads to a nested Tucker decomposition of the fourth-order tensor formed by the signals received at the destination. Two semi-blind receivers are then proposed for jointly estimating the transmitted information symbols and the two individual relay channels. The first one is iterative, based on a three-step alternating least squares (ALS) algorithm, whereas the second one, denoted 2LSKP, is a closed-form solution based on the LS estimations of two Kronecker products. Two supervised receivers are also derived by using a (short) pilot-assisted closed-form solution for calculating channel estimates. These estimates are exploited either for initializing the ALS receiver or for designing a zero-forcing (ZF) receiver. Extensive Monte Carlo simulation results are provided to demonstrate the performance of the proposed relay system.

Nonlinearly Preconditioned Optimization on Grassmann Manifolds for Computing Approximate Tucker Tensor Decompositions

Two accelerated optimization algorithms are presented for computing approximate Tucker tensor decompositions, formulated using orthonormal factor matrices, by minimizing error as measured by the Frobenius norm. The first is a nonlinearly preconditioned conjugate gradient (NPCG) algorithm, wherein a nonlinear preconditioner is used to generate a direction which replaces the gradient in the nonlinear conjugate gradient iteration. The second is a nonlinear GMRES (N-GMRES) algorithm, in which a linear combination of past iterates and a tentative new iterate, generated by a nonlinear preconditioner, is minimized to produce an improved search direction. The Euclidean versions of these methods are extended to the manifold setting, where optimization on Grassmann manifolds is used to handle orthonormality constraints and to allow isolated minimizers. Several modifications are required for use on manifolds: logarithmic maps are used to determine required tangent vectors, retraction mappings are used in the line se...

Multiple rank multi-linear kernel support vector machine for matrix data classification

High-order tensors especially matrices are one of the common forms of data in real world. How to classify tensor data is an important research topic. We know that all high-order tensor data can be transformed into matrix data through tucker tensor decomposition and most of them are linear inseparable and the matrices involved are multiple rank. However, up to now most known classifiers for matrix data are linear and a few nonlinear classifiers are only for rank-one matrices. In order to classify linear inseparable multiple rank matrix data, in this paper, a novel nonlinear classifier named as multiple rank multi-linear kernel SVM (MRMLKSVM) is proposed, which is also an extension of MRMLSVM and an improvement of NLS-TSTM. For verifying the effectiveness of the proposed method, a series of comparative experiments are performed on four data sets taken from different databases. Experiment results indicate that MRMLKSVM is an effective and efficient nonlinear classifier.

Lossy volume compression using Tucker truncation and thresholding

Tensor decompositions, in particular the Tucker model, are a powerful family of techniques for dimensionality reduction and are being increasingly used for compactly encoding large multidimensional arrays, images and other visual data sets. In interactive applications, volume data often needs to be decompressed and manipulated dynamically; when designing data reduction and reconstruction methods, several parameters must be taken into account, such as the achievable compression ratio, approximation error and reconstruction speed. Weighing these variables in an effective way is challenging, and here we present two main contributions to solve this issue for Tucker tensor decompositions. First, we provide algorithms to efficiently compute, store and retrieve good choices of tensor rank selection and decompression parameters in order to optimize memory usage, approximation quality and computational costs. Second, we propose a Tucker compression alternative based on coefficient thresholding and zigzag traversal, followed by logarithmic quantization on both the transformed tensor core and its factor matrices. In terms of approximation accuracy, this approach is theoretically and empirically better than the commonly used tensor rank truncation method.

Optimization on the Hierarchical Tucker manifold – Applications to tensor completion

In this work, we develop an optimization framework for problems whose solutions are well-approximated by Hierarchical Tucker (HT) tensors, an efficient structured tensor format based on recursive subspace factorizations. By exploiting the smooth manifold structure of these tensors, we construct standard optimization algorithms such as Steepest Descent and Conjugate Gradient for completing tensors from missing entries. Our algorithmic framework is fast and scalable to large problem sizes as we do not require SVDs on the ambient tensor space, as required by other methods. Moreover, we exploit the structure of the Gramian matrices associated with the HT format to regularize our problem, reducing overfitting for high subsampling ratios. We also find that the organization of the tensor can have a major impact on completion from realistic seismic acquisition geometries. These samplings are far from idealized randomized samplings that are usually considered in the literature but are realizable in practical scenarios. Using these algorithms, we successfully interpolate large-scale seismic data sets and demonstrate the competitive computational scaling of our algorithms as the problem sizes grow.