Canonical Polyadic Decomposition Research Articles

CP decomposition for tensors via alternating least squares with QR decomposition

AbstractThe CP tensor decomposition is used in applications such as machine learning and signal processing to discover latent low‐rank structure in multidimensional data. Computing a CP decomposition via an alternating least squares (ALS) method reduces the problem to several linear least squares problems. The standard way to solve these linear least squares subproblems is to use the normal equations, which inherit special tensor structure that can be exploited for computational efficiency. However, the normal equations are sensitive to numerical ill‐conditioning, which can compromise the results of the decomposition. In this paper, we develop versions of the CP‐ALS algorithm using the QR decomposition and the singular value decomposition, which are more numerically stable than the normal equations, to solve the linear least squares problems. Our algorithms utilize the tensor structure of the CP‐ALS subproblems efficiently, have the same complexity as the standard CP‐ALS algorithm when the input is dense and the rank is small, and are shown via examples to produce more stable results when ill‐conditioning is present. Our MATLAB implementation achieves the same running time as the standard algorithm for small ranks, and we show that the new methods can obtain lower approximation error.

Hybrid Low-Rank Tensor CP and Tucker Decomposition with Total Variation Regularization for HSI Noise Removal

The acquired hyperspectral images (HSIs) are affected by a mixture of several types of noise, which often suffer from information missing. Corrupted HSIs limit the precision of the subsequent processing. In this paper, the weighted Total Variation-regularized Hybrid Model of CP and Tucker (WTV-HMCT) is proposed to accurately identify the intrinsic structures of the clean HSIs. By jointly minimizing CP rank and Tucker rank in the low-rank tensor approximation, WTV-HMCT fully exploits the high-dimensional structure correlations of HSI. To ensure the piecewise smoothness of the recovered image, the hybrid low-rank tensor decomposition approach integrates the weighted spatial spectral total variation regularization for the separation of the noise-free HSI and mixed noise. By the Alternating Direction Method of Multipliers (ADMM), the optimization model is transformed into two subproblems. Finally, an efficient proximal alternating minimization algorithm is developed to optimize the proposed hybrid low-rank tensor decomposition efficiently. The experimental results show that the proposed model effectively handles Gauss noise, striping noise, and mixed noise and that it outperforms the most advanced methods in terms of evaluation metrics and visual evaluation.

Zero-truncated Poisson regression for sparse multiway count data corrupted by false zeros

Abstract We propose a novel statistical inference methodology for multiway count data that is corrupted by false zeros that are indistinguishable from true zero counts. Our approach consists of zero-truncating the Poisson distribution to neglect all zero values. This simple truncated approach dispenses with the need to distinguish between true and false zero counts and reduces the amount of data to be processed. Inference is accomplished via tensor completion that imposes low-rank tensor structure on the Poisson parameter space. Our main result shows that an $N$-way rank-$R$ parametric tensor $\boldsymbol{\mathscr{M}}\in (0,\infty )^{I\times \cdots \times I}$ generating Poisson observations can be accurately estimated by zero-truncated Poisson regression from approximately $IR^2\log _2^2(I)$ non-zero counts under the nonnegative canonical polyadic decomposition. Our result also quantifies the error made by zero-truncating the Poisson distribution when the parameter is uniformly bounded from below. Therefore, under a low-rank multiparameter model, we propose an implementable approach guaranteed to achieve accurate regression in under-determined scenarios with substantial corruption by false zeros. Several numerical experiments are presented to explore the theoretical results.

Probability-weighted tensor robust PCA with CP decomposition for hyperspectral image restoration

This paper presents a novel probability-weighted tensor robust principal component analysis (TRPCA) method based on CANDECOMP/PARAFAC decomposition (CPD) for hyperspectral image (HSI) restoration with low computational complexity from the coexistence of dense noise and sparse outliers. In the proposed method, via the CPD property, the tensor nuclear norm (TNN) optimization object is replaced as the CPD factor matrix of HSI with lower dimensions without losing low-rank property, so as to the computational complexity could be reduced during optimizing TNN in TRPCA model. To demonstrate the different effects of noise and outliers on HSI restoration, two weighted sets are defined with probability by the prior knowledge of the occurrence of outliers, which are used to analyze the probability whether the HSI is corrupted by dense noise or outliers. Via the probability theory, information entropy is employed to measure the uncertainty of noise and outlier occurrence in received data, so that to demonstrate their difference impacts on HSI restoration to improve the recovery accuracy. Theoretical analysis and simulation results show that the proposed method has lower computational complexity and better restoration performance compared with other related methods.

Multi-domain feature analysis method of MI-EEG signal based on Sparse Regularity Tensor-Train decomposition

Tensor analysis can comprehensively retain multidomain characteristics, which has been employed in EEG studies. However, existing EEG tensor has large dimension, making it difficult to extract features. Traditional Tucker decomposition and Canonical Polyadic decomposition(CP) decomposition algorithms have problems of low computational efficiency and weak capability to extract features. To solve the above problems, Tensor-Train(TT) decomposition is adopted to analyze the EEG tensor. Meanwhile, sparse regularization term can then be added to TT decomposition, resulting in a sparse regular TT decomposition (SR-TT). The SR-TT algorithm is proposed in this paper, which has higher accuracy and stronger generalization ability than state-of-the-art decomposition methods. The SR-TT algorithm was verified with BCI competition III and BCI competition IV dataset and achieved 86.38% and 85.36% classification accuracies, respectively. Meanwhile, compared with traditional tensor decomposition (Tucker and CP) method, the computational efficiency of the proposed algorithm was improved by 16.49 and 31.08 times in BCI competition III and 20.72 and 29.45 times more efficient in BCI competition IV. Besides, the method can leverage tensor decomposition to extract spatial features, and the analysis is performed by pairs of brain topography visualizations to show the changes of active brain regions under the task condition. In conclusion, the proposed SR-TT algorithm in the paper provides a novel insight for tensor EEG analysis.

Hyperspectral Anomaly Detection Based on Regularized Background Abundance Tensor Decomposition

The low spatial resolution of hyperspectral images means that existing mixed pixels rely heavily on spectral information, making it difficult to differentiate between the target of interest and the background. The endmember extraction method is powerful in enhancing spatial structure information for hyperspectral anomaly detection, whereas most approaches are based on matrix representation, which inevitably destroys the spatial or spectral information. In this paper, we treated the hyperspectral image as a third-order tensor and proposed a novel anomaly detection method based on a low-rank linear mixing model of the scene background. The obtained abundance maps possessed more distinctive features than the raw data, which was beneficial for identifying anomalies in the background. Specifically, the low-rank tensor background was approximated as the mode-3 product of an abundance tensor and endmember matrix. Due to the distinctive features of the background’s abundance maps, we characterized them by tensor regularization and imposed low rankness through CP decomposition, smoothness, and sparsity. In addition, we utilized the ℓ1,1,2-norm to characterize the tube-wise sparsity of the anomaly, since it accounted for a small portion of the scene. The experimental results obtained using five real datasets demonstrated the outstanding performance of our proposed method.

Low Rank Tensor Decompositions and Approximations

AbstractThere exist linear relations among tensor entries of low rank tensors. These linear relations can be expressed by multi-linear polynomials, which are called generating polynomials. We use generating polynomials to compute tensor rank decompositions and low rank tensor approximations. We prove that this gives a quasi-optimal low rank tensor approximation if the given tensor is sufficiently close to a low rank one.

Tensor decomposition-based channel selection for motor imagery-based brain-computer interfaces.

The number of electrode channels in a brain-computer interface (BCI) affects not only its classification performance, but also its convenience in practical applications. Despite many studies on channel selection in motor imagery (MI)-based BCI systems, they consist in matrix analysis of EEG signals, which inevitably loses the interactive information among multiple domains such as space, time and frequency. In this paper, a tensor decomposition-based channel selection (TCS) method is employed for MI BCIs. A three-way tensor is yielded by wavelet transform of a single-trial EEG signal and decomposed into three factor matrices by a regularized canonical polyadic decomposition (CPD). The channel factor matrix is used for channel selection and the important channels are selected by calculating the correlation between channels. Regularized common spatial pattern (RCSP) is employed for feature extraction and support vector machine (SVM) for classification. The proposed TCS-RCSP algorithm was evaluated on three BCI data sets and compared with the RCSP with all channels (AC-RCSP) and the RCSP with selected channels by correlation-based channel selection method (CCS-RCSP). The results indicate that TCS-RCSP achieved significantly better overall accuracy than AC-RCSP (94.4% vs. 86.3%) with 0.01 and CCS-RCSP (94.4% vs. 90.2%) with 0.05, proving the efficacy of the proposed algorithm for classifying MI tasks.

Accelerating probabilistic tensor canonical polyadic decomposition with nonnegative factors: An inexact BCD Approach

Recently, Bayesian modeling and variational inference (VI) were leveraged to enable the nonnegative factor matrix learning with automatic rank determination in tensor canonical polyadic decomposition (CPD), which has found various applications in big data analytics. However, since VI inherently performs block coordinate descent (BCD) steps over the functional space, it generally does not allow integration with modern large-scale optimization methods, making the scalability a critical issue. In this paper, it is revealed that the expectations of the variables updated by the VI algorithm is equivalent to the block minimization steps of a deterministic optimization problem. This equivalence further enables the adoption of inexact BCD method for devising a fast nonnegative factor matrix learning algorithm with automatic tensor rank determination. Numerical results using synthetic data and real-world applications show that the performance of the proposed algorithm is comparable with that of the VI-based algorithm, but with computation times reduced significantly.

Tensor Principal Component Analysis in High Dimensional CP Models

The CP decomposition for high dimensional non-orthogonal spiked tensors is an important problem with broad applications across many disciplines. However, previous works with theoretical guarantee typically assume restrictive incoherence conditions on the basis vectors for the CP components. In this paper, we propose new computationally efficient composite PCA and concurrent orthogonalization algorithms for tensor CP decomposition with theoretical guarantees under mild incoherence conditions. The composite PCA applies the principal component or singular value decompositions twice, first to a matrix unfolding of the tensor data to obtain singular vectors and then to the matrix folding of the singular vectors obtained in the first step. It can be used as an initialization for any iterative optimization schemes for the tensor CP decomposition. The concurrent orthogonalization algorithm iteratively estimates the basis vector in each mode of the tensor by simultaneously applying projections to the orthogonal complements of the spaces generated by other CP components in other modes. It is designed to improve the alternating least squares estimator and other forms of the high order orthogonal iteration for tensors with low or moderately high CP ranks, and it is guaranteed to have second or higher order convergence when the error of any given initial estimator is bounded by a small constant. Our theoretical investigation provides estimation accuracy and convergence rates for the two proposed algorithms. Both proposed algorithms are applicable to deterministic tensor, its noisy version, and the order- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2K$ </tex-math></inline-formula> covariance tensor of order- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> tensor data in a factor model with uncorrelated factors. Simulation experiments demonstrate significant practical superiority of our approach over existing methods.

Tensor Ring decomposition for context-aware recommendation

As a powerful tool for processing high-dimensional data, tensor decomposition has been widely used in context-aware recommendation. Most of the current tensor decomposition-based recommendation models use CP decomposition or Tucker decomposition. In practical recommendation applications, fewer parameters limit the performance of CP decomposition, and the high-order tensor kernel makes the computational complexity of Tucker decomposition exponential. In order to improve the performance of recommendation while maintaining high computational efficiency, we propose a bias Tensor Ring decomposition framework for context-aware recommendation, which employs Tensor Ring decomposition to extract user-item-context latent factors. Comparing to current tensor decomposition-based recommendation models, our framework achieves a better balance between recommendation performance and computational complexity by Tensor Ring decomposition. This is a highly scalable recommendation framework that can easily further integrate additional information such as social trust and implicit feedback, thus used for more complex recommendation tasks. To the best of our knowledge, this work is the first to formulate context-aware recommendation as Tensor Ring decomposition process. Our in-depth analyses confirm that Tensor Ring decomposition has certain advantages over CP decomposition and Tucker decomposition. Multiple experiments on three real datasets also show that the proposed framework helps to improve the recommendation performance.

Tensor-Based Sparse Bayesian Learning With Intra-Dimension Correlation

Unlike the flat-view matrix, tensor provides an elegant representation to preserve the structure of data in the multidimensional nature. The wide use of tensor-based approaches highlights the limitation of matrix-based methods and has generated increasing needs in multidimensional data analytics. Driven by such demands, tensor decomposition with automatic rank determination has recently emerged as a promising tool for image processing, wireless communications and neural network compression. However, unlike the success in matrix-based schemes of exploring the correlation for performance improvement, existing tensor-based methods fail to consider the correlation structure in the tensor. In this paper, we propose to exploit the tensor intra-dimension correlation within the framework of sparse Bayesian learning. We provide the sparsity-inducing probabilistic models under both canonical polyadic decomposition and Tucker decomposition to capture the correlation structure effectively. We derive an efficient model inference method with a fast convergence rate based on the expectation-maximization algorithm. Furthermore, a complexity reduction method is proposed to speed up the computation by exploiting several properties of the Kronecker product. We further extend the proposed framework for the incomplete tensor corrupted by sparse outliers to achieve effective tensor completion and outlier mitigation. Simulation results demonstrate the superior performance of the proposed framework over those ignoring the intra-dimension correlation in terms of accuracy and sparsity.

Tensor Singular Spectrum Decomposition: Multisensor Denoising Algorithm and Application

Realising multi-sensor signal fusion and weak feature adaptive extraction is a challenging task. Therefore, a new algorithm called tensor singular spectrum decomposition is proposed in this study for the adaptive decomposition of multi-sensor time series. Traditional tensor decomposition algorithms, such as CP, HOSVD and Tucker decomposition, are derived from <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> -mode product. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> -mode product essentially uses the idea of matrices to deal with tensors given that it defines the multiplication between matrix and higher-order tensor, thereby creating problems of non-pseudodiagonal core tensor and nonunique decomposition results in traditional tensor decomposition algorithms. To this end, decomposition of the original tensor signal and reconstruction of multi-sensor component signals are realised in this study by combining the trajectory tensor construction, superposition of the Gaussian function spectral model, adaptive iterative optimisation of embedding dimension and diagonal average method on the basis of the principle of tensor–tensor order-preserving multiplication. The proposed algorithm inherits the perfect mathematical theory and excellent properties of matrix singular value decomposition in processing single-sensor signals whilst retaining the inherent structure and coupling relationship between multi-sensor data and realising the organic fusion and adaptive decomposition of multi-sensor signals. The analysis results of simulation, experimental and engineering signals showed that the proposed method can effectively extract weak fault quantification features hidden in original multi-sensor signals compared with existing methods.

Tucker Decomposition Based on a Tensor Train of Coupled and Constrained CP Cores

Many real-life signal-based applications use the Tucker decomposition of a high dimensional/order tensor. A well-known problem with the Tucker model is that its number of entries increases exponentially with its order, a phenomenon known as the “curse of the dimensionality”. The Higher-Order Orthogonal Iteration (HOOI) and Higher-Order Singular Value Decomposition (HOSVD) are known as the gold standard for computing the range span of the factor matrices of a Tucker Decomposition but also suffer from the curse. In this paper, we propose a new methodology with a similar estimation accuracy as the HOSVD with non-exploding computational and storage costs. If the noise-free data follows a Tucker decomposition, the corresponding Tensor Train (TT) decomposition takes a remarkable specific structure. More precisely, we prove that for a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -order Tucker tensor, the corresponding TT decomposition is constituted by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q-3$</tex-math></inline-formula> 3-order TT-core tensors that follow a Constrained Canonical Polyadic Decomposition. Using this new formulation and the coupling property between neighboring TT-cores, we propose a JIRAFE-type scheme for the Tucker decomposition, called TRIDENT. Our numerical simulations show that the proposed method offers a drastically reduced complexity compared to the HOSVD and HOOI while outperforming the Fast Multilinear Projection (FMP) method in terms of estimation accuracy.

A generalization of Kruskal’s theorem on tensor decomposition

Abstract Kruskal’s theorem states that a sum of product tensors constitutes a unique tensor rank decomposition if the so-called k-ranks of the product tensors are large. We prove a ‘splitting theorem’ for sets of product tensors, in which the k-rank condition of Kruskal’s theorem is weakened to the standard notion of rank, and the conclusion of uniqueness is relaxed to the statement that the set of product tensors splits (i.e., is disconnected as a matroid). Our splitting theorem implies a generalization of Kruskal’s theorem. While several extensions of Kruskal’s theorem are already present in the literature, all of these use Kruskal’s original permutation lemma and hence still cannot certify uniqueness when the k-ranks are below a certain threshold. Our generalization uses a completely new proof technique, contains many of these extensions and can certify uniqueness below this threshold. We obtain several other useful results on tensor decompositions as consequences of our splitting theorem. We prove sharp lower bounds on tensor rank and Waring rank, which extend Sylvester’s matrix rank inequality to tensors. We also prove novel uniqueness results for nonrank tensor decompositions.

A Coarse-to-Fine Hyperspectral Target Detection Method Based on Low-Rank Tensor Decomposition

A Coarse-to-Fine Hyperspectral Target Detection Method Based on Low-Rank Tensor Decomposition

基于张量CP分解的高频数据波动率矩阵预测

基于张量CP分解的高频数据波动率矩阵预测

Splitting-merging transitions in tensor-vectors systems in exact large- N limits

Matrix models have phase transitions in which distributions of variables change topologically like the Gross-Witten-Wadia transition. In a recent study, similar splitting-merging behavior of distributions of dynamical variables was observed in a tensor-vectors system by numerical simulations. In this paper, we study the system exactly in some large-$N$ limits, in which the distributions are discrete sets of configurations rather than continuous. We find cascades of first-order phase transitions for fixed tensors, and first- and second-order phase transitions for random tensors, being characterized by breaking patterns of replica symmetries. The system is of interest across three different subjects at least: The splitting dynamics plays essential roles in emergence of classical spacetimes in a tensor model of quantum gravity; The splitting dynamics automatically detects the rank of a tensor in the tensor rank decomposition in data analysis; The system provides a variant of the spherical $p$-spin model for spin glasses with a new non-trivial parameter. We discuss some implications of the results from these perspectives. The results are compared with some numerical simulations to check the large-$N$ convergence and the assumptions made in the analysis.

A Normal Form Algorithm for Tensor Rank Decomposition

We propose a new numerical algorithm for computing the tensor rank decomposition or canonical polyadic decomposition of higher-order tensors subject to a rank and genericity constraint. Reformulating this computational problem as a system of polynomial equations allows us to leverage recent numerical linear algebra tools from computational algebraic geometry. We characterize the complexity of our algorithm in terms of an algebraic property of this polynomial system—the multigraded regularity. We prove effective bounds for many tensor formats and ranks, which are of independent interest for overconstrained polynomial system solving. Moreover, we conjecture a general formula for the multigraded regularity, yielding a (parameterized) polynomial time complexity for the tensor rank decomposition problem in the considered setting. Our numerical experiments show that our algorithm can outperform state-of-the-art numerical algorithms by an order of magnitude in terms of accuracy, computation time, and memory consumption.

Multi‐view side information‐incorporated tensor completion

AbstractTensor completion originates in numerous applications where data utilized are of high dimensions and gathered from multiple sources or views. Existing methods merely incorporate the structure information, ignoring the fact that ubiquitous side information may be beneficial to estimate the missing entries from a partially observed tensor. Inspired by this, we formulate a sparse and low‐rank tensor completion model named SLRMV. The ‐norm instead of its relaxation is used in the objective function to constrain the sparseness of noise. The CP decomposition is used to decompose the high‐quality tensor, based on which the combination of Schatten ‐norm on each latent factor matrix is employed to characterize the low‐rank tensor structure with high computation efficiency. Diverse similarity matrices for the same factor matrix are regarded as multi‐view side information for guiding the tensor completion task. Although SLRMV is a nonconvex and discontinuous problem, the optimality analysis in terms of Karush‐Kuhn‐Tucker (KKT) conditions is accordingly proposed, based on which a hard‐thresholding based alternating direction method of multipliers (HT‐ADMM) is designed. Extensive experiments remarkably demonstrate the efficiency of SLRMV in tensor completion.