Tensor Approximation Problem Research Articles

Local and Global Convergence of General Burer-Monteiro Tensor Optimizations

Tensor optimization is crucial to massive machine learning and signal processing tasks. In this paper, we consider tensor optimization with a convex and well-conditioned objective function and reformulate it into a nonconvex optimization using the Burer-Monteiro type parameterization. We analyze the local convergence of applying vanilla gradient descent to the factored formulation and establish a local regularity condition under mild assumptions. We also provide a linear convergence analysis of the gradient descent algorithm started in a neighborhood of the true tensor factors. Complementary to the local analysis, this work also characterizes the global geometry of the best rank-one tensor approximation problem and demonstrates that for orthogonally decomposable tensors the problem has no spurious local minima and all saddle points are strict except for the one at zero which is a third-order saddle point.

Kronecker Product-Based Space-Time Block Codes

This letter presents a new approach to construct space-time block codes (STBCs) based on the Kronecker product. The proposed Kronecker-structured STBCs (K-STBCs) are specially designed for constant modulus constellations and include the classical linear dispersion (LD) codes as a special case. Its inherent Kronecker structure offers additional coding gains due to the mutual spreading of multiple space-time codewords. At the receiver, an efficient algorithm is derived by recasting the decoding of the K-STBCs as a rank-one tensor approximation problem. The proposed decoding algorithm provides a good bit error ratio (BER) performance especially, in the low signal-to-noise ratio (SNR) regime. In particular, our numerical results show that K-STBCs outperform the existing quasi-orthogonal STBCs due to the denoising gain offered by the proposed rank-one approximation based detector.

Riemannian Newton optimization methods for the symmetric tensor approximation problem

The Symmetric Tensor Approximation problem (STA) consists of approximating a symmetric tensor or a homogeneous polynomial by a linear combination of symmetric rank-1 tensors or powers of linear forms of low symmetric rank. We present two new Riemannian Newton-type methods for low rank approximation of symmetric tensor with complex coefficients.The first method uses the parametrization of the set of tensors of rank at most r by weights and unit vectors. Exploiting the properties of the apolar product on homogeneous polynomials combined with efficient tools from complex optimization, we provide an explicit and tractable formulation of the Riemannian gradient and Hessian, leading to Newton iterations with local quadratic convergence. We prove that under some regularity conditions on non-defective tensors in the neighborhood of the initial point, the Newton iteration (completed with a trust-region scheme) is converging to a local minimum.The second method is a Riemannian Gauss–Newton method on the Cartesian product of Veronese manifolds. An explicit orthonormal basis of the tangent space of this Riemannian manifold is described. We deduce the Riemannian gradient and the Gauss–Newton approximation of the Riemannian Hessian. We present a new retraction operator on the Veronese manifold.We analyze the numerical behavior of these methods, with an initial point provided by Simultaneous Matrix Diagonalisation (SMD). Numerical experiments show the good numerical behavior of the two methods in different cases and in comparison with existing state-of-the-art methods.

A nonlocal enhanced Low-Rank tensor approximation framework for 3D Magnetic Resonance image denoising

Magnetic Resonance Imaging (MRI) has become an increasingly essential tool in clinical detection and diagnosis for its ability to disclose the distinctive information of human anatomic structures in vivo. However, the image intensity in magnitude MRI images is frequently corrupted by Rician noise which is inherent to the acquisition process. This study aimed to propose a denoising model for 3D MR image restoration based on low-rank tensor approximation with a Logarithmic-Sum regularization framework. Nonlocal similarity of images was exploited with grouping and block matching techniques, and tensor decomposition approaches were utilized taking consideration of multi-dimensional structural dependence conservation. The low-rank tensor approximation problem was regulated by introducing a Logarithmic-Sum trace norm, which thresholded the singular value spectrum adaptively with different thresholds. Noise removal was processed through the low-rank approximation, and the denoised tensor blocks were aggregated to restore the clear images. Numerical experiments were conducted under comprehensive noise conditions for 3D MR volumetric datasets. Through denoising comparison, the results demonstrated that the proposed algorithm achieves state-of-the-art performance for Rician noise removal with excellent detail preservation..

Tensor Linear Regression: Degeneracy and Solution

Tensor regression is an important and useful tool for analyzing multidimensional array data. To deal with high dimensionality, CANDECOMP/PARAFAC (CP) low-rank constraints are often imposed on the coefficient tensor parameter in the (penalized) loss functions. However, besides the well-known non-identifiability issue of the CP parameters, we demonstrate that the corresponding optimization may not have any attainable solutions, and thus the estimation of the coefficient tensor is not well-defined when this happens. This is closely related to a phenomenon, called CP degeneracy, in low-rank tensor approximation problems. In this article, we show some useful results of CP degeneracy in the context of tensor regression problems. To overcome the theoretical and numerical issues associated with the degeneracy, we provide a general penalized strategy as a solution to the degeneracy. The related results also explain why some of the existing methods are more stable than the others. The asymptotic properties of the resulting estimation are also studied. Numerical experiments are conducted to illustrate our findings.

Alternating Group Lasso for Block-Term Tensor Decomposition and Application to ECG Source Separation

In some applications, blind source separation can be performed by computing an approximate block-term tensor decomposition (BTD), under much milder constraints than matrix-based techniques. However, choosing the BTD model structure ( i.e. , the number of blocks and their ranks) is a difficult problem, and the standard least-squares formulation can be ill-posed. This paper proposes an alternating group lasso algorithm to compute approximate low-rank BTDs. It solves, in a provably convergent manner, a well-posed mixed-norm regularized tensor approximation problem that allows estimating the model parameters and its structure jointly. A variant is also put forward for dealing with linearly constrained blocks, motivated by the problem of blind separation of sums of complex exponentials, which can be cast as a low-rank Hankel-structured block-term tensor approximation problem. An experimental comparison with a standard nonlinear least-squares algorithm on synthetic tensor data indicates that the proposed algorithm is much more robust with respect to initialization. We also apply the constrained variant to the extraction of atrial activity from semi-synthetic and real-world electrocardiogram recordings during atrial fibrillation episodes. Our results show its ability to consistently select an adequate structure and to extract multiple signals which can be physiologically interpreted as atrial fibrillation patterns.

A self-adaptive regularized alternating least squares method for tensor decomposition problems

Though the alternating least squares algorithm (ALS), as a classic and easily implemented algorithm, has been widely applied to tensor decomposition and approximation problems, it has some drawbacks: the convergence of ALS is not guaranteed, and the swamp phenomenon appears in some cases, causing the convergence rate to slow down dramatically. To overcome these shortcomings, the regularized-ALS algorithm (RALS) was proposed in the literature. By employing the optimal step-size selection rule, we develop a self-adaptive regularized alternating least squares method (SA-RALS) to accelerate RALS in this paper. Theoretically, we show that the step-size is always larger than unity, and can be larger than [Formula: see text], which is quite different from several optimization algorithms. Furthermore, under mild assumptions, we prove that the whole sequence generated by SA-RALS converges to a stationary point of the objective function. Numerical results verify that the SA-RALS performs better than RALS in terms of the number of iterations and the CPU time.

Dual L1-Normalized Context Aware Tensor Power Iteration and Its Applications to Multi-object Tracking and Multi-graph Matching

The multi-dimensional assignment problem is universal for data association analysis such as data association-based visual multi-object tracking and multi-graph matching. In this paper, multi-dimensional assignment is formulated as a rank-1 tensor approximation problem. A dual L1-normalized context/hyper-context aware tensor power iteration optimization method is proposed. The method is applied to multi-object tracking and multi-graph matching. In the optimization method, tensor power iteration with the dual unit norm enables the capture of information across multiple sample sets. Interactions between sample associations are modeled as contexts or hyper-contexts which are combined with the global affinity into a unified optimization. The optimization is flexible for accommodating various types of contextual models. In multi-object tracking, the global affinity is defined according to the appearance similarity between objects detected in different frames. Interactions between objects are modeled as motion contexts which are encoded into the global association optimization. The tracking method integrates high order motion information and high order appearance variation. The multi-graph matching method carries out matching over graph vertices and structure matching over graph edges simultaneously. The matching consistency across multi-graphs is based on the high-order tensor optimization. Various types of vertex affinities and edge/hyper-edge affinities are flexibly integrated. Experiments on several public datasets, such as the MOT16 challenge benchmark, validate the effectiveness of the proposed methods.

Semi-blind receivers for MIMO multi-relaying systems via rank-one tensor approximations

This paper proposes two tensor-based receivers for multiple-input multiple-output (MIMO) multi-relaying systems capable of jointly estimating the channels and symbols in a semi-blind fashion. Assuming space-time coding at the source and relay stations, we propose an orthogonal design based on a parallel factor (PARAFAC) analysis of the coding structure. Exploiting the proposed tensor codes and the multi-linear structure of the resulting received signals, we show that the data model for every relay-assisted link after space-time combining/decoding has a Kronecker structure, which can be recast as a rank-one tensor corrupted by noise. The proposed receivers combine the tensor signals for the multiple cooperative links for joint channel and symbol estimation by coupling multiple rank-one tensor approximation problems. The first one is a coupled-SVD based receiver that estimates all the involved communication channels and transmitted symbols in closed form. The second one is an iterative solution based on alternating least squares. The performances of both receivers are evaluated by means of computer simulations in a variety of system configurations. Our results show the effectiveness of the proposed receivers and its good performance-complexity trade-off in comparison with competing receivers.

Rank-1 Tensor Approximation for High-Order Association in Multi-target Tracking

High-order motion information is important in multi-target tracking (MTT) especially when dealing with large inter-target ambiguities. Such high-order information can be naturally modeled as a multi-dimensional assignment (MDA) problem, whose global solution is however intractable in general. In this paper, we propose a novel framework to the problem by reshaping MTT as a rank-1 tensor approximation problem (R1TA). We first show that MDA and R1TA share the same objective function and similar constraints. This discovery opens a door to use high-order tensor analysis for MTT and suggests the exploration of R1TA. In particular, we develop a tensor power iteration algorithm to effectively capture high-order motion information as well as appearance variation. The proposed algorithm is evaluated on a diverse set of datasets including aerial video sequences containing ariel borne dense highway scenes, top-view pedestrian trajectories, multiple similar objects, normal view pedestrians and vehicles. The effectiveness of the proposed algorithm is clearly demonstrated in these experiments.

The Dynamics of Swamps in the Canonical Tensor Approximation Problem

The ability to approximate a multivariate function/tensor as a sum of separable functions/tensors is quite useful. Unfortunately, optimization-based algorithms to do so regularly exhibit unusual tr...

Nonlocal Similarity Based Nonnegative Tucker Decomposition for Hyperspectral Image Denoising

Compared with color or grayscale images, hyperspectral images deliver more informative representation of ground objects and enhance the performance of many recognition and classification applications. However, hyperspectral images are normally corrupted by various types of noises, which have a serious impact on the subsequent image processing tasks. In this paper, we propose a novel hyperspectral image denoising method based on tucker decomposition to model the nonlocal similarity across the spatial domain and global similarity along the spectral domain. In this method, 3-D full band patches extracted from a hyperspectral image are grouped to form a third-order tensor by utilizing the nonlocal similarity in a proper window size. In this way, the task of image denoising is transformed into a high-order tensor approximation problem, which can be solved by nonnegative tucker decomposition. Instead of a traditional alternative least square based tucker decomposition, we propose a hierarchical least square based nonnegative tucker decomposition method to reduce the computational cost and improve the denoising effect. In addition, an iterative denoising strategy is adopted to achieve better denoising outcome in practice. Experimental results on three datasets show that the proposed method outperforms several state-of-the-art methods and significantly enhances the quality of the corrupted hyperspectral image.

On the reduction of multivariate quadratic systems to best rank-1 approximation of three-way tensors

In this paper, we show that a general quadratic multivariate system in the real field can be reduced to a best rank-1 three-way tensor approximation problem. This fact provides a new approach to tackle a system of quadratic polynomials equations. Some experiments using the standard alternating least squares (ALS) algorithm are drawn to evince the usefulness of rank-1 tensor approximation methods.

Rank-1 Tensor Properties with Applications to a Class of Tensor Optimization Problems

This paper studies models and algorithms for a class of tensor optimization problems, based on a rank-1 equivalence property between a tensor and certain unfoldings. It is first shown that in $d$th order tensor space, the set of rank-1 tensors is the same as the intersection of $\lceil \log_2(d) \rceil$ tensor sets, of which tensors have a specific rank-1 balanced unfolding matrix. Moreover, the number $\lceil \log_2(d) \rceil$ is proved to be optimal in some sense. Based on the above equivalence property, three relaxation approaches for solving the best rank-1 tensor approximation problems are proposed, including two convex relaxations and a nonconvex one. The two convex relaxations utilize the matrix nuclear norm regularization/constraints. They have the advantage of identifying whether the solution is a global optimizer of the original problem, by computing the nuclear norm or the Frobenius norm of a certain matrix. Under certain assumptions, the optimal solution of the original problem is characterize...

Low-Rank Approximation and Completion of Positive Tensors

Unlike the matrix case, computing low-rank approximations of tensors is NP-hard and numerically ill-posed in general. Even the best rank-1 approximation of a tensor is NP-hard. In this paper, we use convex optimization to develop polynomial-time algorithms for low-rank approximation and completion of positive tensors. Our approach is to use algebraic topology to define a new (numerically well-posed) decomposition for positive tensors, which we show is equivalent to the standard tensor decomposition in important cases. Though computing this decomposition is a nonconvex optimization problem, we prove it can be exactly reformulated as a convex optimization problem. This allows us to construct polynomial-time randomized algorithms for computing this decomposition and for solving low-rank tensor approximation problems. Among the consequences is that best rank-1 approximations of positive tensors can be computed in polynomial time. Our framework is next extended to the tensor completion problem, where noisy ent...

SDPNAL $$+$$ + : a majorized semismooth Newton-CG augmented Lagrangian method for semidefinite programming with nonnegative constraints

In this paper, we present a majorized semismooth Newton-CG augmented Lagrangian method, called SDPNAL $$+$$ , for semidefinite programming (SDP) with partial or full nonnegative constraints on the matrix variable. SDPNAL $$+$$ is a much enhanced version of SDPNAL introduced by Zhao et al. (SIAM J Optim 20:1737–1765, 2010) for solving generic SDPs. SDPNAL works very efficiently for nondegenerate SDPs but may encounter numerical difficulty for degenerate ones. Here we tackle this numerical difficulty by employing a majorized semismooth Newton-CG augmented Lagrangian method coupled with a convergent 3-block alternating direction method of multipliers introduced recently by Sun et al. (SIAM J Optim, to appear). Numerical results for various large scale SDPs with or without nonnegative constraints show that the proposed method is not only fast but also robust in obtaining accurate solutions. It outperforms, by a significant margin, two other competitive publicly available first order methods based codes: (1) an alternating direction method of multipliers based solver called SDPAD by Wen et al. (Math Program Comput 2:203–230, 2010) and (2) a two-easy-block-decomposition hybrid proximal extragradient method called 2EBD-HPE by Monteiro et al. (Math Program Comput 1–48, 2014). In contrast to these two codes, we are able to solve all the 95 difficult SDP problems arising from the relaxations of quadratic assignment problems tested in SDPNAL to an accuracy of $$10^{-6}$$ efficiently, while SDPAD and 2EBD-HPE successfully solve 30 and 16 problems, respectively. In addition, SDPNAL $$+$$ appears to be the only viable method currently available to solve large scale SDPs arising from rank-1 tensor approximation problems constructed by Nie and Wang (SIAM J Matrix Anal Appl 35:1155–1179, 2014). The largest rank-1 tensor approximation problem we solved (in about 14.5 h) is nonsym(21,4), in which its resulting SDP problem has matrix dimension $$n = 9261$$ and the number of equality constraints $$m =12{,}326{,}390$$ .

Tensor decompositions with banded matrix factors

The computation of the model parameters of a Canonical Polyadic Decomposition (CPD), also known as the parallel factor (PARAFAC) or canonical decomposition (CANDECOMP) or CP decomposition, is typically done by resorting to iterative algorithms, e.g. either iterative alternating least squares type or descent methods. In many practical problems involving tensor decompositions such as signal processing, some of the matrix factors are banded. First, we develop methods for the computation of CPDs with one banded matrix factor. It results in best rank-1 tensor approximation problems. Second, we propose methods to compute CPDs with more than one banded matrix factor. Third, we extend the developed methods to also handle banded and structured matrix factors such as Hankel or Toeplitz. Computer results are also reported.

Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme

Higher-order tensors are used in many application fields, such as statistics, signal processing, and scientific computing. Efficient and reliable algorithms for manipulating these multi-way arrays are thus required. In this paper, we focus on the best rank-$(R_1,R_2,R_3)$ approximation of third-order tensors. We propose a new iterative algorithm based on the trust-region scheme. The tensor approximation problem is expressed as a minimization of a cost function on a product of three Grassmann manifolds. We apply the Riemannian trust-region scheme, using the truncated conjugate-gradient method for solving the trust-region subproblem. Making use of second order information of the cost function, superlinear convergence is achieved. If the stopping criterion of the subproblem is chosen adequately, the local convergence rate is quadratic. We compare this new method with the well-known higher-order orthogonal iteration method and discuss the advantages over Newton-type methods.

Factorization strategies for third-order tensors

Operations with tensors, or multiway arrays, have become increasingly prevalent in recent years. Traditionally, tensors are represented or decomposed as a sum of rank-1 outer products using either the CANDECOMP/PARAFAC (CP) or the Tucker models, or some variation thereof. Such decompositions are motivated by specific applications where the goal is to find an approximate such representation for a given multiway array. The specifics of the approximate representation (such as how many terms to use in the sum, orthogonality constraints, etc.) depend on the application. In this paper, we explore an alternate representation of tensors which shows promise with respect to the tensor approximation problem. Reminiscent of matrix factorizations, we present a new factorization of a tensor as a product of tensors. To derive the new factorization, we define a closed multiplication operation between tensors. A major motivation for considering this new type of tensor multiplication is to devise new types of factorizations for tensors which can then be used in applications. Specifically, this new multiplication allows us to introduce concepts such as tensor transpose, inverse, and identity, which lead to the notion of an orthogonal tensor. The multiplication also gives rise to a linear operator, and the null space of the resulting operator is identified. We extend the concept of outer products of vectors to outer products of matrices. All derivations are presented for third-order tensors. However, they can be easily extended to the order- p ( p > 3 ) case. We conclude with an application in image deblurring.

Tensor Products and Approximation Problems of $C^*$-Algebras

For C^* -algebras C and D and their C^* -subalgebras A and B , it is shown with other related results that if all irreducible representations of A are finite dimensional with bounded degree then the family of product functionals in the minimal C^* -tensor product C\otimes_\alpha D determines completely the subalgebra A\otimes_\alpha B . Relations to Effros' recent work are also discussed.