Optimal Low-rank Approximation Research Articles

Low-rank quaternion tensor completion for color video inpainting via a novel factorization strategy

Recently, a quaternion tensor product named Qt-product was proposed, and then the singular value decomposition and the rank of a third-order quaternion tensor were given. From a more applicable perspective, we extend the Qt-product and propose a novel multiplication principle for third-order quaternion tensor named gQt-product. With the gQt-product, we introduce a brand-new singular value decomposition for third-order quaternion tensors named gQt-SVD and then define gQt-rank and multi-gQt-rank. We prove that the optimal low-rank approximation of a third-order quaternion tensor exists and some numerical experiments demonstrate the low-rankness of color videos. So, we apply the low-rank quaternion tensor completion to color video inpainting problems and present alternating least-square algorithms to solve the proposed low gQt-rank and multi-gQt-rank quaternion tensor completion models. The convergence analyses of the proposed algorithms are established and some numerical experiments on various color video datasets show the high recovery accuracy and computational efficiency of our methods.

Decomposition and graphical correspondence analysis of checkerboard copulas

Abstract We analyze optimal low-rank approximations and correspondence analysis of the dependence structure given by arbitrary bivariate checkerboard copulas. Methodologically, we make use of the truncation of singular value decompositions of doubly stochastic matrices representing the copulas. The resulting (truncated) representations of the dependence structures are sparse, in particular, compared to the number of squares on the checkerboard. The additive structure of the decomposition carries through to statistical functionals of the copula, such as Kendall’s τ \tau or Spearman’s ρ \rho , and also motivates similarity measures for checkerboard copulas. We link our analysis to continuous decompositions of copula densities and copula-generating algorithms and discuss further general properties of the decomposition and its truncation. For example, truncated series might lack nonnegativity, and approximation errors increase for monotonicity-like copulas. We provide algorithms and extensions that account for and counteract these properties. The low-rank representation is illustrated for various copula examples, and some analytical results are derived. The resulting correspondence analysis profile plots are analyzed, providing graphical insights into the dependence structure implied by the copula. An illustration is provided with an empirical data set on fuel injector spray characteristics in jet engines.

Compact Model Training by Low-Rank Projection With Energy Transfer.

Low-rankness plays an important role in traditional machine learning but is not so popular in deep learning. Most previous low-rank network compression methods compress networks by approximating pretrained models and retraining. However, the optimal solution in the Euclidean space may be quite different from the one with low-rank constraint. A well-pretrained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared with other network compression methods such as pruning, low-rank methods attract less attention in recent years. In this article, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. We propose to alternately perform stochastic gradient descent training and projection of each weight matrix onto the corresponding low-rank manifold. Compared to retraining on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. The matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. In modern networks, a batch normalization (BN) layer can be merged into the previous convolution layer for inference, thereby influencing the optimal low-rank approximation (LRA) of the previous layer. We propose BN rectification to cut off its effect on the optimal LRA, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. For object detection and semantic segmentation, our method still achieves good compression results. In addition, we combine LRPET with quantization and hashing methods and achieve even better compression than the original single method. We further apply it in Transformer-based models to demonstrate its transferability. Our code is available at https://github.com/BZQLin/LRPET.

On Best Low Rank Approximation of Positive Definite Tensors

Tensors, or multi-indexed arrays, play an important role in machine learning and signal processing. These higher-order generalizations of matrices allow for preservation of higher-order structure in data, and low rank decompositions of tensors allow for recovery of underlying information. In many cases, the underlying tensor of interest is positive (semi)definite, i.e., the homogeneous polynomial associated to the tensor has nonnegative evaluation on all inputs. An archetypal problem is that one has a noisy measurement of some low rank signal tensor of interest. This measurement itself does not have low rank, so one must compute a best low rank approximation of the measured tensor to recover component information. As it turns out, the set of tensors of rank less than or equal to is, in general, not closed when , and, as a consequence, best low rank approximations can fail to exist. In the case when a best low rank approximation does not exist, near optimal low rank approximations suffer numerical issues and cannot be used to reliably approximate underlying component information. As a consequence, existence guarantees for best low rank tensor approximations are of great practical and theoretical interest. This article develops deterministic guarantees for the existence of best low rank approximations of tensors which are positive semidefinite. In particular, we show that the set of low rank positive semidefinite tensors is relatively closed as a subset of the set of positive semidefinite tensors. We use this fact to give a deterministic bound which may be used to guarantee the existence of a best low rank approximation of a noisy low rank positive semidefinite tensor. Furthermore, we show that this condition guarantees uniqueness of the canonical polyadic decomposition for best low rank approximations. In addition, for order three tensors, we prove that our bound is sharp and that it can be computed using semidefinite programming. The main results of this article are illustrated through numerical experiments, which show that our bound is highly predictive of numerical issues when attempting to recover underlying component information from a noisy low rank positive semidefinite tensor.

Non-Local Robust Quaternion Matrix Completion for Large-Scale Color Image and Video Inpainting.

The image nonlocal self-similarity (NSS) prior refers to the fact that a local patch often has many nonlocal similar patches to it across the image and has been widely applied in many recently proposed machining learning algorithms for image processing. However, there is no theoretical analysis on its working principle in the literature. In this paper, we discover a potential causality between NSS and low-rank property of color images, which is also available to grey images. A new patch group based NSS prior scheme is proposed to learn explicit NSS models of natural color images. The numerical low-rank property of patched matrices is also rigorously proved. The NSS-based QMC algorithm computes an optimal low-rank approximation to the high-rank color image, resulting in high PSNR and SSIM measures and particularly the better visual quality. A new tensor NSS-based QMC method is also presented to solve the color video inpainting problem based on quaternion tensor representation. The numerical experiments on color images and videos indicate the advantages of NSS-based QMC over the state-of-the-art methods.

Low Rank Pure Quaternion Approximation for Pure Quaternion Matrices

Quaternion matrices are employed successfully in many color image processing applications. In particular, a pure quaternion matrix can be used to represent red, green, and blue channels of color images. A low-rank approximation for a pure quaternion matrix can be obtained by using the quaternion singular value decomposition. However, this approximation is not optimal in the sense that the resulting low-rank approximation matrix may not be pure quaternion, i.e., the low-rank matrix contains a real component which is not useful for the representation of a color image. The main contribution of this paper is to find an optimal rank-$r$ pure quaternion matrix approximation for a pure quaternion matrix (a color image). Our idea is to use a projection on a low-rank quaternion matrix manifold and a projection on a quaternion matrix with zero real component, and develop an alternating projections algorithm to find such optimal low-rank pure quaternion matrix approximation. The convergence of the projection algorithm can be established by showing that the low-rank quaternion matrix manifold and the zero real component quaternion matrix manifold has a nontrivial intersection point. Numerical examples on synthetic pure quaternion matrices and color images are presented to illustrate the projection algorithm can find optimal low-rank pure quaternion approximation for pure quaternion matrices or color images.

Curing basis set overcompleteness with pivoted Cholesky decompositions.

The description of weakly bound electronic states is especially difficult with atomic orbital basis sets. The diffuse atomic basis functions that are necessary to describe the extended electronic state generate significant linear dependencies in the molecular basis set, which may make the electronic structure calculations ill-convergent. We propose a method where the overcomplete molecular basis set is pruned by a pivoted Cholesky decomposition of the overlap matrix, yielding an optimal low-rank approximation that is numerically stable, the pivot indices determining a reduced basis set that is complete enough to describe all the basis functions in the original overcomplete basis. The method can be implemented either by a simple modification to the usual canonical orthogonalization procedure, which hides the excess functions and yields fewer efficiency benefits, or by generating custom basis sets for all the atoms in the system, yielding significant cost reductions in electronic structure calculations. The pruned basis sets from the latter choice allow accurate calculations to be performed at a lower cost even at the self-consistent field level, as illustrated on a solvated (H2O)24 - anion. Our results indicate that the Cholesky procedure allows one to perform calculations with accuracies close to standard augmented basis sets with cost savings which increase with the size of the basis set, ranging from 9% fewer functions in single-ζ basis sets to 28% fewer functions in triple-ζ basis sets.

Lanczos method for large-scale quaternion singular value decomposition

In many color image processing and recognition applications, one of the most important targets is to compute the optimal low-rank approximations to color images, which can be reconstructed with a small number of dominant singular value decomposition (SVD) triplets of quaternion matrices. All existing methods are designed to compute all SVD triplets of quaternion matrices at first and then to select the necessary dominant ones for reconstruction. This way costs quite a lot of operational flops and CPU times to compute many superfluous SVD triplets. In this paper, we propose a Lanczos-based method of computing partial (several dominant) SVD triplets of the large-scale quaternion matrices. The partial bidiagonalization of large-scale quaternion matrices is derived by using the Lanczos iteration, and the reorthogonalization and thick-restart techniques are also utilized in the implementation. An algorithm is presented to compute the partial quaternion singular value decomposition. Numerical examples, including principal component analysis, color face recognition, video compression and color image completion, illustrate that the performance of the developed Lanczos-based method for low-rank quaternion approximation is better than that of the state-of-the-art methods.

A Simple Yet Efficient Evolution Strategy for Large-Scale Black-Box Optimization

We propose an evolution strategy algorithm using a sparse plus low rank model for large-scale optimization in this paper. We first develop a rank one evolution strategy using a single principal search direction. It is of linear complexity. Then we extend it to multiple search directions, and develop a rank- ${m}$ evolution strategy. We illustrate that the principal search direction accumulates the natural gradients with respect to the distribution mean, and acts as a momentum term. Further, we analyze the optimal low rank approximation to the covariance matrix, and experimentally show that the principal search direction can effectively learn the long valley of the function with predominant search direction. Then we investigate the effects of Hessian on the algorithm performance. We conduct experiments on a class of test problems and the CEC’2010 LSGO benchmarks. The experimental results validate the effectiveness of our proposed algorithms.

Optimal Data-Driven Estimation of Generalized Markov State Models for Non-Equilibrium Dynamics

There are multiple ways in which a stochastic system can be out of statistical equilibrium. It might be subject to time-varying forcing; or be in a transient phase on its way towards equilibrium; it might even be in equilibrium without us noticing it, due to insufficient observations; and it even might be a system failing to admit an equilibrium distribution at all. We review some of the approaches that model the effective statistical behavior of equilibrium and non-equilibrium dynamical systems, and show that both cases can be considered under the unified framework of optimal low-rank approximation of so-called transfer operators. Particular attention is given to the connection between these methods, Markov state models, and the concept of metastability, further to the estimation of such reduced order models from finite simulation data. All these topics bear an important role in, e.g., molecular dynamics, where Markov state models are often and successfully utilized, and which is the main motivating application in this paper. We illustrate our considerations by numerical examples.

SAR Target Recognition via Local Sparse Representation of Multi-Manifold Regularized Low-Rank Approximation

The extraction of a valuable set of features and the design of a discriminative classifier are crucial for target recognition in SAR image. Although various features and classifiers have been proposed over the years, target recognition under extended operating conditions (EOCs) is still a challenging problem, e.g., target with configuration variation, different capture orientations, and articulation. To address these problems, this paper presents a new strategy for target recognition. We first propose a low-dimensional representation model via incorporating multi-manifold regularization term into the low-rank matrix factorization framework. Two rules, pairwise similarity and local linearity, are employed for constructing multiple manifold regularization. By alternately optimizing the matrix factorization and manifold selection, the feature representation model can not only acquire the optimal low-rank approximation of original samples, but also capture the intrinsic manifold structure information. Then, to take full advantage of the local structure property of features and further improve the discriminative ability, local sparse representation is proposed for classification. Finally, extensive experiments on moving and stationary target acquisition and recognition (MSTAR) database demonstrate the effectiveness of the proposed strategy, including target recognition under EOCs, as well as the capability of small training size.

Modal Analysis of Fluid Flows: An Overview

Modal Analysis of Fluid Flows: An Overview

Optimal Low-rank Approximations of Bayesian Linear Inverse Problems

In the Bayesian approach to inverse problems, data are often informative, relative to the prior, only on a low-dimensional subspace of the parameter space. Significant computational savings can be achieved by using this subspace to characterize and approximate the posterior distribution of the parameters. We first investigate approximation of the posterior covariance matrix as a low-rank update of the prior covariance matrix. We prove optimality of a particular update, based on the leading eigendirections of the matrix pencil defined by the Hessian of the negative log-likelihood and the prior precision, for a broad class of loss functions. This class includes the Forstner metric for symmetric positive definite matrices, as well as the Kullback--Leibler divergence and the Hellinger distance between the associated distributions. We also propose two fast approximations of the posterior mean and prove their optimality with respect to a weighted Bayes risk under squared-error loss. These approximations are dep...

Image reconstruction from double random projection.

We present double random projection methods for reconstruction of imaging data. The methods draw upon recent results in the random projection literature, particularly on low-rank matrix approximations, and the reconstruction algorithm has only two simple and noniterative steps, while the reconstruction error is close to the error of the optimal low-rank approximation by the truncated singular-value decomposition. We extend the often-required symmetric distributions of entries in a random-projection matrix to asymmetric distributions, which can be more easily implementable on imaging devices. Experimental results are provided on the subsampling of natural images and hyperspectral images, and on simulated compressible matrices. Comparisons with other random projection methods are also provided.

Optimal Kullback–Leibler approximation of Markov chains via nuclear norm regularisation

This paper is concerned with model reduction for Markov chain models. The goal is to obtain a low-rank approximationto the original Markov chain. The Kullback–Leibler divergence rate is used to measure the similarity between two Markov chains; the nuclear norm is used to approximate the rank function. A nuclear-norm regularised optimisation problem is formulated to approximately find the optimal low-rank approximation. The proposed regularised problem is analysed and performance bounds are obtained through the convex analysis. An iterative fixed point algorithm is developed based on the proximal splitting technique to compute the optimal solutions. The effectiveness of this approach is illustrated via numerical examples.

Sparse Planar Array Synthesis Using Matrix Enhancement and Matrix Pencil

The matrix enhancement and matrix pencil (MEMP) plays important roles in modern signal processing applications. In this paper, MEMP is applied to attack the problem of two-dimensional sparse array synthesis. Firstly, the desired array radiation pattern, as the original pattern for approximating, is sampled to form an enhanced matrix. After performing the singular value decomposition (SVD) and discarding the insignificant singular values according to the prior approximate error, the minimum number of elements can be obtained. Secondly, in order to obtain the eigenvalues, the generalized eigen-decomposition is employed on the approximate matrix, which is the optimal low-rank approximation of the enhanced matrix corresponding to sparse planar array, and then the ESPRIT algorithm is utilized to pair the eigenvalues related to each dimension of the planar array. Finally, element positions and excitations of the sparse planar array are calculated according to the correct pairing of eigenvalues. Simulation results are presented to illustrate the effectiveness of the proposed approach.

Survey and Review

The first Survey and Review article in this issue is “Decay Properties of Spectral Projectors with Applications to Electronic Structure,” by Michele Benzi, Paolo Boito, and Nader Razouk. The linear scaling methods that motivate this work use a reformulation of the conventional electronic structure calculation. Instead of iterating through linear eigenvalue problems, a sequence of spectral projectors, known as density matrices, is computed, from which physical quantities of interest follow directly. Large-scale computations then become feasible, provided that entries in the density matrices are localized; that is, they decay rapidly as we move away from the diagonal. The authors first introduce the basic principles of electronic structure theory and then survey current computational approaches, most of which have arisen in the physical sciences, and their underlying localization requirements. The overview highlights a distinction, often blurred outside the mathematics literature, between physical insight/intuition and mathematical rigor. The key contribution is then to formulate the localization problem mathematically, with transparent assumptions, and develop a unified analytical approach. The tools of matrix analysis and approximation theory are used to justify, where feasible, the localization “results” that are relied upon by today's algorithms. The theory is framed in an asymptotic regime where the system volume increases but the density of particles remains fixed. This type of thermodynamic limit will be familiar to readers who work on multiscale models in physics and chemistry, but may seem strange to numerical analysts who are used to “convergence” in the sense of fixed problem size and arbitrarily fine mesh. The upshot is a sequence of matrices of increasing dimension that look very different from those arising in the discretization of PDEs. After pushing the theory as far as possible, the authors comment on the practical implications of their bounds and raise a number of open questions. They also point out that these decay results may find useful application in other, unrelated, areas: quantum information theory, complex networks, and eigenvalue solvers for tridiagonal matrices. This article illustrates how applied analysis can justify some of the leaps of faith made in the physical sciences and also resolve controversies (section 8.7). It will be of particular interest to readers who work in matrix computation and approximation theory. The second article, “Conditional Gradient Algorithms for Rank-One Matrix Approximations with a Sparsity Constraint,” by Ronny Luss and Marc Teboulle, shares with the first article the theme of studying algorithms based on large-scale matrix computations for which our current analytical understanding fails to explain the performance seen in practice. Given a symmetric matrix, $A$, most applied mathematicians know that dimension reduction can be achieved by using the dominant eigenvectors (singular vectors) of $A$ to form optimal low-rank approximations. In the statistics literature, modulo a centering operation, the same idea is called principal component analysis (PCA). This type of least-squares approach tries to recover all rows and columns of $A$; equivalently, if we think of $A$ as representing pairwise interactions between nodes in a network, it aims to summarize the full set of interactions. It is, of course, possible to impose a postprocessing threshold to cut off the less important contributions, but it is also reasonable to shoot directly for the important rows/columns or the important network nodes. This alternative leads us to the class of problems discussed here---matrix optimization problems where there is a prescribed upper bound on the number of nonzeros in the required “eigenvectors.” The authors discuss a variety of computational approaches that have been proposed in the literature and introduce a unifying framework, which they call ConGradU, to characterize and analyze them. The basic iterative scheme, summarized in Algorithm 1, has an optimization substep with closed-form solutions in a number of important cases. The authors also include an informative computational example based on the textual content of State of the Union addresses from 1790--2011. For example, Table 6.2 shows the key word stems associated with the three principal factors for both thresholded PCA and the more direct sparsity-seeking alternative. It is clear that the sparse version is summarizing different information. This article will appeal to readers who are keen to keep up with developments in modern optimization and to learn about techniques that fall under the current “Big Data” banner.

On the ADI method for the Sylvester equation and the optimal-[formula omitted] points

The ADI iteration is closely related to the rational Krylov projection methods for constructing low rank approximations to the solution of Sylvester equations. In this paper we show that the ADI and rational Krylov approximations are in fact equivalent when a special choice of shifts are employed in both methods. We will call these shifts pseudo H2-optimal shifts. These shifts are also optimal in the sense that for the Lyapunov equation, they yield a residual which is orthogonal to the rational Krylov projection subspace. Via several examples, we show that the pseudo H2-optimal shifts consistently yield nearly optimal low rank approximations to the solutions of the Lyapunov equations.

Inductive Robust Principal Component Analysis

In this paper we address the error correction problem that is to uncover the low-dimensional subspace structure from high-dimensional observations, which are possibly corrupted by errors. When the errors are of Gaussian distribution, Principal Component Analysis (PCA) can find the optimal (in terms of least-square-error) low-rank approximation to highdimensional data. However, the canonical PCA method is known to be extremely fragile to the presence of gross corruptions. Recently, Wright et al. established a so-called Robust Principal Component Analysis (RPCA) method, which can well handle grossly corrupted data [14]. However, RPCA is a transductive method and does not handle well the new samples which are not involved in the training procedure. Given a new datum, RPCA essentially needs to recalculate over all the data, resulting in high computational cost. So, RPCA is inappropriate for the applications that require fast online computation. To overcome this limitation, in this paper we propose an Inductive Robust Principal Component Analysis (IRPCA) method. Given a set of training data, unlike RPCA that targets on recovering the original data matrix, IRPCA aims at learning the underlying projection matrix, which can be used to efficiently remove the possible corruptions in any datum. The learning is done by solving a nuclear norm regularized minimization problem, which is convex and can be solved in polynomial time. Extensive experiments on a benchmark human face dataset and two video surveillance datasets show that IRPCA can not only be robust to gross corruptions, but also handle well the new data in an efficient way.

Canonical Polyadic Decomposition with a Columnwise Orthonormal Factor Matrix

Canonical polyadic decomposition (CPD) of a higher-order tensor is an important tool in mathematical engineering. In many applications at least one of the matrix factors is constrained to be columnwise orthonormal. We first derive a relaxed condition that guarantees uniqueness of the CPD under this constraint. Second, we give a simple proof of the existence of the optimal low-rank approximation of a tensor in the case that a factor matrix is columnwise orthonormal. Third, we derive numerical algorithms for the computation of the constrained CPD. In particular, orthogonality-constrained versions of the CPD methods based on simultaneous matrix diagonalization and alternating least squares are presented. Numerical experiments are reported.