Low Rank Approximation Of Matrices Research Articles

On a Problem of Weighted Low-Rank Approximation of Matrices

We study a weighted low rank approximation that is inspired by a problem of constrained low rank approximation of matrices as initiated by the work of Golub, Hoffman, and Stewart (Linear Algebra and Its Applications, 88-89(1987), 317-327). Our results reduce to that of Golub, Hoffman, and Stewart in the limiting cases. We also propose an algorithm based on the alternating direction method to solve our weighted low rank approximation problem and compare it with the state-of-art general algorithms such as the weighted total alternating least squares and the EM algorithm.

Literature survey on low rank approximation of matrices

Low rank approximation of matrices has been well studied in literature. Singular value decomposition, QR decomposition with column pivoting, rank revealing QR factorization, Interpolative decomposition, etc. are classical deterministic algorithms for low rank approximation. But these techniques are very expensive ( operations are required for matrices). There are several randomized algorithms available in the literature which are not so expensive as the classical techniques (but the complexity is not linear in n). So, it is very expensive to construct the low rank approximation of a matrix if the dimension of the matrix is very large. There are alternative techniques like Cross/Skeleton approximation which gives the low-rank approximation with linear complexity in n. In this article we review low rank approximation techniques briefly and give extensive references of many techniques.

Descriptor Learning via Supervised Manifold Regularization for Multioutput Regression.

Multioutput regression has recently shown great ability to solve challenging problems in both computer vision and medical image analysis. However, due to the huge image variability and ambiguity, it is fundamentally challenging to handle the highly complex input-target relationship of multioutput regression, especially with indiscriminate high-dimensional representations. In this paper, we propose a novel supervised descriptor learning (SDL) algorithm for multioutput regression, which can establish discriminative and compact feature representations to improve the multivariate estimation performance. The SDL is formulated as generalized low-rank approximations of matrices with a supervised manifold regularization. The SDL is able to simultaneously extract discriminative features closely related to multivariate targets and remove irrelevant and redundant information by transforming raw features into a new low-dimensional space aligned to targets. The achieved discriminative while compact descriptor largely reduces the variability and ambiguity for multioutput regression, which enables more accurate and efficient multivariate estimation. We conduct extensive evaluation of the proposed SDL on both synthetic data and real-world multioutput regression tasks for both computer vision and medical image analysis. Experimental results have shown that the proposed SDL can achieve high multivariate estimation accuracy on all tasks and largely outperforms the algorithms in the state of the arts. Our method establishes a novel SDL framework for multioutput regression, which can be widely used to boost the performance in different applications.

Robust Generalized Low Rank Approximations of Matrices.

In recent years, the intrinsic low rank structure of some datasets has been extensively exploited to reduce dimensionality, remove noise and complete the missing entries. As a well-known technique for dimensionality reduction and data compression, Generalized Low Rank Approximations of Matrices (GLRAM) claims its superiority on computation time and compression ratio over the SVD. However, GLRAM is very sensitive to sparse large noise or outliers and its robust version does not have been explored or solved yet. To address this problem, this paper proposes a robust method for GLRAM, named Robust GLRAM (RGLRAM). We first formulate RGLRAM as an l 1-norm optimization problem which minimizes the l 1-norm of the approximation errors. Secondly, we apply the technique of Augmented Lagrange Multipliers (ALM) to solve this l 1-norm minimization problem and derive a corresponding iterative scheme. Then the weak convergence of the proposed algorithm is discussed under mild conditions. Next, we investigate a special case of RGLRAM and extend RGLRAM to a general tensor case. Finally, the extensive experiments on synthetic data show that it is possible for RGLRAM to exactly recover both the low rank and the sparse components while it may be difficult for previous state-of-the-art algorithms. We also discuss three issues on RGLRAM: the sensitivity to initialization, the generalization ability and the relationship between the running time and the size/number of matrices. Moreover, the experimental results on images of faces with large corruptions illustrate that RGLRAM obtains the best denoising and compression performance than other methods.

Incremental generalized low rank approximation of matrices for visual learning and recognition

In this paper, we consider the problem of incrementally computing low-rank approximations of matrices. We propose an incremental low rank approximation algorithm of a sequence of matrices instead of one single matrix. Based on the generalized Low Rank Approximation of matrices approach, the proposed method uses an iterative approach to update the generated low rank structures. We tested the proposed incremental learning method on five face databases to evaluate its efficiency in terms of recognition rate and compared its performance with the Incremental Singular Value Decomposition method. We also showed that the overall reconstruction error is kept bounded during the incremental learning process.

A fast numerical method for the Cauchy problem for the Smoluchowski equation

A new solution technique is proposed for one-dimensional Smoluchowski equations. It is based on the finite-difference predictor–corrector scheme and is faster than other methods using this kind of scheme. The new technique capitalizes on low-rank approximations of matrices arising after discretization of the coagulation kernel and includes a new fast convolution algorithm with the trapezoidal quadrature rule. For the grids with N nodes, the complexity of the new method is O(Nlog⁡N) for each step with time instead of O(N2) operations required by the standard implementation of the predictor–corrector scheme.

Inexact and incremental bilinear Lanczos components algorithms for high dimensionality reduction and image reconstruction

Recently, matrix-based methods have gained wide attentions in pattern recognition and machine learning communities. The generalized low rank approximations of matrices (GLRAM) and the bilinear Lanczos components (BLC) algorithm are two popular algorithms that treat data as the native two-dimensional matrix patterns. However, these two algorithms often require heavy computation time and memory space in practice, especially for large scale problems. In this paper, we propose inexact and incremental bilinear Lanczos components algorithms for high dimensionality reduction and image reconstruction. We first introduce the thick-restarting strategy to the BLC algorithm, and present a thick-restarted Lanczos components (TRBLC) algorithm. In this algorithm, we use the Ritz vectors as approximations to dominant eigenvectors instead of the Lanczos vectors. In our implementation, the iterative matrices are neither formed nor stored explicitly, thanks to the characteristic of the Lanczos procedure. Then, we explore the relationship between the reconstruction error and the accuracy of the Ritz vectors, so that the computational complexities of eigenpairs can be reduced significantly. As a result, we propose an inexact thick-restarted Lanczos components (Inex-TRBLC) algorithm. Moreover, we investigate the problem of incremental generalized low rank approximations of matrices, and propose an incremental and inexact TRBLC (Incr-TRBLC) algorithm. Numerical experiments illustrate the superiority of the new algorithms over the GLRAM algorithm and its variations, as well as the BLC algorithm for some real-world image reconstruction and face recognition problems.

A partial proximal point algorithm for nuclear norm regularized matrix least squares problems

We introduce a partial proximal point algorithm for solving nuclear norm regularized matrix least squares problems with equality and inequality constraints. The inner subproblems, reformulated as a system of semismooth equations, are solved by an inexact smoothing Newton method, which is proved to be quadratically convergent under a constraint non-degeneracy condition, together with the strong semi-smoothness property of the singular value thresholding operator. Numerical experiments on a variety of problems including those arising from low-rank approximations of transition matrices show that our algorithm is efficient and robust.

A QUINTET SINGULAR VALUE DECOMPOSITION THROUGH EMPIRICAL MODE DECOMPOSITIONS

A particular quintet singular valued decomposition (Quintet-SVD) is introduced in this paper via empirical mode decompositions (EMDs). The Quintet-SVD results in four specific orthogonal matrices with a diagonal matrix of singular values. Furthermore, this paper shows relationships between the Quintet-SVD and traditional SVD, generalized low rank approximations of matrices (GLRAM) of one single matrix, and EMDs. One application of the Quintet-SVD for speech enhancement is shown and compared with an application of traditional SVD.

Spatially correlated nonnegative matrix factorization

Low rank approximation of matrices has been frequently applied in information processing tasks, and in recent years, Nonnegative Matrix Factorization (NMF) has received considerable attentions for its straightforward interpretability and superior performance. When applied to image processing, ordinary NMF merely views a p1×p2 image as a vector in p1×p2-dimensional space and the pixels of the image are considered as independent. It fails to consider the fact that an image displayed in the plane is intrinsically a matrix, and pixels spatially close to each other may probably be correlated. Even though we have p1×p2 pixels per image, this spatial correlation suggests that the real number of freedom is far less. In this paper, we introduce a Spatially Correlated Nonnegative Matrix Factorization algorithm, which explicitly models the spatial correlation between neighboring pixels in the parts-based image representation. A multiplicative updating algorithm is also proposed to solve the corresponding optimization problem. Experimental results on benchmark image data sets demonstrate the effectiveness of the proposed method.

A projector-splitting integrator for dynamical low-rank approximation

The dynamical low-rank approximation of time-dependent matrices is a low-rank factorization updating technique. It leads to differential equations for factors of the matrices, which need to be solved numerically. We propose and analyze a fully explicit, computationally inexpensive integrator that is based on splitting the orthogonal projector onto the tangent space of the low-rank manifold. As is shown by theory and illustrated by numerical experiments, the integrator enjoys robustness properties that are not shared by any standard numerical integrator. This robustness can be exploited to change the rank adaptively. Another application is in optimization algorithms for low-rank matrices where truncation back to the given low rank can be done efficiently by applying a step of the integrator proposed here.

An equi-directional generalization of adaptive cross approximation for higher-order tensors

A scheme for the equi-directional approximation of third-order tensors by sums of products of vectors is presented and analyzed. The new method is a generalization of the adaptive cross approximation (ACA) method initially designed for the low-rank approximation of matrices using only few of its original entries. Moreover, a recursive formula is presented which generalizes the three-way ACA to tensors of arbitrary order.

Rank-preserving geometric means of positive semi-definite matrices

The generalization of the geometric mean of positive scalars to positive definite matrices has attracted considerable attention since the seminal work of Ando. The paper generalizes this framework of matrix means by proposing the definition of a rank-preserving mean for two or an arbitrary number of positive semi-definite matrices of fixed rank. The proposed mean is shown to be geometric in that it satisfies all the expected properties of a rank-preserving geometric mean. The work is motivated by operations on low-rank approximations of positive definite matrices in high-dimensional spaces.

Adaptation of Hidden Markov Models Using Model-as-Matrix Representation

In this paper, we describe basis-based speaker adaptation techniques using the matrix representation of training models. Bases are obtained from training models by decomposition techniques for matrix-variate objects: two-dimensional principal component analysis (2DPCA) and generalized low rank approximations of matrices (GLRAM). The motivation for using matrix representation is that the sample covariance matrix of training models can be more accurately computed and the speaker weight becomes a matrix. Speaker adaptation equations are derived in the maximum-likelihood (ML) framework, and the adaptation equations can be solved using the maximum-likelihood linear regression technique. Additionally, novel applications of probabilistic 2DPCA and GLRAM to speaker adaptation are presented. From the probabilistic 2DPCA/GLRAM of training models, speaker adaptation equations are formulated in the maximum a posteriori (MAP) framework. The adaptation equations can be solved using the MAP linear regression technique. In the isolated-word experiments, the matrix representation-based methods in the ML and MAP frameworks outperformed maximum-likelihood linear regression adaptation, MAP adaptation, eigenvoice, and probabilistic PCA-based model for adaptation data longer than 20 s. Furthermore, the adaptation methods using probabilistic 2DPCA/GLRAM showed additional performance improvement over the adaptation methods using 2DPCA/GLRAM for small amounts of adaptation data.

Fast low rank approximations of matrices and tensors

In many applications such as data compression, imaging or genomic data analysis, it is important to approximate a given $m\times n$ matrix $A$ by a matrix $B$ of rank at most $k$ which is much smaller than $m$ and $n$. The best rank $k$ approximation can be determined via the singular value decomposition which, however, has prohibitively high computational complexity and storage requirements for very large $m$ and $n$. We present an optimal least squares algorithm for computing a rank $k$ approximation to an $m\times n$ matrix $A$ by reading only a limited number of rows and columns of $A$. The algorithm has complexity $\mathcal O(k^2\max(m,n))$ and allows to iteratively improve given rank $k$ approximations by reading additional rows and columns of $A$. We also show how this approach can be extended to tensors and present numerical results.

Tensor-Train Decomposition

A simple nonrecursive form of the tensor decomposition in d dimensions is presented. It does not inherently suffer from the curse of dimensionality, it has asymptotically the same number of parameters as the canonical decomposition, but it is stable and its computation is based on low-rank approximation of auxiliary unfolding matrices. The new form gives a clear and convenient way to implement all basic operations efficiently. A fast rounding procedure is presented, as well as basic linear algebra operations. Examples showing the benefits of the decomposition are given, and the efficiency is demonstrated by the computation of the smallest eigenvalue of a 19-dimensional operator.

Bilinear Lanczos components for fast dimensionality reduction and feature extraction

Matrix-based methods such as generalized low rank approximations of matrices (GLRAM) have gained wide attention from researchers in pattern recognition and machine learning communities. In this paper, a novel concept of bilinear Lanczos components (BLC) is introduced to approximate the projection vectors obtained from eigen-based methods without explicit computing eigenvectors of the matrix. This new method sequentially reduces the reconstruction error for a Frobenius-norm based optimization criterion, and the resulting approximation performance is thus improved during successive iterations. In addition, a theoretical clue for selecting suitable dimensionality parameters without losing classification information is presented in this paper. The BLC approach realizes dimensionality reduction and feature extraction by using a small number of Lanczos components. Extensive experiments on face recognition and image classification are conducted to evaluate the efficiency and effectiveness of the proposed algorithm. Results show that the new approach is competitive with the state-of-the-art methods, while it has a much lower training cost.

Generalized Low-Rank Approximations of Matrices Revisited

Compared to singular value decomposition (SVD), generalized low-rank approximations of matrices (GLRAM) can consume less computation time, obtain higher compression ratio, and yield competitive classification performance. GLRAM has been successfully applied to applications such as image compression and retrieval, and quite a few extensions have been successively proposed. However, in literature, some basic properties and crucial problems with regard to GLRAM have not been explored or solved yet. For this sake, we revisit GLRAM in this paper. First, we reveal such a close relationship between GLRAM and SVD that GLRAM's objective function is identical to SVD's objective function except the imposed constraints. Second, we derive a lower bound of GLRAM's objective function, and discuss when the lower bound can be touched. Moreover, from the viewpoint of minimizing the lower bound, we answer one open problem raised by Ye (Machine Learning, 2005), i.e., a theoretical justification of the experimental phenomenon that, under given number of reduced dimension, the lowest reconstruction error is obtained when the left and right transformations have equal number of columns. Third, we explore when and why GLRAM can perform well in terms of compression, which is a fundamental problem concerning the usability of GLRAM.

Speaker adaptation using generalised low rank approximations of training matrices

A speaker adaptation method based on the low rank approximation of matrices (GLRAM) of training models is described. In the method, each model is represented as a matrix, and a set of such training matrices is decomposed into a set of speaker weights and two basis matrices for row and column spaces by reducing both row and column ranks of the training models. As a result, the speaker weight becomes a matrix, the row and column dimensions of which can be adjusted. In the isolated-word experiment, the proposed method showed better performance than both eigenvoice and MLLR for the adaptation data of about 20 s or longer.

A Randomized Algorithm for Principal Component Analysis

Principal component analysis (PCA) requires the computation of a low-rank approximation to a matrix containing the data being analyzed. In many applications of PCA, the best possible accuracy of any rank-deficient approximation is at most a few digits (measured in the spectral norm, relative to the spectral norm of the matrix being approximated). In such circumstances, efficient algorithms have not come with guarantees of good accuracy, unless one or both dimensions of the matrix being approximated are small. We describe an efficient algorithm for the low-rank approximation of matrices that produces accuracy that is very close to the best possible accuracy, for matrices of arbitrary sizes. We illustrate our theoretical results via several numerical examples.