Approximate Singular Value Decomposition Research Articles

Fast and efficient algorithm for matrix completion via closed-form 2/3-thresholding operator

Matrix completion is arguably one of the most studied problems in machine learning and data analysis. Inspired by the closed-form formulas for L2/3 regularization, we employ the Schatten 2/3 quasi-norm to approximate the rank of a matrix, which can provide a better approximation than traditional ways. We also establish a necessary optimal condition and propose a fixed point iterative scheme for solving L2/3 regularization problem. Through analysing the monotonicity and the accumulation point of L2/3 regularization problem, the convergence of this iteration is analysed. By discussing the optimal selection of the regularization parameter together with a fast Monte Carlo algorithm and an approximate singular value decomposition (SVD) procedure, we build a fast and efficient algorithm that solves the induced optimization problem well. Extensive experiments have been conducted and the results show that the proposed algorithm is fast, efficient and robust. Specifically, we compare the proposed algorithm with state-of-the-art matrix completion algorithms on many synthetic data and large recommendation datasets. Our proposed algorithm is able to achieve similar or better prediction performance, while being faster and more efficient than alternatives. Furthermore, we demonstrate the effectiveness of our proposed algorithm by solving image inpainting problems.

Matrix decompositions using sub-Gaussian random matrices

Abstract In recent years, several algorithms which approximate matrix decomposition have been developed. These algorithms are based on metric conservation features for linear spaces of random projection types. We present a new algorithm, which achieves with high probability a rank-$r$ singular value decomposition (SVD) approximation of an $n \times n$ matrix and derive an error bound that does not depend on the first $r$ singular values. Although the algorithm has an asymptotic complexity similar to state-of-the-art algorithms and the proven error bound is not as tight as the state-of-the-art bound, experiments show that the proposed algorithm is faster in practice while providing the same error rates as those of the state-of-the-art algorithms. We also show that an i.i.d. sub-Gaussian matrix with large probability of having null entries is metric conserving. This result is used in the SVD approximation algorithm, as well as to improve the performance of a previously proposed approximated LU decomposition algorithm.

A Fast and Accurate Matrix Completion Method Based on QR Decomposition and L2,1 -Norm Minimization.

Low-rank matrix completion aims to recover matrices with missing entries and has attracted considerable attention from machine learning researchers. Most of the existing methods, such as weighted nuclear-norm-minimization-based methods and Qatar Riyal (QR)-decomposition-based methods, cannot provide both convergence accuracy and convergence speed. To investigate a fast and accurate completion method, an iterative QR-decomposition-based method is proposed for computing an approximate singular value decomposition. This method can compute the largest singular values of a matrix by iterative QR decomposition. Then, under the framework of matrix trifactorization, a method for computing an approximate SVD based on QR decomposition (CSVD-QR)-based L2,1 -norm minimization method (LNM-QR) is proposed for fast matrix completion. Theoretical analysis shows that this QR-decomposition-based method can obtain the same optimal solution as a nuclear norm minimization method, i.e., the L2,1 -norm of a submatrix can converge to its nuclear norm. Consequently, an LNM-QR-based iteratively reweighted L2,1 -norm minimization method (IRLNM-QR) is proposed to improve the accuracy of LNM-QR. Theoretical analysis shows that IRLNM-QR is as accurate as an iteratively reweighted nuclear norm minimization method, which is much more accurate than the traditional QR-decomposition-based matrix completion methods. Experimental results obtained on both synthetic and real-world visual data sets show that our methods are much faster and more accurate than the state-of-the-art methods.

Weak fault feature extraction of rolling bearings based on globally optimized sparse coding and approximate SVD

Abstract Fault feature extraction is crucial to condition monitoring and fault prognostics. However, when fault is in the initial stage, it is often very weak and submerged in the strong noise. This makes the fault feature very difficult to be extracted. In this paper, we propose a novel method based on sparse representation theory. It is inspired by the traditional K-SVD based de-noising method and can penetrate into the underlying structure of the signal. It learns sparse coefficients and dictionary from the noisy signal itself. The coefficients are globally optimized based on an l 1 -regularized least square problem solving method, which can locate the impulse coordinates more accurately compared with orthonormal matching pursuit (OMP) applied in the traditional K-SVD. The dictionary learning is based on an approximation of singular value decomposition (SVD). With the learned dictionary, we can capture the higher-level structure of the signal. Combining the sparse coefficients and the learned dictionary, we can de-noise the signal effectively and extract the incipient weak fault features of rolling bearings. The results of processing both simulated and experimental signals are illustrated and both validate the proposed method. All the experimental data are also processed by SpaEIAD, wavelet shrinkage, and fast kurtogram for comparison.

Validation of the ASVDADD Constraint Selection Algorithm for Effective RCCE Modeling of Natural Gas Ignition in Air

The rate-controlled constrained-equilibrium (RCCE) model reduction scheme for chemical kinetics provides acceptable accuracies in predicting hydrocarbon ignition delays by solving a smaller number of differential equations than the number of species in the underlying detailed kinetic model (DKM). To yield good approximations, the method requires accurate identification of the rate controlling constraints. Until recently, a drawback of the RCCE scheme has been the absence of a systematic procedure capable of identifying optimal constraints for a given range of thermodynamic conditions and a required level of approximation. A recent methodology has proposed for such identification an algorithm based on a simple algebraic analysis of the results of a preliminary simulation of the underlying DKM, focused on the degrees of disequilibrium (DoD) of the individual chemical reactions. It is based on computing an approximate singular value decomposition of the actual degrees of disequilibrium (ASVDADD) obtained from the DKM simulation. The effectiveness and robustness of the method have been demonstrated for methane/oxygen ignition by considering a C1/H/O (29 species/133 reactions) submechanism of the GRI-Mech 3.0 scheme and comparing the results of a DKM simulation with those of RCCE simulations based on increasing numbers of ASVDADD constraints. Here, we demonstrate the new method for shock-tube ignition of a natural gas/air mixture, with higher hydrocarbons approximately represented by propane according to the full (53 species/325 reactions) GRI-Mech 3.0 scheme including NOx formation.

Fast SVD With Random Hadamard Projection for Hyperspectral Dimensionality Reduction

While data-dependent dimensionality reduction has dominated in many applications of hyperspectral imagery, there is increasing interest in data-independent strategies—such as random projections—due to their promise for reduced computational complexity as well as their demonstrated ability to preserve application-important information. Such random-projection-based dimensionality reduction is investigated in the specific context of supervised hyperspectral classification. Both Hadamard- and Gaussian-based random projections are considered, applied alone as well as incorporated into a fast approximate singular value decomposition (SVD). Experimental results reveal that the proposed Hadamard-based random projection with the fast SVD (FSVD) offers a computationally attractive alternative to not only traditional SVD but also Gaussian-based FSVD for dimensionality reduction in hyperspectral classification.

Matrix and Tensor Based Methods for Missing Data Estimation in Large Traffic Networks

Intelligent transportation systems (ITSs) gather information about traffic conditions by collecting data from a wide range of on-ground sensors. The collected data usually suffer from irregular spatial and temporal resolution. Consequently, missing data is a common problem faced by ITSs. In this paper, we consider the problem of missing data in large and diverse road networks. We propose various matrix and tensor based methods to estimate these missing values by extracting common traffic patterns in large road networks. To obtain these traffic patterns in the presence of missing data, we apply fixed-point continuation with approximate singular value decomposition, canonical polyadic decomposition, least squares, and variational Bayesian principal component analysis. For analysis, we consider different road networks, each of which is composed of around 1500 road segments. We evaluate the performance of these methods in terms of estimation accuracy, variance of the data set, and the bias imparted by these methods.

Degree of Disequilibrium analysis for automatic selection of kinetic constraints in the Rate-Controlled Constrained-Equilibrium method

The Rate-Controlled Constrained-Equilibrium (RCCE) model reduction scheme for chemical kinetics provides acceptable accuracies with a number of differential equations much lower than the number of species in the underlying Detailed Kinetic Model (DKM). To yield good approximations, however, the method requires accurate identification of the rate controlling constraints. So far, a drawback of the RCCE scheme has been the absence of a fully automatable and systematic procedure that is capable of identifying the best constraints for a given range of thermodynamic conditions and a required level of approximation. In this paper, we propose a new methodology for such identification based on a simple algebraic analysis of the results of a preliminary simulation of the underlying DKM, which is focused on the behaviour of the degrees of disequilibrium (DoD) of the individual chemical reactions. The new methodology is based on computing an Approximate Reduced Row Echelon Form of the Actual Degrees of Disequilibrium (ARREFADD) with respect to a preset tolerance level. An alternative variant is to select an Approximate Singular Value Decomposition of the Actual Degrees of Disequilibrium (ASVDADD). Either procedure identifies a low dimensional subspace in the DoD space, from which the actual DoD traces do not depart beyond a fixed distance related to the preset tolerance (ARREFADD methodology) or to the first neglected singular value of the matrix of DoD traces (ASVDADD methodology). The effectiveness and robustness of the method is demonstrated for the case of a very rapid supersonic nozzle expansion of the products of hydrogen and methane oxycombustion and for the case of methane/oxygen ignition. The results are in excellent agreement with DKM predictions. For both variants of the method, we provide a simple Matlab code implementing the proposed constraint selection algorithm.

Randomized Dimensionality Reduction for <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-Means Clustering

We study the topic of dimensionality reduction for $k$ -means clustering. Dimensionality reduction encompasses the union of two approaches: 1) feature selection and 2) feature extraction. A feature selection-based algorithm for $k$ -means clustering selects a small subset of the input features and then applies $k$ -means clustering on the selected features. A feature extraction-based algorithm for $k$ -means clustering constructs a small set of new artificial features and then applies $k$ -means clustering on the constructed features. Despite the significance of $k$ -means clustering as well as the wealth of heuristic methods addressing it, provably accurate feature selection methods for $k$ -means clustering are not known. On the other hand, two provably accurate feature extraction methods for $k$ -means clustering are known in the literature; one is based on random projections and the other is based on the singular value decomposition (SVD). This paper makes further progress toward a better understanding of dimensionality reduction for $k$ -means clustering. Namely, we present the first provably accurate feature selection method for $k$ -means clustering and, in addition, we present two feature extraction methods. The first feature extraction method is based on random projections and it improves upon the existing results in terms of time complexity and number of features needed to be extracted. The second feature extraction method is based on fast approximate SVD factorizations and it also improves upon the existing results in terms of time complexity. The proposed algorithms are randomized and provide constant-factor approximation guarantees with respect to the optimal $k$ -means objective value.

Large-scale Nyström kernel matrix approximation using randomized SVD.

The Nyström method is an efficient technique for the eigenvalue decomposition of large kernel matrices. However, to ensure an accurate approximation, a sufficient number of columns have to be sampled. On very large data sets, the singular value decomposition (SVD) step on the resultant data submatrix can quickly dominate the computations and become prohibitive. In this paper, we propose an accurate and scalable Nyström scheme that first samples a large column subset from the input matrix, but then only performs an approximate SVD on the inner submatrix using the recent randomized low-rank matrix approximation algorithms. Theoretical analysis shows that the proposed algorithm is as accurate as the standard Nyström method that directly performs a large SVD on the inner submatrix. On the other hand, its time complexity is only as low as performing a small SVD. Encouraging results are obtained on a number of large-scale data sets for low-rank approximation. Moreover, as the most computational expensive steps can be easily distributed and there is minimal data transfer among the processors, significant speedup can be further obtained with the use of multiprocessor and multi-GPU systems.

Incremental N-mode SVD for large-scale multilinear generative models.

Tensor decomposition is frequently used in image processing and machine learning for its ability to express higher order characteristics of data. Among tensor decomposition methods, N-mode singular value decomposition (SVD) is widely used owing to its simplicity. However, the data dimension often becomes too large to perform N-mode SVD directly due to memory limitation. An incremental method to N-mode SVD can be used to resolve this issue, but existing approaches only provide a result, which is just enough to solve discriminative problems, not the full factorization result. In this paper, we present a complete derivation of the incremental N-mode SVD, which can be applied to generative models, accompanied by a technique that can reduce the computational cost by reordering calculations. The proposed incremental N-mode SVD can also be used effectively to update the current result of N-mode SVD when new training data is received. The proposed method provides a very good approximation of N -mode SVD for the experimental data, and requires much less computation in updating a multilinear model.

Exact minimum rank approximation via Schatten [formula omitted]-norm minimization

Minimizing the rank of a matrix with a given system of affine constraints is to recover the lowest-rank matrix with many important applications in engineering and science. A convex relaxation of the rank minimization problem by minimizing the nuclear norm instead of the rank of the matrix has recently been proposed. Recht and Fazel concluded that nuclear norm minimization subject to affine constraints is equivalent to rank minimization under a certain condition in terms of the rank-restricted isometry property. In this paper, we extend some recent results from nuclear norm minimization to Schatten p-norm minimization and present a theoretical guarantee for the Schatten p-norm minimization if a certain restricted isometry property holds for the linear affine transform. Our results improve on the previous works where recovery is used for nuclear norm minimization. An algorithm based on the Majorization Minimization algorithm has been proposed to solve Schatten p-norm minimization. By using an approximate singular value decomposition procedure, we obtain a fast and robust algorithm. The numerical results on the matrix completion problem with noisy measurements indicate that our algorithm gives a more accurate reconstruction and takes less time compared with some other existing algorithms.

Nonlinear Dimensionality Reduction via the ENH-LTSA Method for Hyperspectral Image Classification

The problems of neglecting spatial features in hyperspectral imagery (HSI) and the high complexity of Local Tangent Space Alignment (LTSA) still exist in the nonlinear dimensionality reduction with LTSA for classification. Therefore, this paper proposes an innovative ENH-LTSA (Enhanced-Local Tangent Space Alignment) method to solve the two problems. First, random projection is introduced to preliminarily reduce the dimension of HSI data. It aims to improve the speed of neighbor searching and the local tangent space construction. Then, the new method presents the similarity measure via the adaptive weighted summation kernel (AWSK) distance. The AWSK distance considers both spectral and spatial features in HSI data, and attempts to ameliorate the k-nearest neighbors (KNNs) of each pixel. Furthermore, the adaptive spatial window is proposed to automatically estimate the proper window size for the description of spatial features. After that, fast approximate KNNs graph construction via Recursive Lanczos Bisection is incorporated into the new method to reduce the complexity of KNNs searching. When finishing constructing each local tangent space, the new method uses a fast low-rank approximate singular value decomposition to speed up eigenvalue decomposition of the global alignment matrix that is constituted with local manifold coordinates. Five groups of experiments with two different HSI datasets are designed to completely analyze and testify the ENH-LTSA method. Experimental results show that ENH-LTSA outperforms LTSA, both in classification results and in computational speed.

Fast Updating Algorithms for Latent Semantic Indexing

This paper discusses a few algorithms for updating the approximate singular value decomposition (SVD) in the context of information retrieval by latent semantic indexing (LSI) methods. A unifying framework is considered which is based on Rayleigh--Ritz projection methods. First, a Rayleigh--Ritz approach for the SVD is discussed and it is then used to interpret the Zha and Simon algorithms [SIAM J. Sci. Comput., 21 (1999), pp. 782--791]. This viewpoint leads to a few alternatives whose goal is to reduce computational cost and storage requirement by projection techniques that utilize subspaces of much smaller dimension. Numerical experiments show that the proposed algorithms yield accuracies comparable to those obtained from standard ones at a much lower computational cost.

Large-scale SVD and manifold learning

This paper examines the efficacy of sampling-based low-rank approximation techniques when applied to large dense kernel matrices. We analyze two common approximate singular value decomposition tech...

FaIMS: A fast algorithm for the inverse medium problem with multiple frequencies and multiple sources for the scalar Helmholtz equation

We propose an algorithm to compute an approximate singular value decomposition (SVD) of least-squares operators related to linearized inverse medium problems with multiple events. Such factorizations can be used to accelerate matrix-vector multiplications and to precondition iterative solvers.We describe the algorithm in the context of an inverse scattering problem for the low-frequency time-harmonic wave equation with broadband and multi-point illumination. This model finds many applications in science and engineering (e.g., seismic imaging, subsurface imaging, impedance tomography, non-destructive evaluation, and diffuse optical tomography).We consider small perturbations of the background medium and, by invoking the Born approximation, we obtain a linear least-squares problem. The scheme we describe in this paper constructs an approximate SVD of the Born operator (the operator in the linearized least-squares problem). The main feature of the method is that it can accelerate the application of the Born operator to a vector.If Nω is the number of illumination frequencies, Ns the number of illumination locations, Nd the number of detectors, and N the discretization size of the medium perturbation, a dense singular value decomposition of the Born operator requires O(min(NsNωNd,N)]2×max(NsNωNd,N)) operations. The application of the Born operator to a vector requires O(NωNsμ(N)) work, where μ(N) is the cost of solving a forward scattering problem. We propose an approximate SVD method that, under certain conditions, reduces these work estimates significantly. For example, the asymptotic cost of factorizing and applying the Born operator becomes O(μ(N)Nω). We provide numerical results that demonstrate the scalability of the method.

Intelligent Control of a Sensor-Actuator System via Kernelized Least-Squares Policy Iteration

In this paper a new framework, called Compressive Kernelized Reinforcement Learning (CKRL), for computing near-optimal policies in sequential decision making with uncertainty is proposed via incorporating the non-adaptive data-independent Random Projections and nonparametric Kernelized Least-squares Policy Iteration (KLSPI). Random Projections are a fast, non-adaptive dimensionality reduction framework in which high-dimensionality data is projected onto a random lower-dimension subspace via spherically random rotation and coordination sampling. KLSPI introduce kernel trick into the LSPI framework for Reinforcement Learning, often achieving faster convergence and providing automatic feature selection via various kernel sparsification approaches. In this approach, policies are computed in a low-dimensional subspace generated by projecting the high-dimensional features onto a set of random basis. We first show how Random Projections constitute an efficient sparsification technique and how our method often converges faster than regular LSPI, while at lower computational costs. Theoretical foundation underlying this approach is a fast approximation of Singular Value Decomposition (SVD). Finally, simulation results are exhibited on benchmark MDP domains, which confirm gains both in computation time and in performance in large feature spaces.

Convergence of Fixed-Point Continuation Algorithms for Matrix Rank Minimization

The matrix rank minimization problem has applications in many fields, such as system identification, optimal control, low-dimensional embedding, etc. As this problem is NP-hard in general, its convex relaxation, the nuclear norm minimization problem, is often solved instead. Recently, Ma, Goldfarb and Chen proposed a fixed-point continuation algorithm for solving the nuclear norm minimization problem (Math. Program., doi: 10.1007/s10107-009-0306-5 , 2009). By incorporating an approximate singular value decomposition technique in this algorithm, the solution to the matrix rank minimization problem is usually obtained. In this paper, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving affinely constrained matrix rank minimization problems are reported.

A Practical Subspace Approach To Landmarking

A probabilistic, maximum aposteriori approach to finding landmarks in a face image is proposed, which provides a theoretical framework for template based landmarkers. One such landmarker, based on a likelihood ratio detector, is discussed in detail. Special attention is paid to training and implementation issues, in order to minimize storage and processing requirements. In particular a fast approximate singular value decomposition method is proposed to speed up the training process and implementation of the landmarker in the Fourier domain is presented that will speed up the search process. A subspace method for outlier correction and an iterative implementation of the landmarker are both shown to improve its accuracy. The impact of carefully tuning the many parameters of the method is illustrated. The method is extensively tested and compared with alternatives.

Fixed point and Bregman iterative methods for matrix rank minimization

The linearly constrained matrix rank minimization problem is widely applicable in many fields such as control, signal processing and system identification. The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization. Although the latter can be cast as a semidefinite programming problem, such an approach is computationally expensive to solve when the matrices are large. In this paper, we propose fixed point and Bregman iterative algorithms for solving the nuclear norm minimization problem and prove convergence of the first of these algorithms. By using a homotopy approach together with an approximate singular value decomposition procedure, we get a very fast, robust and powerful algorithm, which we call FPCA (Fixed Point Continuation with Approximate SVD), that can solve very large matrix rank minimization problems (the code can be downloaded from http://www.columbia.edu/~sm2756/FPCA.htmfor non-commercial use). Our numerical results on randomly generated and real matrix completion problems demonstrate that this algorithm is much faster and provides much better recoverability than semidefinite programming solvers such as SDPT3. For example, our algorithm can recover 1000 × 1000 matrices of rank 50 with a relative error of 10−5 in about 3 min by sampling only 20% of the elements. We know of no other method that achieves as good recoverability. Numerical experiments on online recommendation, DNA microarray data set and image inpainting problems demonstrate the effectiveness of our algorithms.