Low-rank Matrix Factorization Research Articles

Active Subspace: Toward Scalable Low-Rank Learning

We address the scalability issues in low-rank matrix learning problems. Usually these problems resort to solving nuclear norm regularized optimization problems (NNROPs), which often suffer from high computational complexities if based on existing solvers, especially in large-scale settings. Based on the fact that the optimal solution matrix to an NNROP is often low rank, we revisit the classic mechanism of low-rank matrix factorization, based on which we present an active subspace algorithm for efficiently solving NNROPs by transforming large-scale NNROPs into small-scale problems. The transformation is achieved by factorizing the large solution matrix into the product of a small orthonormal matrix (active subspace) and another small matrix. Although such a transformation generally leads to nonconvex problems, we show that a suboptimal solution can be found by the augmented Lagrange alternating direction method. For the robust PCA (RPCA) (Candès, Li, Ma, & Wright, 2009 ) problem, a typical example of NNROPs, theoretical results verify the suboptimality of the solution produced by our algorithm. For the general NNROPs, we empirically show that our algorithm significantly reduces the computational complexity without loss of optimality.

Low-Rank Kernel Matrix Factorization for Large-Scale Evolutionary Clustering

Traditional clustering techniques are inapplicable to problems where the relationships between data points evolve over time. Not only is it important for the clustering algorithm to adapt to the recent changes in the evolving data, but it also needs to take the historical relationship between the data points into consideration. In this paper, we propose ECKF, a general framework for evolutionary clustering large-scale data based on low-rank kernel matrix factorization. To the best of our knowledge, this is the first work that clusters large evolutionary data sets by the amalgamation of low-rank matrix approximation methods and matrix factorization-based clustering. Since the low-rank approximation provides a compact representation of the original matrix, and especially, the near-optimal low-rank approximation can preserve the sparsity of the original data, ECKF gains computational efficiency and hence is applicable to large evolutionary data sets. Moreover, matrix factorization-based methods have been shown to effectively cluster high-dimensional data in text mining and multimedia data analysis. From a theoretical standpoint, we mathematically prove the convergence and correctness of ECKF, and provide detailed analysis of its computational efficiency (both time and space). Through extensive experiments performed on synthetic and real data sets, we show that ECKF outperforms the existing methods in evolutionary clustering.

Inducible regularization for low-rank matrix factorizations for collaborative filtering

Low-rank matrix factorization with missing data has become an effective methodology for collaborative filtering applications since it can generate high quality rating predictions for recommendation systems. The performance of low-rank factorization, however, critically depends on how the low-rank model is regularized in order to mitigate the over-fitting problem to the observed data. The objective of this paper is to propose a novel regularization technique which we call inducible regularization. It utilizes pre-estimated ratings on a pre-specified subset of the ratings to regularize the solutions of low-rank matrix factorization. We develop two algorithms for solving the new regularized problem via alternating least squares iterations and stochastic gradient descent. We also devise a fast implementation of the alternating least squares algorithm which is suitable for parallel computing. Numerical experiments on three real-world data sets MovieLens, Jester, and EachMovie are given for comparing the proposed algorithms with existing algorithms ALS, SGD, and SVD++ that solve low-rank matrix factorization with classical regularizations, illustrating superior performance of our proposed algorithms.

Resilience and optimization of identifiable bipartite graphs

A bipartite graph G=(L,R;E) with at least one edge is said to be identifiable if for every vertex v∈L, the subgraph induced by its non-neighbors has a matching of cardinality |L|−1. This definition arises in the context of low-rank matrix factorization and is motivated by signal processing applications.In this paper, we study the resilience of identifiability with respect to edge additions, edge deletions and edge modifications. These can all be seen as measures of evaluating how strongly a bipartite graph possesses the identifiability property. On the one hand, we show that computing the resilience of this non-monotone property can be done in polynomial time for edge additions or edge modifications. On the other hand, for edge deletions this is an NP-complete problem. Our polynomial results are based on polynomial algorithms for computing the surplus of a bipartite graph G and finding a tight set in G, which might be of independent interest.We also deal with some complexity results for the optimization problem related to the isolation of a smallest set J⊆L that, together with all vertices with neighbors only in J, induces an identifiable subgraph. We obtain an APX-hardness result for the problem and identify some polynomially solvable cases.

Graph dual regularization non-negative matrix factorization for co-clustering

Low-rank matrix factorization is one of the most useful tools in scientific computing, data mining and computer vision. Among of its techniques, non-negative matrix factorization (NMF) has received considerable attention due to producing a parts-based representation of the data. Recent research has shown that not only the observed data are found to lie on a nonlinear low dimensional manifold, namely data manifold, but also the features lie on a manifold, namely feature manifold. In this paper, we propose a novel algorithm, called graph dual regularization non-negative matrix factorization (DNMF), which simultaneously considers the geometric structures of both the data manifold and the feature manifold. We also present a graph dual regularization non-negative matrix tri-factorization algorithm (DNMTF) as an extension of DNMF. Moreover, we develop two iterative updating optimization schemes for DNMF and DNMTF, respectively, and provide the convergence proofs of our two optimization schemes. Experimental results on UCI benchmark data sets, several image data sets and a radar HRRP data set demonstrate the effectiveness of both DNMF and DNMTF. ► Graph dual regularization non-negative matrix factorization algorithm is proposed. ► Graph dual regularization non-negative matrix tri-factorization algorithm is proposed. ► DNMF and DNMTF simultaneously consider geometric structures of data and features. ► Two iterative multiplicative updating optimization schemes are presented. ► The convergence proofs of two optimization schemes are provided.

Successively alternate least square for low-rank matrix factorization with bounded missing data

The problem of low-rank matrix factorization with missing data has attracted many significant attention in the fields related to computer vision. The previous model mainly minimizes the total errors of the recovered low-rank matrix on observed entries. It may produce an optimal solution with less physical meaning. This paper gives a theoretical analysis of the sensitivities of the original model and proposes a modified constrained model and iterative methods for solving the constrained problem. We show that solutions of original model can be arbitrarily far from each others. Two kinds of sufficient conditions of this catastrophic phenomenon are given. In general case, we also give a low bound of error between an ϵ-optimal solution that is practically obtained in computation and a theoretically optimal solution. A constrained model on missing entries is considered for this missing data problem. We propose a two-step projection method for solving the constrained problem. We also modify the method by a successive alternate technique. The proposed algorithm, named as SALS, is easy to implement, as well as converges very fast even for a large matrix. Numerical experiments on simulation data and real examples are given to illuminate the algorithm behaviors of SALS.

Recent Advances in Computation and Combinatorial Optimization

This special issue contains six papers selected from 172 submissions to the Fourteenth Annual International Conference on Combinatorics and Computation (COCOON 2008), held in Dalian, China, June 2008. These papers highlight some of the current research interests and achievements in the area of theoretical computer science, ranging from practical exact algorithms for studying complex genetic diseases to improved exact algorithms for classic problems of enumerating cuts, from low-rank matrix factorization and its applications to approximation algorithms for the selected-internal Steiner minimum trees, and from resource bounded frequency computations to quantum separation of local search and fixed point computation. J. Wang, M. Xie, and J. Chen’s paper presents a practical and exact algorithm for studying the individual haplotyping problem on the computational model of Minimum Error Correction with Genotype Information (MEC/GI). Human haplotypes are believed to hold the keys to reveal the mysteries of complex genetic diseases. Computational methods have been explored to tackle challenging issues in the study of human haplotypes. In particular, MEC/GI is a computational model that can be used to infer a pair of haplotypes compatible with the genotype by correcting minimum number of single-nucleotide polymorphisms (SNPs) in the given DNA fragments. The authors develop a dynamic programming algorithm to solve the problem with running time of O(mk2k+m logm+mk), where m is the number of DNA fragments

Algorithms and applications for approximate nonnegative matrix factorization

The development and use of low-rank approximate nonnegative matrix factorization (NMF) algorithms for feature extraction and identification in the fields of text mining and spectral data analysis are presented. The evolution and convergence properties of hybrid methods based on both sparsity and smoothness constraints for the resulting nonnegative matrix factors are discussed. The interpretability of NMF outputs in specific contexts are provided along with opportunities for future work in the modification of NMF algorithms for large-scale and time-varying data sets.