Nonnegative Matrix Factorization Problem Research Articles

An efficient iterative method for solving the graph regularization Q-weighted nonnegative matrix factorization problem in multi-view clustering

In this paper, we consider the graph regularization Q-weighted nonnegative matrix factorization problem in multi-view clustering. Based on the Q-weighted norm property, this problem is transformed into the minimization problem of the trace function. The necessary condition for the existence of a solution is given. The proximal alternating nonnegative least squares method and its acceleration method are designed to solve it. The convergence theorem is also given. The feasibility and effectiveness of the proposed methods are verified by numerical experiments.

On solving a revised model of the nonnegative matrix factorization problem by the modified adaptive versions of the Dai–Liao method

AbstractWe suggest a revised form of a classic measure function to be employed in the optimization model of the nonnegative matrix factorization problem. More exactly, using sparse matrix approximations, the revision term is embedded to the model for penalizing the ill-conditioning in the computational trajectory to obtain the factorization elements. Then, as an extension of the Euclidean norm, we employ the ellipsoid norm to gain adaptive formulas for the Dai–Liao parameter in a least-squares framework. In essence, the parametric choices here are obtained by pushing the Dai–Liao direction to the direction of a well-functioning three-term conjugate gradient algorithm. In our scheme, the well-known BFGS and DFP quasi–Newton updating formulas are used to characterize the positive definite matrix factor of the ellipsoid norm. To see at what level our model revisions as well as our algorithmic modifications are effective, we seek some numerical evidence by conducting classic computational tests and assessing the outputs as well. As reported, the results weigh enough value on our analytical efforts.

Alternating block linearized Bregman iterations for regularized nonnegative matrix factorization

In this paper, we propose an alternating block variant of the linearized Bregman iterations for a class of regularized nonnegative matrix factorization (NMF) problems. The proposed method exploits the block structure of NMF, utilizes the smooth adaptable property of the loss function based on the Bregman distance, and at the same time follows the iterative regularization idea of the linearized Bregman iterations method. Theoretically, we show that the proposed method is a descent method by adjusting the involved parameters. Finally, we end with several illustrative numerical experiments.

Graph regularized non-negative matrix factorization with L_{2,1} norm regularization terms for drug–target interactions prediction

BackgroundIdentifying drug–target interactions (DTIs) plays a key role in drug development. Traditional wet experiments to identify DTIs are costly and time consuming. Effective computational methods to predict DTIs are useful to speed up the process of drug discovery. A variety of non-negativity matrix factorization based methods are proposed to predict DTIs, but most of them overlooked the sparsity of feature matrices and the convergence of adopted matrix factorization algorithms, therefore their performances can be further improved.ResultsIn order to predict DTIs more accurately, we propose a novel method iPALM-DLMF. iPALM-DLMF models DTIs prediction as a problem of non-negative matrix factorization with graph dual regularization terms and L_{2,1} norm regularization terms. The graph dual regularization terms are used to integrate the information from the drug similarity matrix and the target similarity matrix, and L_{2,1} norm regularization terms are used to ensure the sparsity of the feature matrices obtained by non-negative matrix factorization. To solve the model, iPALM-DLMF adopts non-negative double singular value decomposition to initialize the nonnegative matrix factorization, and an inertial Proximal Alternating Linearized Minimization iterating process, which has been proved to converge to a KKT point, to obtain the final result of the matrix factorization. Extensive experimental results show that iPALM-DLMF has better performance than other state-of-the-art methods. In case studies, in 50 highest-scoring proteins targeted by the drug gabapentin predicted by iPALM-DLMF, 46 have been validated, and in 50 highest-scoring drugs targeting prostaglandin-endoperoxide synthase 2 predicted by iPALM-DLMF, 47 have been validated.

Block-Active ADMM to Minimize NMF with Bregman Divergences.

Over the last ten years, there has been a significant interest in employing nonnegative matrix factorization (NMF) to reduce dimensionality to enable a more efficient clustering analysis in machine learning. This technique has been applied in various image processing applications within the fields of computer vision and sensor-based systems. Many algorithms exist to solve the NMF problem. Among these algorithms, the alternating direction method of multipliers (ADMM) and its variants are one of the most popular methods used in practice. In this paper, we propose a block-active ADMM method to minimize the NMF problem with general Bregman divergences. The subproblems in the ADMM are solved iteratively by a block-coordinate-descent-type (BCD-type) method. In particular, each block is chosen directly based on the stationary condition. As a result, we are able to use much fewer auxiliary variables and the proposed algorithm converges faster than the previously proposed algorithms. From the theoretical point of view, the proposed algorithm is proved to converge to a stationary point sublinearly. We also conduct a series of numerical experiments to demonstrate the superiority of the proposed algorithm.

Non-negative low-rank approximations for multi-dimensional arrays on statistical manifold

Although low-rank approximation of multi-dimensional arrays has been widely discussed in linear algebra, its statistical properties remain unclear. In this paper, we use information geometry to uncover a statistical picture of non-negative low-rank approximations. First, we treat each input array as a probability distribution using a log-linear model on a poset, where a structure of an input array is realized as a partial order. We then describe the low-rank condition of arrays as constraints on parameters of the model and formulate the low-rank approximation as a projection onto a subspace that satisfies such constraints, where parameters correspond to coordinate systems of a statistical manifold. Second, based on information-geometric analysis of low-rank approximation, we point out the unexpected relationship between the rank-1 non-negative low-rank approximation and mean-field approximation, a well-established method in physics that uses a one-body problem to approximate a many-body problem. Third, our theoretical discussion leads to a novel optimization method of non-negative low-rank approximation, called Legendre Tucker rank reduction. Because the proposed method does not use the gradient method, it does not require tuning parameters such as initial position, learning rate, and stopping criteria. In addition, the flexibility of the log-linear model enables us to treat the problem of non-negative multiple matrix factorization (NMMF), a variant of low-rank approximation with shared factors. We find the best rank-1 NMMF formula as a closed form and develop a rapid rank-1 NMF method for arrays with missing entries based on the closed form, called A1GM.

DESCENT SPECTRAL VERSIONS OF THE TRADITIONAL CONJUGATE GRADIENT ALGORITHMS WITH APPLICATION TO NONNEGATIVE MATRIX FACTORIZATION

Despite computational superiorities, some traditional conjugate gradient algorithms such as Polak–Ribiére–Polyak and Hestenes–Stiefel methods generally fail to guarantee the descent condition. Here, in a matrix viewpoint, spectral versions of such methods are developed which fulfill the descent condition. The convergence of the given spectral algorithms is argued briefly. Afterwards, we propose an improved version of the nonnegative matrix factorization problem by adding penalty terms to the model, for controlling the condition number of one of the factorization elements. Finally, the computational merits of the method are examined using a set of CUTEr test problems as well as some random nonnegative matrix factorization models. The results typically agree with our analytical spectrum.

A non-convex optimization framework for large-scale low-rank matrix factorization

Low-rank matrix factorization problems such as non negative matrix factorization (NMF) can be categorized as a clustering or dimension reduction technique. The latter denotes techniques designed to find representations of some high dimensional dataset in a lower dimensional manifold without a significant loss of information. If such a representation exists, the features ought to contain the most relevant features of the dataset. Many linear dimensionality reduction techniques can be formulated as a matrix factorization. In this paper, we combine the conjugate gradient (CG) method with the Barzilai and Borwein (BB) gradient method, and propose a BB scaling CG method for NMF problems. The new method does not require to compute and store matrices associated with Hessian of the objective functions. Moreover, adopting a suitable BB step size along with a proper nonmonotone strategy which comes by the size of convex parameter ηk, results in a new algorithm that can significantly improve the CPU time, efficiency, the number of function evaluation. Convergence result is established and numerical comparisons of methods on both synthetic and real-world datasets show that the proposed method is efficient in comparison with existing methods and demonstrate the superiority of our algorithms.

Nonnegative matrix factorization with combined kernels for small data representation

Kernel nonnegative matrix factorization (KNMF) has emerged as a promising nonlinear data representation method, especially for applications with small sample sizes. Existing methods are usually based on a single kernel function, representing samples by the global or local features learned by the matrix factorization algorithm. In this paper, a combined kernel method is proposed and applied to data representation for small-sample face recognition in particular. Based on the combined kernel, which is a linear combination of the fractional power inner-product kernel and a newly defined Gaussian-type kernel, the new KNMF method is able to extract both global and local nonlinear features from the inputs. An efficient gradient decent algorithm is derived to solve the combined kernel nonnegative matrix factorization (CKNMF) problem, and a rigorous convergence proof is presented. The proposed method is experimentally evaluated on small image datasets, and the results demonstrate its superior performance than the state-of-the-art KNMF methods.

Refinement of Hottopixx Method for Nonnegative Matrix Factorization Under Noisy Separability

Hottopixx, proposed by Bittorf et al. at NIPS 2012, is an algorithm for solving nonnegative matrix factorization (NMF) problems under the separability assumption. Separable NMFs have important applications, such as topic extraction from documents and unmixing of hyperspectral images. In such applications, the robustness of the algorithm to noise is the key to success. Hottopixx has been shown to be robust to noise, and its robustness can be further enhanced through postprocessing. However, there is a drawback. Hottopixx and its postprocessing require us to estimate the noise level involved in the matrix we want to factorize before running, since they use it as part of the input data. The noise-level estimation is not an easy task. In this paper, we overcome this drawback. We present a refinement of Hottopixx and its postprocessing that runs without prior knowledge of the noise level. We show that the refinement has almost the same robustness to noise as the original algorithm.

An exact penalty approach for optimization with nonnegative orthogonality constraints

Optimization with nonnegative orthogonality constraints has wide applications in machine learning and data sciences. It is NP-hard due to some combinatorial properties of the constraints. We first propose an equivalent optimization formulation with nonnegative and multiple spherical constraints and an additional single nonlinear constraint. Various constraint qualifications, the first- and second-order optimality conditions of the equivalent formulation are discussed. By establishing a local error bound of the feasible set, we design a class of (smooth) exact penalty models via keeping the nonnegative and multiple spherical constraints. The penalty models are exact if the penalty parameter is sufficiently large but finite. A practical penalty algorithm with postprocessing is then developed to approximately solve a series of subproblems with nonnegative and multiple spherical constraints. We study the asymptotic convergence and establish that any limit point is a weakly stationary point of the original problem and becomes a stationary point under some additional mild conditions. Extensive numerical results on the problem of computing the orthogonal projection onto nonnegative orthogonality constraints, the orthogonal nonnegative matrix factorization problems and the K-indicators model show the effectiveness of our proposed approach.

Orthogonal nonnegative matrix factorization problems for clustering: A new formulation and a competitive algorithm

Orthogonal nonnegative matrix factorization problems for clustering: A new formulation and a competitive algorithm

A Novel Enhanced Nonnegative Matrix Factorization Method for Face Recognition

Nonnegative matrix factorization (NMF), distinguished from the approaches for holistic feature representation, is able to acquire meaningful basis images for parts-based representation. However, NMF does not utilize the data-label information and usually achieves undesirable performance in classification. To address the above-mentioned problem of NMF, this paper proposes a new enhanced NMF (ENMF) method for facial image representation and recognition. We seek to learn powerfully discriminative feature by a label-based regularizer which describes the relationship between the data. It is desired that minimizing the regularizer makes the data from the same class have high similarity and the data from different classes have low similarity. This good property will contribute to improving the performance of NMF. Therefore, we propose an objective function of ENMF by incorporating the label-based regularizer into the loss function. Subsequently, we find the stationary point of the constructed auxiliary function by means of Cardano’s formula and derive the update rules of our ENMF algorithm. The convergence of the proposed ENMF is both theoretically sound and empirically validated. Finally, the proposed ENMF method is successfully applied to face recognition. Four publicly available face datasets, namely AR, Caltech 101, Yale, and CMU PIE, are chosen for evaluations. Compared with the state-of-the-art NMF-based algorithms, the experimental results illustrate that the proposed ENMF algorithm achieves superior performance.

Exact solutions in low-rank approximation with zeros

Low-rank approximation with zeros aims to find a matrix of fixed rank and with a fixed zero pattern that minimizes the Euclidean distance to a given data matrix. We study the critical points of this optimization problem using algebraic tools. In particular, we describe special linear, affine, and determinantal relations satisfied by the critical points. We also investigate the number of critical points and how this number is related to the complexity of nonnegative matrix factorization problem.

Block Bregman Majorization Minimization with Extrapolation

In this paper, we consider a class of nonsmooth nonconvex optimization problems whose objective is the sum of a block relative smooth function and a proper and lower semicontinuous block separable function. Although the analysis of block proximal gradient (BPG) methods for the class of block $L$-smooth functions have been successfully extended to Bregman BPG methods that deal with the class of block relative smooth functions, accelerated Bregman BPG methods are scarce and challenging to design. Taking our inspiration from Nesterov-type acceleration and the majorization-minimization scheme, we propose a block alternating Bregman Majorization-Minimization framework with Extrapolation (BMME). We prove subsequential convergence of BMME to a first-order stationary point under mild assumptions, and study its global convergence under stronger conditions. We illustrate the effectiveness of BMME on the penalized orthogonal nonnegative matrix factorization problem.

Non-negative Matrix Factorization for Dimensionality Reduction

Abstract—What matrix factorization methods do is reduce the dimensionality of the data without losing any important information. In this work, we present the Non-negative Matrix Factorization (NMF) method, focusing on its advantages concerning other methods of matrix factorization. We discuss the main optimization algorithms, used to solve the NMF problem, and their convergence. The paper also contains a comparative study between principal component analysis (PCA), independent component analysis (ICA), and NMF for dimensionality reduction using a face image database. Index Terms—NMF, PCA, ICA, dimensionality reduction.

Quantum approximate optimization for hard problems in linear algebra

The quantum approximate optimization algorithm (QAOA) by Farhi et al. is a quantum computational framework for solving quantum or classical optimization tasks. Here, we explore using QAOA for binary linear least squares (BLLS); a problem that can serve as a building block of several other hard problems in linear algebra, such as the non-negative binary matrix factorization (NBMF) and other variants of the non-negative matrix factorization (NMF) problem. Most of the previous efforts in quantum computing for solving these problems were done using the quantum annealing paradigm. For the scope of this work, our experiments were done on noiseless quantum simulators, a simulator including a device-realistic noise-model, and two IBM Q 5-qubit machines. We highlight the possibilities of using QAOA and QAOA-like variational algorithms for solving such problems, where trial solutions can be obtained directly as samples, rather than being amplitude-encoded in the quantum wavefunction. Our numerics show that even for a small number of steps, simulated annealing can outperform QAOA for BLLS at a QAOA depth of p\leq3p≤3 for the probability of sampling the ground state. Finally, we point out some of the challenges involved in current-day experimental implementations of this technique on cloud-based quantum computers.

A projective approach to nonnegative matrix factorization

In data science and machine learning, the method of nonnegative matrix factorization (NMF) is a powerful tool that enjoys great popularity. Depending on the concrete application, there exist several subclasses each of which performs a NMF under certain constraints. Consider a given square matrix $A$. The symmetric NMF aims for a nonnegative low-rank approximation $A\approx XX^T$ to $A$, where $X$ is entrywise nonnegative and of given order. Considering a rectangular input matrix $A$, the general NMF again aims for a nonnegative low-rank approximation to $A$ which is now of the type $A\approx XY$ for entrywise nonnegative matrices $X,Y$ of given order. In this paper, we introduce a new heuristic method to tackle the exact nonnegative matrix factorization problem (of type $A=XY$), based on projection approaches to solve a certain feasibility problem.

Factor-Bounded Nonnegative Matrix Factorization

Nonnegative Matrix Factorization (NMF) is broadly used to determine class membership in a variety of clustering applications. From movie recommendations and image clustering to visual feature extractions, NMF has applications to solve a large number of knowledge discovery and data mining problems. Traditional optimization methods, such as the Multiplicative Updating Algorithm (MUA), solves the NMF problem by utilizing an auxiliary function to ensure that the objective monotonically decreases. Although the objective in MUA converges, there exists no proof to show that the learned matrix factors converge as well. Without this rigorous analysis, the clustering performance and stability of the NMF algorithms cannot be guaranteed. To address this knowledge gap, in this article, we study the factor-bounded NMF problem and provide a solution algorithm with proven convergence by rigorous mathematical analysis, which ensures that both the objective and matrix factors converge. In addition, we show the relationship between MUA and our solution followed by an analysis of the convergence of MUA. Experiments on both toy data and real-world datasets validate the correctness of our proposed method and its utility as an effective clustering algorithm.

Characterizing the Loss Landscape in Non-Negative Matrix Factorization

Non-negative matrix factorization (NMF) is a highly celebrated algorithm for matrix decomposition that guarantees non-negative factors. The underlying optimization problem is computationally intractable, yet in practice, gradient-descent-based methods often find good solutions. In this paper, we revisit the NMF optimization problem and analyze its loss landscape in non-worst-case settings. It has recently been observed that gradients in deep networks tend to point towards the final minimizer throughout the optimization procedure. We show that a similar property holds (with high probability) for NMF, provably in a non-worst case model with a planted solution, and empirically across an extensive suite of real-world NMF problems. Our analysis predicts that this property becomes more likely with growing number of parameters, and experiments suggest that a similar trend might also hold for deep neural networks---turning increasing dataset sizes and model sizes into a blessing from an optimization perspective.