Non-negative Factorization Research Articles

The Nonnegative Rank of a Matrix: Hard Problems, Easy Solutions

Using elementary linear algebra, we develop a technique that leads to solutions of two widely known problems on nonnegative matrices. First, we give a short proof of the result by Vavasis stating that the nonnegative rank of a matrix is NP-hard to compute. This proof is essentially contained in the paper by Jiang and Ravikumar, who discussed this topic in different terms fifteen years before the work of Vavasis. Second, we present a solution of the Cohen--Rothblum problem on rational nonnegative factorizations, which was posed in 1993 and remained open until now.

Improve the Prediction of Student Performance with Hint's Assistance Based on an Efficient Non-Negative Factorization

Improve the Prediction of Student Performance with Hint's Assistance Based on an Efficient Non-Negative Factorization

A Dynamical System Approach for Continuous Nonnegative Matrix Factorization

Nonnegative matrix factorization is a linear dimensionality reduction technique used for decomposing high-dimensional nonnegative data matrices for extracting basic and latent features. This technique plays fundamental roles in music analysis, signal processing, sound separation, and spectral data analysis. Given a time-varying objective function or a nonnegative time-dependent data matrix Y(t), the nonnegative factors of Y(t) can be obtained by taking the limit points of the trajectories of the corresponding ordinary differential equations. When the data are time dependent, it is natural to devise factorization techniques that capture the time dependency. To achieve this, one needs to solve continuous-time dynamical systems derived from iterative optimization schemes and construct nonnegative matrix factorization algorithms based on the solution curves. This article presents continuous nonnegative matrix factorization methods based on the solution of systems of ordinary differential equations associated with time-dependent data. In particular, we propose two new continuous-time algorithms based on the Kullback–Leibler divergence and the Amari \(\alpha \)-divergence.

A Model Selection Approach for Clustering a Multinomial Sequence with Non-Negative Factorization.

We consider a problem of clustering a sequence of multinomial observations by way of a model selection criterion. We propose a form of a penalty term for the model selection procedure. Our approach subsumes both the conventional AIC and BIC criteria but also extends the conventional criteria in a way that it can be applicable also to a sequence of sparse multinomial observations, where even within a same cluster, the number of multinomial trials may be different for different observations. In addition, as a preliminary estimation step to maximum likelihood estimation, and more generally, to maximum Lq estimation, we propose to use reduced rank projection in combination with non-negative factorization. We motivate our approach by showing that our model selection criterion and preliminary estimation step yield consistent estimates under simplifying assumptions. We also illustrate our approach through numerical experiments using real and simulated data.

Two Stage Approaches for the Detection and Suppression of Typed Keystrokes in Speech Signals

In recent decades, keystroke suppression has got a particular attention due to the increasing use of laptops and computers to capture audio in various communication scenarios such as meetings, audio/video instant messaging etc. In many of these situations, a unique problem of additive keystroke transient noise is faced. Because of the non-stationary, short time and abrupt nature of the keystroke transient, it has been a challenging task for many years. In this paper, two new twostage approaches for the suppression of keystrokes are proposed. In the first stage the speech is estimated using supervised sparse non-negative factorization, which is common in both of the methods. Then, in the second stage, keystrokes are detected and are suppressed by replacing the corrupted speech frames with the corresponding estimated speech frames obtained in the first stage using two new techniques, which is the core contribution of this work. Experimental results show that the proposed approaches exhibit good performance without significantly degrading the quality of speech.

A non-negative matrix factorization model based on the zero-inflated Tweedie distribution

Non-negative matrix factorization (NMF) is a technique of multivariate analysis used to approximate a given matrix containing non-negative data using two non-negative factor matrices that has been applied to a number of fields. However, when a matrix containing non-negative data has many zeroes, NMF encounters an approximation difficulty. This zero-inflated situation occurs often when a data matrix is given as count data, and becomes more challenging with matrices of increasing size. To solve this problem, we propose a new NMF model for zero-inflated non-negative matrices. Our model is based on the zero-inflated Tweedie distribution. The Tweedie distribution is a generalization of the normal, the Poisson, and the gamma distributions, and differs from each of the other distributions in the degree of robustness of its estimated parameters. In this paper, we show through numerical examples that the proposed model is superior to the basic NMF model in terms of approximation of zero-inflated data. Furthermore, we show the differences between the estimated basis vectors found using the basic and the proposed NMF models for $$\beta $$β divergence by applying it to real purchasing data.

On generalized Borgen plots II: The line‐moving algorithm and its numerical implementation

Borgen plots are geometric constructions that represent the set of all nonnegative factorizations of spectral data matrices for three‐component systems. The classical construction by Borgen and Kowalski (Anal. Chim. Acta 174, 1‐26 (1985)) is limited to nonnegative data and results in nonnegative factorizations. The new approach of generalized Borgen plots allows factors with small negative entries. This makes it possible to construct Borgen plots for perturbed or noisy spectral data and stabilizes the computation. In the first part of this paper, the mathematical theory of generalized Borgen plots has been introduced. This second part presents the line‐moving algorithm for the construction of generalized Borgen plots. The algorithm is justified, and the implementation in the FACPACK software is validated.

Orthogonal Nonnegative Matrix Factorization Combining Multiple Features for Spectral–Spatial Dimensionality Reduction of Hyperspectral Imagery

Nonnegative matrix factorization (NMF), which can lead to nonsubtractive parts-based representation, has been demonstrated to be effective for dimensionality reduction of hyperspectral imagery (HSI). However, existing NMF methods applied to HSI use only a single spectral feature and do not take into consideration spatial information, such as texture or morphological features, while it has been widely acknowledged that exploiting multiple features can improve performance. Consequently, a variant of orthogonal NMF, which can not only achieve a nonnegative factorization but also exploit the complementary information that arises among heterogeneous features, is proposed for hyperspectral dimensionality reduction. The proposed method, which couples orthogonal NMF with a previous multiple-features-combining algorithm, yields a discriminative low-dimensional feature representation that matches the intuition that parts should sum to produce a whole. An efficient multiplicative updating procedure is derived, and its local convergence is guaranteed theoretically. Experimental results on two hyperspectral data sets demonstrate the effectiveness of the proposed method.

Projective robust nonnegative factorization

Nonnegative matrix factorization (NMF) has been successfully used in many fields as a low-dimensional representation method. Projective nonnegative matrix factorization (PNMF) is a variant of NMF that was proposed to learn a subspace for feature extraction. However, both original NMF and PNMF are sensitive to noise and are unsuitable for feature extraction if data is grossly corrupted. In order to improve the robustness of NMF, a framework named Projective Robust Nonnegative Factorization (PRNF) is proposed in this paper for robust image feature extraction and classification. Since learned projections can weaken noise disturbances, PRNF is more suitable for classification and feature extraction. In order to preserve the geometrical structure of original data, PRNF introduces a graph regularization term which encodes geometrical structure. In the PRNF framework, three algorithms are proposed that add a sparsity constraint on the noise matrix based on L1/2 norm, L1 norm, and L2, 1 norm, respectively. Robustness and classification performance of the three proposed algorithms are verified with experiments on four face image databases and results are compared with state-of-the-art robust NMF-based algorithms. Experimental results demonstrate the robustness and effectiveness of the algorithms for image classification and feature extraction.

Efficient Nonnegative Matrix Factorization by DC Programming and DCA.

In this letter, we consider the nonnegative matrix factorization (NMF) problem and several NMF variants. Two approaches based on DC (difference of convex functions) programming and DCA (DC algorithm) are developed. The first approach follows the alternating framework that requires solving, at each iteration, two nonnegativity-constrained least squares subproblems for which DCA-based schemes are investigated. The convergence property of the proposed algorithm is carefully studied. We show that with suitable DC decompositions, our algorithm generates most of the standard methods for the NMF problem. The second approach directly applies DCA on the whole NMF problem. Two algorithms-one computing all variables and one deploying a variable selection strategy-are proposed. The proposed methods are then adapted to solve various NMF variants, including the nonnegative factorization, the smooth regularization NMF, the sparse regularization NMF, the multilayer NMF, the convex/convex-hull NMF, and the symmetric NMF. We also show that our algorithms include several existing methods for these NMF variants as special versions. The efficiency of the proposed approaches is empirically demonstrated on both real-world and synthetic data sets. It turns out that our algorithms compete favorably with five state-of-the-art alternating nonnegative least squares algorithms.

Incremental Stochastic Factorization for Online Reinforcement Learning

A construct that has been receiving attention recently in reinforcement learning is stochastic factorization (SF), a particular case of non-negative factorization (NMF) in which the matrices involved are stochastic. The idea is to use SF to approximate the transition matrices of a Markov decision process (MDP). This is useful for two reasons. First, learning the factors of the SF instead of the transition matrices can reduce significantly the number of parameters to be estimated. Second, it has been shown that SF can be used to reduce the number of operations needed to compute an MDP's value function. Recently, an algorithm called expectation-maximization SF (EMSF) has been proposed to compute a SF directly from transitions sampled from an MDP. In this paper we take a closer look at EMSF. First, by exploiting the assumptions underlying the algorithm, we show that it is possible to reduce it to simple multiplicative update rules similar to the ones that helped popularize NMF. Second, we analyze the optimization process underlying EMSF and find that it minimizes a modified version of the Kullback-Leibler divergence that is particularly well-suited for learning a SF from data sampled from an arbitrary distribution. Third, we build on this improved understanding of EMSF to draw an interesting connection with NMF and probabilistic latent semantic analysis. We also exploit the simplified update rules to introduce a new version of EMSF that generalizes and significantly improves its precursor. This new algorithm provides a practical mechanism to control the trade-off between memory usage and computing time, essentially freeing the space complexity of EMSF from its dependency on the number of sample transitions. The algorithm can also compute its approximation incrementally, which makes it possible to use it concomitantly with the collection of data. This feature makes the new version of EMSF particularly suitable for online reinforcement learning. Empirical results support the utility of the proposed algorithm.

Analysis on a Nonnegative Matrix Factorization and Its Applications

In this work we perform some mathematical analysis on a special nonnegative matrix trifactorization (NMF) and apply this NMF to some imaging and inverse problems. We will propose a sparse low-rank approximation of positive data and images in terms of tensor products of positive vectors and investigate its effectiveness in terms of the number of tensor products to be used in the approximation. A new multilevel analysis (MLA) framework is suggested to extract major components in the matrix representing structures of different resolutions but still preserve the positivity of the basis and sparsity of the approximation. We will also propose and formulate a semismooth Newton method based on primal-dual active sets for the nonnegative factorization. Numerical results are given to demonstrate the effectiveness of the proposed method at capturing features in images and structures of inverse problems under no a priori assumption on the underlying structure in the data as well as to provide a sparse low-rank representation of the data.

Data visualization via latent variables and mixture models: a brief survey

In the literature, data visualization is extensively studied via diverse parametric probabilistic distributions for the exploration of continuous, binary, and counting data. An overview of the existing methods for non-symmetric data matrices is presented in an unified framework via the Bernoulli law and binary variables. An extension to continuous or counting variables is available by using instead any another univariate distribution such as the Poisson or Gaussian one. Several approaches are possible when the model is with a distribution on the rows, the columns, the row clusters, the column clusters, the cells, the blocks, or a transformed matrix of the distances from the pairs of rows or columns. The objective functions are presented with their full expressions in separated sections, one for each method: Kohonen's map and related methods of self-organizing maps, generative topographic mapping as a probabilistic self-organizing map, linear principal component analysis and related matricial methods (non-negative factorization, factorization), probabilistic parametric embedding, probabilistic latent semantic visualization, latent cluster position model, t-distributed stochastic neighbor embedding. The conclusion is a discussion of the contribution with perspectives.

Set Identification and Estimation of Factor and Topic Models

The paper presents sharp bounds on the identified set for classical factor models and non-parametric topic models based on results from the non-negative factorization literature. It compares the standard assumption (for factor models) of orthonormality of the factors (principal components analysis) to the natural assumption of topic models of additivity and non-negativity. For the former, the model is point identified when the number of factors is small but further restrictions such as those presented in Bai and Ng (2013) are needed to identify larger models. Under the latter, the paper characterizes the identified set and shows the necessary condition for point identification presented in Huang et al (2013) is also sufficient. In the two factor case this condition states that for each latent factor there must be some asset whose return gives it zero weight and there must be some time periods where each factor's normalized return is zero. These sparsity conditions are characteristics of the observed data, not assumptions on the data generating process. The paper presents a least squares estimator where the number of parameters to be estimated is not increasing in the size of the data set. The paper shows that this estimator is consistent both when the number time periods increases in the factor model and when the number of documents increases in the topic model. Unlike the similar estimator presented in the classical factor model literature (Stock and Watson (2002), Bai (2003)) this estimator does not rely on orthonormality.

A predictive model of muscle excitations based on muscle modularity for a large repertoire of human locomotion conditions

Humans can efficiently walk across a large variety of terrains and locomotion conditions with little or no mental effort. It has been hypothesized that the nervous system simplifies neuromuscular control by using muscle synergies, thus organizing multi-muscle activity into a small number of coordinative co-activation modules. In the present study we investigated how muscle modularity is structured across a large repertoire of locomotion conditions including five different speeds and five different ground elevations. For this we have used the non-negative matrix factorization technique in order to explain EMG experimental data with a low-dimensional set of four motor components. In this context each motor components is composed of a non-negative factor and the associated muscle weightings. Furthermore, we have investigated if the proposed descriptive analysis of muscle modularity could be translated into a predictive model that could: (1) Estimate how motor components modulate across locomotion speeds and ground elevations. This implies not only estimating the non-negative factors temporal characteristics, but also the associated muscle weighting variations. (2) Estimate how the resulting muscle excitations modulate across novel locomotion conditions and subjects. The results showed three major distinctive features of muscle modularity: (1) the number of motor components was preserved across all locomotion conditions, (2) the non-negative factors were consistent in shape and timing across all locomotion conditions, and (3) the muscle weightings were modulated as distinctive functions of locomotion speed and ground elevation. Results also showed that the developed predictive model was able to reproduce well the muscle modularity of un-modeled data, i.e., novel subjects and conditions. Muscle weightings were reconstructed with a cross-correlation factor greater than 70% and a root mean square error less than 0.10. Furthermore, the generated muscle excitations matched well the experimental excitation with a cross-correlation factor greater than 85% and a root mean square error less than 0.09. The ability of synthetizing the neuromuscular mechanisms underlying human locomotion across a variety of locomotion conditions will enable solutions in the field of neurorehabilitation technologies and control of bipedal artificial systems. Open-access of the model implementation is provided for further analysis at https://simtk.org/home/p-mep/.

Heuristics for exact nonnegative matrix factorization

The exact nonnegative matrix factorization (exact NMF) problem is the following: given an $m$-by-$n$ nonnegative matrix $X$ and a factorization rank $r$, find, if possible, an $m$-by-$r$ nonnegative matrix $W$ and an $r$-by-$n$ nonnegative matrix $H$ such that $X = WH$. In this paper, we propose two heuristics for exact NMF, one inspired from simulated annealing and the other from the greedy randomized adaptive search procedure. We show that these two heuristics are able to compute exact nonnegative factorizations for several classes of nonnegative matrices (namely, linear Euclidean distance matrices, slack matrices, unique-disjointness matrices, and randomly generated matrices) and as such demonstrate their superiority over standard multi-start strategies. We also consider a hybridization between these two heuristics that allows us to combine the advantages of both methods. Finally, we discuss the use of these heuristics to gain insight on the behavior of the nonnegative rank, i.e., the minimum factorization rank such that an exact NMF exists. In particular, we disprove a conjecture on the nonnegative rank of a Kronecker product, propose a new upper bound on the extension complexity of generic $n$-gons and conjecture the exact value of (i) the extension complexity of regular $n$-gons and (ii) the nonnegative rank of a submatrix of the slack matrix of the correlation polytope.

Studying Non-negative Factorizations with Tools from Linear Algebra over a Semiring

We develop the technique useful for studying the problem of factoring nonnegative matrices. We illustrate our method, based on the tools from linear algebra over a semiring, by applying it to studying the problem of existence of a rank-three matrix with full nonnegative rank equal to n.

Analysis of trends in seasonal electrical energy consumption via non-negative tensor factorization

This paper looks at the extraction of trends of household electrical seasonal consumption via load disaggregation. With the proviso that data for several home devices can be embedded in a tensor, non-negative multi-way array factorization is performed in order to extract the most relevant components. In the initial decomposition step the decomposed signals are incorporated in the test signal consisting of the whole-home measured consumption. After this the disaggregated data corresponding to each electrical device is obtained by factorizing the associated matrix through the learned model. Finally, we evaluate the performance of load disaggregation by the supervised method, and study the trends along several years and across seasons. Towards this end, computational experiments were yielded using real-world data from household electrical consumption measurements along several years. While breaking down the whole house energy consumption into appliance level gives less accurate estimates in the late years, we empirically show the adequacy of this method for handling the earlier years and the estimates of the underlying seasonal trend-cycle.

Characterization of Five Shu Acupoint Pattern in Saam Acupuncture Using Text Mininig

Background : Saam acupuncture were composed by applying the elemental concepts from the Five Phase theory - the relationships between the cycles such as Saeng(Sheng, ‘nourishing’ or ‘creating’) and Geuk(Ke, ‘suppressing’ or ‘controlling’) - onto the Five Phase points and 12 channels to compensate for the imbalance in each of the 12 main energy traits. Objective : The present study is aimed to find out the characteristics of Five Phase points pattern in Saam acupuncture. Methods : We analysed the characteristics of five elements of the Five Phase points in Korean medical texts such as Saamdoinchimguyogyeol, Dongeuibogam and Chimgugyeongheombang in mid Chosun Dynasty. Using non-negative factorization(NNMF) methods, we extracted the feature matrix of five elements of Five Phase points in each classic medical text. Results : In Saam acupuncture, two characteristics were most prominent: (1) Self component of Five elements, (2) Mother and Grandmother component of Five elements. Conclusions : Saam acupuncture used the combination of Five-Shu acupoint based on ZangFu pattern identification. Our findings suggest that grasping the characteristics of Five Phase points combinations can improve the understanding the selection of the relevant acupoints based on the ZangFu pattern identifications.

On generalized Borgen plots. I: From convex to affine combinations and applications to spectral dataSpectra

In 1985, Borgen and Kowalski (Anal. Chim. Acta 1985; 174: 1‐26) published their landmark paper on the geometric construction of feasible regions for nonnegative factorizations of spectral data matrices for three‐component systems. These geometric constructions are called Borgen plots. Borgen plots are principally restricted to nonnegative data and are sometimes considered as analytical tool. Major contributions to this theory have been given by Rajkó. In contrast to these geometric constructions, numerical methods to compute the so‐called area of feasible solutions (AFS) have been studied by Golshan et al. (Anal. Chem. 2011; 83 (3): 836‐841) and by Sawall et al. (J. Chemom. 2013; 27: 106‐116). These numerical methods can even treat spectral data, which include slightly negative components.In this work, the concept of generalized Borgen plots is introduced for spectral data, which are polluted by small negative entries. The analysis is not restricted to three‐component systems but can be applied to general s‐component systems. Generalized Borgen plots are identical to the classical Borgen plots for nonnegative data. The analysis in this work also bridges the gap between the different scalings (Borgen norms) used for AFS computations.The algorithmic procedure of generalized Borgen plots for three‐component systems and its implementation in the FAC‐PACK software are described in the second part of this paper. Copyright © 2015 John Wiley & Sons, Ltd.