Multiplicative Update Rules Research Articles

Blind Audio Source Separation with Sparse Nonnegative Matrix Factorization

In this study, a new technique in source separation using Two-Dimensional Nonnegative Matrix Factorization (NMF2D) with the Beta-divergence is proposed. The Time-Frequency (TF) profile of each source is modeled as two-dimensional convolution of the temporal code and the spectral basis. In addition, adaptive sparsity constraint was imposed to reduce the ambiguity and provide uniqueness to the solution. The proposed model used Beta-divergence as a cost function and updated by maximizing the joint probability of the mixing spectral basis and temporal codes using the multiplicative update rules. Experimental tests have been conducted in audio application to blindly separate the source in musical mixture. Results have shown the effectiveness of the algorithm in separating the audio sources from single channel mixture.

Underdetermined Blind Sources Separation Based on Nonnegative Tri-Matrix Factorization

In this paper, a nonnegative tri-Matrix factorization (NTMF) algorithm is proposed for underdetermined blind sources separation with the assumption of that the source signals are nonnegative and sparse. By incorporating the regularization and sparse penalty into the cost function, a novel multiplicative update rules is proposed to solve the problem of UBSS based on NTMF. The simulation results are presented to show the validity and competitive performances of the proposed algorithm.

A Bayesian approach to the Lee–Seung update rules for NMF

NMF is a Blind Source Separation technique decomposing multivariate non-negative data sets into meaningful non-negative basis components and non-negative weights. In its canonical form an NMF algorithm was proposed by Lee and Seung (1999) [31] employing multiplicative update rules. In this study we show how the latter follow from a new variational Bayes NMF algorithm VBNMF employing a Gaussian noise kernel.

Adaptive multiplicative updates for quadratic nonnegative matrix factorization

In Nonnegative Matrix Factorization (NMF), a nonnegative matrix is approximated by a product of lower-rank factorizing matrices. Quadratic Nonnegative Matrix Factorization (QNMF) is a new class of NMF methods where some factorizing matrices occur twice in the approximation. QNMF finds its applications in graph partition, bi-clustering, graph matching, etc. However, the original QNMF algorithms employ constant multiplicative update rules and thus have mediocre convergence speed. Here we propose an adaptive multiplicative algorithm for QNMF which is not only theoretically convergent but also significantly faster than the original implementation. An adaptive exponent scheme has been adopted for our method instead of the old constant ones, which enables larger learning steps for improved efficiency. The proposed method is general and thus can be applied to QNMF with a variety of factorization forms and with the most commonly used approximation error measures. We have performed extensive experiments, where the results demonstrate that the new method is effective in various QNMF applications on both synthetic and real-world datasets.

A New Face Recognition Framework Based on Improved Nonnegative Matrix Factorization

A new face recognition framework based on improved Nonnegative Matrix Factorization (NMF) is proposed in this paper. The improved NMF algorithm adds a sparse constraint which makes the partbased features of images more representative. We develop a novel multiplicative update rule for the algorithm and prove its convergence by using an auxiliary function method. The framework of the face recognition system has three main steps. Firstly, apply the two-level Haar wavelet decomposition to transform images to a low-dimensionality space. Secondly, use the improved NMF algorithm for feature selection. Finally, adopt Support Vector Machine (SVM) for classification. Experiments contrasted with traditional algorithms demonstrate that the proposed method has high classification accuracy with high processing speed. Experimental results also express that the dimensionality of feature subspace is able to affect face recognition accuracy.

Discriminant Projective Non-Negative Matrix Factorization

Projective non-negative matrix factorization (PNMF) projects high-dimensional non-negative examples X onto a lower-dimensional subspace spanned by a non-negative basis W and considers WT X as their coefficients, i.e., X≈WWT X. Since PNMF learns the natural parts-based representation Wof X, it has been widely used in many fields such as pattern recognition and computer vision. However, PNMF does not perform well in classification tasks because it completely ignores the label information of the dataset. This paper proposes a Discriminant PNMF method (DPNMF) to overcome this deficiency. In particular, DPNMF exploits Fisher's criterion to PNMF for utilizing the label information. Similar to PNMF, DPNMF learns a single non-negative basis matrix and needs less computational burden than NMF. In contrast to PNMF, DPNMF maximizes the distance between centers of any two classes of examples meanwhile minimizes the distance between any two examples of the same class in the lower-dimensional subspace and thus has more discriminant power. We develop a multiplicative update rule to solve DPNMF and prove its convergence. Experimental results on four popular face image datasets confirm its effectiveness comparing with the representative NMF and PNMF algorithms.

Fusion of multi‐aspect radar images via sparse non‐negative matrix factorisation

By fusing the radar images of a certain target obtained from multiple aspects, complementary information can be made full use of for better target descriptions and higher image quality. A fusion scheme based on non-negative matrix factorisation (NMF) is proposed. A sparsity-enhancing regularisation term is introduced into the original NMF, and the corresponding modified multiplicative update rule is derived to iteratively fuse the images. The composite image generally demonstrates enhanced feature characteristics and improved signal-to-noise ratio. The experimental results prove the validity and superiority of the proposal.

Limited-Memory Fast Gradient Descent Method for Graph Regularized Nonnegative Matrix Factorization

Graph regularized nonnegative matrix factorization (GNMF) decomposes a nonnegative data matrix to the product of two lower-rank nonnegative factor matrices, i.e., and () and aims to preserve the local geometric structure of the dataset by minimizing squared Euclidean distance or Kullback-Leibler (KL) divergence between X and WH. The multiplicative update rule (MUR) is usually applied to optimize GNMF, but it suffers from the drawback of slow-convergence because it intrinsically advances one step along the rescaled negative gradient direction with a non-optimal step size. Recently, a multiple step-sizes fast gradient descent (MFGD) method has been proposed for optimizing NMF which accelerates MUR by searching the optimal step-size along the rescaled negative gradient direction with Newton's method. However, the computational cost of MFGD is high because 1) the high-dimensional Hessian matrix is dense and costs too much memory; and 2) the Hessian inverse operator and its multiplication with gradient cost too much time. To overcome these deficiencies of MFGD, we propose an efficient limited-memory FGD (L-FGD) method for optimizing GNMF. In particular, we apply the limited-memory BFGS (L-BFGS) method to directly approximate the multiplication of the inverse Hessian and the gradient for searching the optimal step size in MFGD. The preliminary results on real-world datasets show that L-FGD is more efficient than both MFGD and MUR. To evaluate the effectiveness of L-FGD, we validate its clustering performance for optimizing KL-divergence based GNMF on two popular face image datasets including ORL and PIE and two text corpora including Reuters and TDT2. The experimental results confirm the effectiveness of L-FGD by comparing it with the representative GNMF solvers.

Regularized Blind Deconvolution with Poisson Data

We propose easy-to-implement algorithms to perform blind deconvolution of nonnegative images in the presence of noise of Poisson type. Alternate minimization of a regularized Kullback-Leibler cost function is achieved via multiplicative update rules. The scheme allows to prove convergence of the iterates to a stationary point of the cost function. Numerical examples are reported to demonstrate the feasibility of the proposed method.

Non-Negative Temporal Decomposition of Speech Parameters by Multiplicative Update Rules

I invented a non-negative temporal decomposition method for line spectral pairs and articulatory parameters based on the multiplicative update rules. These parameters are decomposed into a set of temporally overlapped unimodal event functions restricted to the range [0,1] and corresponding event vectors. When line spectral pairs are used, event vectors preserve their ordering property. With the proposed method, the RMS error of the measured and reconstructed articulatory parameters is 0.21 mm and the spectral distance of the measured and reconstructed line spectral pairs parameters is 2.0 dB. The RMS error and spectral distance in the proposed method are smaller than those in conventional methods. This technique will be useful for many applications of speech coding and speech modification.

Unsupervised learning of phonemes of whispered speech in a noisy environment based on convolutive non-negative matrix factorization

This paper focuses on the development of an algorithm that can be optimized for a specific acoustic environment to improve the intelligibility of whispered speech. A new convolutive non-negative matrix factorization (NMF) algorithm is proposed to extract phoneme bases from noisy whispered speech with the noise bases from prior learning; these noise bases are obtained from training using the conventional non-negative matrix factorization. The divergence function with a sparseness constraint term is selected as the objective function in the developed algorithm to obtain multiplicative update rules of the phoneme base matrix and the corresponding weight matrix. The weights of the noise bases from prior learning are also updated in the phoneme learning stage. Listening experiments were conducted to assess the intelligibility performance of speech synthesized using the proposed algorithm. The experimental results indicate that the proposed algorithm is very effective for improving the intelligibility of whispers in various noise contexts, and it outperforms conventional algorithms.

Projection-free parallel quadratic programming for linear model predictive control

A key component in enabling the application of model predictive control (MPC) in fields such as automotive, aerospace, and factory automation is the availability of low-complexity fast optimisation algorithms to solve the MPC finite horizon optimal control problem in architectures with reduced computational capabilities. In this paper, we introduce a projection-free iterative optimisation algorithm and discuss its application to linear MPC. The algorithm, originally developed by Brand for non-negative quadratic programs, is based on a multiplicative update rule and it is shown to converge to a fixed point which is the optimum. An acceleration technique based on a projection-free line search is also introduced, to speed-up the convergence to the optimum. The algorithm is applied to MPC through the dual of the quadratic program (QP) formulated from the MPC finite time optimal control problem. We discuss how termination conditions with guaranteed degree of suboptimality can be enforced, and how the algorithm performance can be optimised by pre-computing the matrices in a parametric form. We show computational results of the algorithm in three common case studies and we compare such results with the results obtained by other available free and commercial QP solvers.

Multichannel Extensions of Non-Negative Matrix Factorization With Complex-Valued Data

This paper presents new formulations and algorithms for multichannel extensions of non-negative matrix factorization (NMF). The formulations employ Hermitian positive semidefinite matrices to represent a multichannel version of non-negative elements. Multichannel Euclidean distance and multichannel Itakura-Saito (IS) divergence are defined based on appropriate statistical models utilizing multivariate complex Gaussian distributions. To minimize this distance/divergence, efficient optimization algorithms in the form of multiplicative updates are derived by using properly designed auxiliary functions. Two methods are proposed for clustering NMF bases according to the estimated spatial property. Convolutive blind source separation (BSS) is performed by the multichannel extensions of NMF with the clustering mechanism. Experimental results show that 1) the derived multiplicative update rules exhibited good convergence behavior, and 2) BSS tasks for several music sources with two microphones and three instrumental parts were evaluated successfully.

Multiplicative Update Rules for Concurrent Nonnegative Matrix Factorization and Maximum Margin Classification

The state-of-the-art classification methods which employ nonnegative matrix factorization (NMF) employ two consecutive independent steps. The first one performs data transformation (dimensionality reduction) and the second one classifies the transformed data using classification methods, such as nearest neighbor/centroid or support vector machines (SVMs). In the following, we focus on using NMF factorization followed by SVM classification. Typically, the parameters of these two steps, e.g., the NMF bases/coefficients and the support vectors, are optimized independently, thus leading to suboptimal classification performance. In this paper, we merge these two steps into one by incorporating maximum margin classification constraints into the standard NMF optimization. The notion behind the proposed framework is to perform NMF, while ensuring that the margin between the projected data of the two classes is maximal. The concurrent NMF factorization and support vector optimization are performed through a set of multiplicative update rules. In the same context, the maximum margin classification constraints are imposed on the NMF problem with additional discriminant constraints and respective multiplicative update rules are extracted. The impact of the maximum margin classification constraints on the NMF factorization problem is addressed in Section VI. Experimental results in several databases indicate that the incorporation of the maximum margin classification constraints into the NMF and discriminant NMF objective functions improves the accuracy of the classification.

Enhancing Kernel Maximum Margin Projection for Face Recognition

To efficiently deal with the face recognition problem, a novel face recognition algorithm based on enhancing kernel maximum margin projection(MMP) is proposed in this paper. The main contributions of this work are as follows. First, the nonlinear extension of MMP through kernel trick is adopted to capture the nonlinear structure of face images. Second, the kernel deformation technique is proposed to increase the discriminating capability of original input kernel function. Third, the feature vector selection approach is applied to improve computational efficiency of kernel MMP. Finally, the multiplicative update rule is employed to enhance training speed of SVM classifier for face recognition. Experimental results on face recognition demonstrate the effectiveness and efficiency of the proposed algorithm.

Enhancing Spectral Unmixing by Local Neighborhood Weights

Spectral unmixing is an effective technique to remotely sensed data exploitation. In this paper, appropriate weights in a local neighborhood are designed to enhance spectral unmixing. The weights integrate the spectral and spatial information, and can effectively segment the homogenous and transition areas between different ground cover types. Based on this region-segmentation, pure-pixel-based end-member extraction algorithms are insensitive to the anomalous pixel, and thus perform more robust. In addition, the weights can be used to regularize non-pure-pixel-based unmixing methods, such as nonnegative matrix factorization (NMF). By incorporating the designed local neighborhood weights, a weighted nonnegative matrix factorization (WNMF) algorithm for spectral unmixing is proposed in this paper. Meanwhile, a multiplicative update rule for WNMF is presented, and the monotonic convergence of the rule is proved. Experiments on synthetic and real hyperspectral data validate the effectiveness of the designed weights.

On-line learning parts-based representation via incremental orthogonal projective non-negative matrix factorization

This paper presents a novel incremental orthogonal projective non-negative matrix factorization (IOPNMF) algorithm, which is aimed to learn a parts-based subspace that reveals dynamic data streams. By assuming that the newly added samples only affect basis vectors but do not affect the coefficients of old samples, we propose an objective function for on-line learning and then present a multiplicative update rule to solve it. Compared with other non-negative matrix factorization (NMF) methods, our algorithm can guarantee to learn a linear parts-based subspace in an on-line fashion, which may facilitate some real applications. The facial analysis experiment shows that our IOPNMF method learns parts-based components successfully. In addition, we present an effective tracking method by integrating the IOPNMF method, the idea of sparse representation and the domain information of object tracking. The proposed tracker explicitly takes partial occlusion and mis-alignment into account for appearance model update and object tracking. The experimental results on some challenging image sequences demonstrate the proposed tracking algorithm performs favorably against several state-of-the-art methods.

NeNMF: An Optimal Gradient Method for Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) is a powerful matrix decomposition technique that approximates a nonnegative matrix by the product of two low-rank nonnegative matrix factors. It has been widely applied to signal processing, computer vision, and data mining. Traditional NMF solvers include the multiplicative update rule (MUR), the projected gradient method (PG), the projected nonnegative least squares (PNLS), and the active set method (AS). However, they suffer from one or some of the following three problems: slow convergence rate, numerical instability and nonconvergence. In this paper, we present a new efficient NeNMF solver to simultaneously overcome the aforementioned problems. It applies Nesterov's optimal gradient method to alternatively optimize one factor with another fixed. In particular, at each iteration round, the matrix factor is updated by using the PG method performed on a smartly chosen search point, where the step size is determined by the Lipschitz constant. Since NeNMF does not use the time consuming line search and converges optimally at rate in optimizing each matrix factor, it is superior to MUR and PG in terms of efficiency as well as approximation accuracy. Compared to PNLS and AS that suffer from numerical instability problem in the worst case, NeNMF overcomes this deficiency. In addition, NeNMF can be used to solve -norm, -norm and manifold regularized NMF with the optimal convergence rate. Numerical experiments on both synthetic and real-world datasets show the efficiency of NeNMF for NMF and its variants comparing to representative NMF solvers. Extensive experiments on document clustering suggest the effectiveness of NeNMF.

Image label completion by pursuing contextual decomposability

This article investigates how to automatically complete the missing labels for the partially annotated images, without image segmentation. The label completion procedure is formulated as a nonnegative data factorization problem, to decompose the global image representations that are used for describing the entire images, for instance, various image feature descriptors, into their corresponding label representations, that are used for describing the local semantic regions within images. The solution provided in this work is motivated by following observations. First, label representations of the regions with the same label often share certain commonness, yet may be essentially different due to the large intraclass variations. Thus, each label or concept should be represented by using a subspace spanned by an ensemble of basis, instead of a single one, to characterize the intralabel diversities. Second, the subspaces for different labels are different from each other. Third, while two images are similar with each other, the corresponding label representations should be similar. We formulate this cross-image context as well as the given partial label annotations in the framework of nonnegative data factorization and then propose an efficient multiplicative nonnegative update rules to alternately optimize the subspaces and the reconstruction coefficients. We also provide the theoretic proof of algorithmic convergence and correctness. Extensive experiments over several challenging image datasets clearly demonstrate the effectiveness of our proposed solution in boosting the quality of image label completion and image annotation accuracy. Based on the same formulation, we further develop a label ranking algorithms, to refine the noised image labels without any manual supervision. We compare the proposed label ranking algorithm with the state-of-the-arts over the popular evaluation databases and achieve encouragingly improvements.

The Multiplicative Weights Update Method: A Meta-Algorithm and Applications

Algorithms in varied fields use the idea of maintaining a distribution over a certain set and use the multiplicative update rule to iteratively change these weights. Their analyses are usually very similar and rely on an exponential potential function. In this survey we present a simple meta-algorithm that unifies many of these disparate algorithms and derives them as simple instantiations of the meta-algorithm. We feel that since this meta-algorithm and its analysis are so simple, and its applications so broad, it should be a standard part of algorithms courses, like “divide and conquer.”