Multiplicative Update Rules Research Articles

Hyperspectral Unmixing Using Sparsity-Constrained Deep Nonnegative Matrix Factorization With Total Variation

Hyperspectral unmixing is an important processing step for many hyperspectral applications, mainly including: 1) estimation of pure spectral signatures (endmembers) and 2) estimation of the abundance of each endmember in each pixel of the image. In recent years, nonnegative matrix factorization (NMF) has been highly attractive for this purpose due to the nonnegativity constraint that is often imposed in the abundance estimation step. However, most of the existing NMF-based methods only consider the information in a single layer while neglecting the hierarchical features with hidden information. To alleviate such limitation, in this paper, we propose a new sparsity-constrained deep NMF with total variation (SDNMF-TV) technique for hyperspectral unmixing. First, by adopting the concept of deep learning, the NMF algorithm is extended to deep NMF model. The proposed model consists of pretraining stage and fine-tuning stage , where the former pretrains all factors layer by layer and the latter is used to reduce the total reconstruction error. Second, in order to exploit adequately the spectral and spatial information included in the original hyperspectral image, we enforce two constraints on the abundance matrix. Specifically, the $L_{1/2}$ constraint is adopted, since the distribution of each endmember is sparse in the 2-D space. The TV regularizer is further introduced to promote piecewise smoothness in abundance maps. For the optimization of the proposed model, multiplicative update rules are derived using the gradient descent method. The effectiveness and superiority of the SDNMF-TV algorithm are demonstrated by comparing with other unmixing methods on both synthetic and real data sets.

A novel cuckoo search algorithm with multiple update rules

In this paper, we propose a novel cuckoo search algorithm with multiple update rules, referred to as a hybrid CS algorithm (HCS). In the presented approach, to overcome the mutual interference among dimensions and enhance the local search capability, two different one-dimensional update rules are integrated into CS framework for acquiring the candidate solutions. Moreover, using the characteristic of occasionally long jumps in Levy distribution, the proper selection between the one-dimensional update rules and Levy flight random walk is achieved by setting a limit value, so as to further enhance the exploration ability. The performance of the presented algorithm is then extensively investigated on 49 benchmark test functions including 11 common instances, 10 instances introduced in CEC 2005, and 28 instances presented in CEC 2013. The experimental results indicate that HCS algorithm is better than other CS variants in terms of solution accuracy and robustness, and it also outperforms the seven state-of-the-art intelligent algorithms.

A swap sequence based Artificial Bee Colony algorithm for Traveling Salesman Problem

Abstract In this research paper, Artificial Bee Colony algorithm is modified with multiple update rules and K-opt operation to solve the Traveling Salesman Problem. Here the features of swap sequences and swap operations on the sequence of cities (solution/path) of the problem are used to create different solution (path) update rules of the algorithm. Eight different rules are proposed to update solutions in the algorithm. Updation of a solution by an employed bee or by an onlooker bee is done by a randomly selected rule from the rule set using Roulette Wheel selection process. In the scout bee phase of the algorithm, the perturbation technique, K-opt operation is applied on any stagnant solution for a fixed number of times for the possible improvement of it. The K-opt operation is again used at the end of the search process to improve the quality of the final solution (if possible). Proposed method is tested with a set of benchmark test problems from TSPLIB and it is observed that the efficiency of the algorithm is adequate with respect to the accuracy and the consistency for solving standard TSPs (Symmetric as well as Asymmetric) compared to the existing algorithms in the literature.

Hybrid Projective Nonnegative Matrix Factorization With Drum Dictionaries for Harmonic/Percussive Source Separation

One of the most general models of music signals considers that such signals can be represented as a sum of two distinct components: a tonal part that is sparse in frequency and temporally stable and a transient (or percussive) part that is composed of short-term broadband sounds. In this paper, we propose a novel hybrid method built upon nonnegative matrix factorization (NMF) that decomposes the time frequency representation of an audio signal into such two components. The tonal part is estimated by a sparse and orthogonal nonnegative decomposition, and the transient part is estimated by a straightforward NMF decomposition constrained by a pre-learned dictionary of smooth spectra. The optimization problem at the heart of our method remains simple with very few hyperparameters and can be solved thanks to simple multiplicative update rules. The extensive benchmark on a large and varied music database against four state of the art harmonic/percussive source separation algorithms demonstrate the merit of the proposed approach.

Accelerated image factorization based on improved NMF algorithm

Non-negative matrix factorization (NMF) is widely used in feature extraction and dimension reduction fields. Essentially, it is an optimization problem to determine two non-negative low rank matrices \(W_{m \times k}\) and \(H_{k \times n}\) for a given matrix \(A_{m \times n}\), satisfying \(A_{m \times n} \approx W_{m \times k}H_{k \times n}\). In this paper, a novel approach to improve the image decomposing and reconstruction effects by introducing the Singular Value Decomposing (SVD)-based initialization scheme of factor matrices W and H, and another measure called choosing rule to determine the optimum value of factor rank k, are proposed. The input image is first decomposed using SVD to get its singular values and corresponding eigenvectors. Then, the number of main components as the rank value k is extracted. Then, the singular values and corresponding eigenvectors are used to initialize W and H based on selected rank k. Finally, convergent results are obtained using multiplicative and additive update rules. However, iterative NMF algorithms’ convergence is very slow on most platforms limiting its practicality. To this end, a parallel implementation frame of described improved NMF algorithm using CUDA, a tool for algorithms parallelization on massively parallel processors, i.e., many-core graphics processors, is presented. Experimental results show that our approach can get better decomposing effect than traditional NMF implementations and dramatic accelerate rate comparing to serial schemes as well as existing distributed-system implementations.

A nonlinear orthogonal non-negative matrix factorization approach to subspace clustering

A recent theoretical analysis shows the equivalence between non-negative matrix factorization (NMF) and spectral clustering based approach to subspace clustering. As NMF and many of its variants are essentially linear, we introduce a nonlinear NMF with explicit orthogonality and derive general kernel-based orthogonal multiplicative update rules to solve the subspace clustering problem. In nonlinear orthogonal NMF framework, we propose two subspace clustering algorithms, named kernel-based non-negative subspace clustering KNSC-Ncut and KNSC-Rcut and establish their connection with spectral normalized cut and ratio cut clustering. We further extend the nonlinear orthogonal NMF framework and introduce a graph regularization to obtain a factorization that respects a local geometric structure of the data after the nonlinear mapping. The proposed NMF-based approach to subspace clustering takes into account the nonlinear nature of the manifold, as well as its intrinsic local geometry, which considerably improves the clustering performance when compared to the several recently proposed state-of-the-art methods.

A unified global convergence analysis of multiplicative update rules for nonnegative matrix factorization

Multiplicative update rules are a well-known computational method for nonnegative matrix factorization. Depending on the error measure between two matrices, various types of multiplicative update rules have been proposed so far. However, their convergence properties are not fully understood. This paper provides a sufficient condition for a general multiplicative update rule to have the global convergence property in the sense that any sequence of solutions has at least one convergent subsequence and the limit of any convergent subsequence is a stationary point of the optimization problem. Using this condition, it is proved that many of the existing multiplicative update rules have the global convergence property if they are modified slightly so that all variables take positive values. This paper also proposes new multiplicative update rules based on Kullback–Leibler, Gamma, and Renyi divergences. It is shown that these three rules have the global convergence property if the same modification as above is made.

On equivalence of likelihood maximization of stochastic block model and constrained nonnegative matrix factorization

Community structures detection in complex network is important for understanding not only the topological structures of the network, but also the functions of it. Stochastic block model and nonnegative matrix factorization are two widely used methods for community detection, which are proposed from different perspectives. In this paper, the relations between them are studied. The logarithm of likelihood function for stochastic block model can be reformulated under the framework of nonnegative matrix factorization. Despite model equivalence, the algorithms employed by the two methods are different. Preliminary numerical experiments are carried out to compare the behaviors of the algorithms, demonstrating that the multiplicative update rules for NMF are more effective.

Analyzing the impact of the initialization for the nonnegative tensor factorization in tensor displays

Tensor display is an option in glasses-free three-dimensional (3-D) display technology. An initial solution has to be set to decompose the light-field information to be represented by the system. We have analyzed the impact of the initial guess on the multiplicative update rules in terms of peak signal-to-noise ratio, and proposed a method based on depth map estimation from an input light field. Results from simulations were obtained and compared with previous literature. In our sample, the initial values used have a large influence on results and convergence to a local minimum. The quality of the output stabilizes after a certain number of iterations, suggesting that a limit on such numbers should be imposed. We show that the proposed methods outperform the pre-existing ones.

A unified framework for sparse non-negative least squares using multiplicative updates and the non-negative matrix factorization problem

We study the sparse non-negative least squares (S-NNLS) problem. S-NNLS occurs naturally in a wide variety of applications where an unknown, non-negative quantity must be recovered from linear measurements. We present a unified framework for S-NNLS based on a rectified power exponential scale mixture prior on the sparse codes. We show that the proposed framework encompasses a large class of S-NNLS algorithms and provide a computationally efficient inference procedure based on multiplicative update rules. Such update rules are convenient for solving large sets of S-NNLS problems simultaneously, which is required in contexts like sparse non-negative matrix factorization (S-NMF). We provide theoretical justification for the proposed approach by showing that the local minima of the objective function being optimized are sparse and the S-NNLS algorithms presented are guaranteed to converge to a set of stationary points of the objective function. We then extend our framework to S-NMF, showing that our framework leads to many well known S-NMF algorithms under specific choices of prior and providing a guarantee that a popular subclass of the proposed algorithms converges to a set of stationary points of the objective function. Finally, we study the performance of the proposed approaches on synthetic and real-world data.

Sparsity-Constrained NMF Algorithm Based on Evolution Strategy for Hyperspectral Unmixing

As a powerful and explainable blind separation tool, non-negative matrix factorization (NMF) is attracting increasing attention in Hyperspectral Unmixing(HU). By effectively utilizing the sparsity priori of data, sparsity-constrained NMF has become a representative method to improve the precision of unmixing. However, the optimization technique based on simple multiplicative update rules makes its unmixing results easy to fall into local minimum and lack of robustness. To solve these problems, this paper proposes a new hybrid algorithm for sparsity constrained NMF by intergrating evolutionary computing and multiplicative update rules (MURs). To find the superior solution in each iteration,the proposed algorithm effectively combines the MURs based on alternate optimization technique, the coefficient matrix selection strategy with sparsity measure, as well as the global optimization technique for basis matrix via the differential evolution algorithm .The effectiveness of the proposed method is demonstrated via the experimental results on real data and comparison with representative algorithms.

A Two-Phase Multiobjective Sparse Unmixing Approach for Hyperspectral Data

With the sparse unmixing becoming increasingly popular recently, some advanced regularization algorithms have been proposed for settling this problem. However, they are limited by their “decision ahead of solution” attribute, i.e., the regularization parameters must be preset before the solution is obtained. In this paper, the sparse unmixing problem is first formulated as a two-phase multiobjective problem. The first phase simultaneously minimizes the unmixing residuals and the number of estimated endmembers for automatically finding the real active endmembers from the spectral library. A decomposition-based endmember selection algorithm considering the gene exchange in the population is specially designed for better and quicker search of the decision space. This algorithm can obtain a set of nondominated solutions for better decision of the active endmembers, which are important for the subsequent calculation of the abundance matrix. The second phase concurrently minimizes the unmixing residuals and the total variation term for estimating a preferable abundance matrix. A local search strategy based on the multiplicative update rule is designed in the evolution process for better approximation of the Pareto front. The experimental results on the synthetic as well as the real data reveal that the proposed framework has a better performance in finding the real active endmembers and estimating their corresponding abundances than some advanced regularization algorithms.

The multiplicative update rule for an extension of the iterative constrained endmembers algorithm

ABSTRACTBased on the multiplicative update rule of nonnegative matrix factorization (NMF), we add the nonnegative constraint of endmember matrix to the objective function of iterative constrained endmembers (ICE), and propose a simple multiplicative update rule for ICE, named NMF-ICE. Our method avoids the use of quadratic programming method in ICE, therefore can greatly improve the computational efficiency. Experiments using both simulated and real hyperspectral data show that NMF-ICE is effective in endmember generation.

Audio Source Separation in Reverberant Environments Using $\beta$-Divergence-Based Nonnegative Factorization

In Gaussian model-based multichannel audio source separation, the likelihood of observed mixtures of source signals is parametrized by source spectral variances and by associated spatial covariance matrices. These parameters are estimated by maximizing the likelihood through an expectation-maximization algorithm and used to separate the signals by means of multichannel Wiener filtering. We propose to estimate these parameters by applying nonnegative factorization based on prior information on source variances. In the nonnegative factorization, spectral basis matrices can be defined as the prior information. The matrices can be either extracted or indirectly made available through a redundant library that is trained in advance. In a separate step, applying nonnegative tensor factorization, two algorithms are proposed in order to either extract or detect the basis matrices that best represent the power spectra of the source signals in the observed mixtures. The factorization is achieved by minimizing the $\beta$ -divergence through multiplicative update rules. The sparsity of factorization can be controlled by tuning the value of $\beta$ . Experiments show that sparsity, rather than the value assigned to $\beta$ in the training, is crucial in order to increase the separation performance. The proposed method was evaluated in several mixing conditions. It provides better separation quality with respect to other comparable algorithms.

Hypersharpening by Joint-Criterion Nonnegative Matrix Factorization

Hypersharpening aims at combining an observable low-spatial resolution hyperspectral image with a high-spatial resolution remote sensing image, in particular a multispectral one, to generate an unobservable image with the high spectral resolution of the former and the high spatial resolution of the latter. In this paper, two such new fusion methods are proposed. These methods, related to linear spectral unmixing techniques, and based on nonnegative matrix factorization (NMF), optimize a new joint criterion and extend the recently proposed joint NMF (JNMF) method. The first approach, called gradient-based joint-criterion NMF (Grd-JCNMF), is a gradient-based method. The second one, called multiplicative JCNMF (Mult-JCNMF), uses new designed multiplicative update rules. These two JCNMF approaches are applied to synthetic and semireal data, and their effectiveness, in spatial and spectral domains, is evaluated with commonly used performance criteria. Experimental results show that the proposed JCNMF methods yield sharpened hyperspectral data with good spectral and spatial fidelities. The obtained results are compared with the performance of two NMF-based methods and one approach based on a sparse representation. These results show that the proposed methods significantly outperform the well-known coupled NMF sharpening method for most performance figures. Also, the proposed Mult-JCNMF method provides the results that are similar to those obtained by JNMF, with a lower computational cost. Compared with the tested sparse-representation-based approach, the proposed methods give better results. Moreover, the proposed Grd-JCNMF method considerably surpasses all other tested methods.

Multiplicative Update of Auto-Regressive Gains for Codebook-Based Speech Enhancement

In this paper, we present a novel method for estimating short-term linear predictive parameters of speech and noise in the codebook-driven Wiener filtering speech enhancement method. We only use pretrained spectral shape codebook of speech to model the a priori information about linear predictive coefficients of speech, and the spectral shape of noise is estimated online directly instead of using noise codebook to solve the problem of noise classification. Differing from the existing codebook-driven methods that the linear predictive gains of speech and noise are estimated by maximum-likelihood method, in the proposed method we exploit a multiplicative update rule to estimate the linear predictive gains more accurately. The estimated gains can help to reserve more speech components in the enhanced speech. Meanwhile, the Bayesian parameter-estimator without the noise codebook is also developed. Moreover, we develop an improved codebook-driven Wiener filter combined with the speech-presence probability, so that the proposed method achieves the goal of removing the residual noise between the harmonics of noisy speech.

Robust Non-negative Matrix Factorization with β-Divergence for Speech Separation

This paper addresses the problem of unsupervised speech separation based on robust non-negative matrix factorization (RNMF) with β-divergence, when neither speech nor noise training data is available beforehand. We propose a robust version of non-negative matrix factorization, inspired by the recently developed sparse and low-rank decomposition, in which the data matrix is decomposed into the sum of a low-rank matrix and a sparse matrix. Efficient multiplicative update rules to minimize the β-divergence-based cost function are derived. A convolutional extension of the proposed algorithm is also proposed, which considers the time dependency of the non-negative noise bases. Experimental speech separation results show that the proposed convolutional RNMF successfully separates the repeating time-varying spectral structures from the magnitude spectrum of the mixture, and does so without any prior training.

Local and global regularized concept factorization for image clustering

Concept factorization (CF), as a popular matrix factorization technique, has recently attracted increasing attention in image clustering, due to the strong ability of dimension reduction and data representation. Existing CF variants only consider the local structure of data, but ignore the global structure information embedded in data, which is very crucial for data representation. To address the above issue, we propose an improved CF method, namely local and global regularized concept factorization (LGCF), by considering the local and global structures simultaneously. Specifically, the local geometric structure is depicted in LGCF via a hypergraph, which is capable of precisely capturing high-order geometrical information. In addition, to discover the global structure, we establish an unsupervised discriminant criterion, which characterizes the between-class scatter and the total scatter of the data with the help of latent features in LGCF. For the formulated LGCF, a multiplicative update rule is developed, and the convergence is rigorously proved. Extensive experiments on several real image datasets demonstrate the superiority of the proposed method over the state-of-the-art methods in terms of clustering accuracy and mutual information.

Robust Nonnegative Matrix Factorization via Joint Graph Laplacian and Discriminative Information for Identifying Differentially Expressed Genes

Differential expression plays an important role in cancer diagnosis and classification. In recent years, many methods have been used to identify differentially expressed genes. However, the recognition rate and reliability of gene selection still need to be improved. In this paper, a novel constrained method named robust nonnegative matrix factorization via joint graph Laplacian and discriminative information (GLD-RNMF) is proposed for identifying differentially expressed genes, in which manifold learning and the discriminative label information are incorporated into the traditional nonnegative matrix factorization model to train the objective matrix. Specifically,L2,1-norm minimization is enforced on both the error function and the regularization term which is robust to outliers and noise in gene data. Furthermore, the multiplicative update rules and the details of convergence proof are shown for the new model. The experimental results on two publicly available cancer datasets demonstrate that GLD-RNMF is an effective method for identifying differentially expressed genes.

A Generalized Hierarchical Multi-Latent Space Model for Heterogeneous Learning

In many real world applications such as image annotation, gene function prediction, and insider threat detection, the data collected from heterogeneous sources often exhibit multiple types of heterogeneity, such as task heterogeneity, view heterogeneity, and label heterogeneity. To address this problem, we propose a Hierarchical Multi-Latent Space (HiMLS) learning framework to jointly model the triple types of heterogeneity. The basic idea is to learn a hierarchical multi-latent space by which we can simultaneously leverage the task relatedness, view consistency and the label correlations to improve the learning performance. We first propose a multi-latent space approach to model the complex heterogeneity, which is then used as a building block to stack up a multi-layer structure in order to learn the hierarchical multi-latent space. In such a way, we can gradually learn the more abstract concepts in the higher level. We present two instantiated models of the generalized framework using different divergence measures. The two-phase learning algorithms are used to train the multi-layer models. We drive the multiplicative update rules for pre-training and fine-tuning in each model, and prove the convergence and correctness of the update methods. The effectiveness of the proposed approach is verified on various data sets.