Multiplicative Update Rules Research Articles

Anchor-guided global view reconstruction for multi-view multi-label feature selection

In multi-view multi-label learning (MVML), the accuracy of feature weights is pivotal for establishing feature order. However, conventional MVML methods often struggle with integrating distinct information from multiple views effectively, leading to unclear segmentation and potential noise introduction. To address this challenge, this paper proposes an anchor-based latent representation method for global view learning in MVML. Specifically, we encode inherent information from each view to derive a candidate multi-view representation. Anchors extracted from both the candidate view and the global view are then constrained to approximate equality in the latent space. Furthermore, a carefully designed view matrix serves as supplement, seamlessly integrated into the reconstruction process to augment information. The convergence of results is subsequently validated using multiplicative update rules. Experimental findings showcase the superior performance of our proposed method across various multi-view datasets.

Two-dimensional data partitioning for non-negative matrix tri-factorization

As a two-sided clustering and dimensionality reduction paradigm, Non-negative Matrix Tri-Factorization (NMTF) has attracted much attention in machine learning and data mining researchers due to its excellent performance and reliable theoretical support. Unlike Non-negative Matrix Factorization (NMF) methods applicable to one-sided clustering only, NMTF introduces an additional factor matrix and uses the inherent duality of data to realize the mutual promotion of sample clustering and feature clustering, thus showing great advantages in many scenarios (e.g., text co-clustering). However, the existing methods for solving NMTF usually involve intensive matrix multiplication, which is characterized by high time and space complexities, that is, there are limitations of slow convergence of the multiplicative update rules and high memory overhead. In order to solve the above problems, this paper develops a distributed parallel algorithm with a 2-dimensional data partition scheme for NMTF (i.e., PNMTF-2D). Experiments on multiple text datasets show that the proposed PNMTF-2D can substantially improve the computational efficiency of NMTF (e.g., the average iteration time is reduced by up to 99.7% on Amazon) while ensuring the effectiveness of convergence and co-clustering.

Semi-supervised Multi-view Clustering based on NMF with Fusion Regularization

Multi-view clustering has attracted significant attention and application. Nonnegative matrix factorization is one popular feature of learning technology in pattern recognition. In recent years, many semi-supervised nonnegative matrix factorization algorithms were proposed by considering label information, which has achieved outstanding performance for multi-view clustering. However, most of these existing methods have either failed to consider discriminative information effectively or included too much hyper-parameters. Addressing these issues, a semi-supervised multi-view nonnegative matrix factorization with a novel fusion regularization (FRSMNMF) is developed in this article. In this work, we uniformly constrain alignment of multiple views and discriminative information among clusters with designed fusion regularization. Meanwhile, to align the multiple views effectively, two kinds of compensating matrices are used to normalize the feature scales of different views. Additionally, we preserve the geometry structure information of labeled and unlabeled samples by introducing the graph regularization simultaneously. Due to the proposed methods, two effective optimization strategies based on multiplicative update rules are designed. Experiments implemented on six real-world datasets have demonstrated the effectiveness of our FRSMNMF comparing with several state-of-the-art unsupervised and semi-supervised approaches.

Double-Layer Hybrid-Label Identification Feature Selection for Multi-View Multi-Label Learning

Multi-view multi-label feature selection aims to select informative features where the data are collected from multiple sources with multiple interdependent class labels. For fully exploiting multi-view information, most prior works mainly focus on the common part in the ideal circumstance. However, the inconsistent part hidden in each view, including noises and specific elements, may affect the quality of mapping between labels and feature representations. Meanwhile, ignoring the specific part might lead to a suboptimal result, as each label is supposed to possess specific characteristics of its own. To deal with the double problems in multi-view multi-label feature selection, we propose a unified loss function which is a totally splitting structure for observed labels as hybrid labels that is, common labels, view-to-all specific labels and noisy labels, and the view-to-all specific labels further splits into several specific labels of each view. The proposed method simultaneously considers the consistency and complementarity of different views. Through exploring the feature weights of hybrid labels, the mapping relationships between labels and features can be established sequentially based on their attributes. Additionally, the interrelatedness among hybrid labels is also investigated and injected into the function. Specific to the specific labels of each view, we construct the novel regularization paradigm incorporating logic operations. Finally, the convergence of the result is proved after applying the multiplicative update rules. Experiments on six datasets demonstrate the effectiveness and superiority of our method compared with the state-of-the-art methods.

Dual auto-weighted multi-view clustering via autoencoder-like nonnegative matrix factorization

Multi-view clustering (MVC) can exploit the complementary information among multi-view data to achieve the satisfactory performance, thus having extensive potentials for practical applications. Although Nonnegative Matrix Factorization (NMF) has emerged as an effective technique for MVC, the existing NMF-based methods still have two main limitations: 1) They solely focus on the reconstruction of original data, which can be regarded as the decoder of an autoencoder, while neglecting the low-dimensional representation learning. 2) They lack the ability to effectively capture both linear and nonlinear structures of data. To solve these problems, in this paper, we propose a Dual Auto-weighted multi-view clustering model based on Autoencoder-like NMF (DA2NMF), which enables a comprehensive exploration of both linear and nonlinear structures. Specifically, we establish an autoencoder-like NMF model that learns linear low-dimensional representations by integrating data reconstruction and representation learning within a unified framework. Moreover, the adaptive graph learning is introduced to explore the nonlinear structures in data. We further design a dual auto-weighted strategy to adaptively compute weights for different views and low-dimensional representations, thereby obtaining an enhanced consistent graph. An effective algorithm based on Multiplicative Update Rule (MUR) is developed to solve the DA2NMF with the theoretical convergence guarantee. Experimental results show that the proposed DA2NMF can effectively improve the clustering performance compared with the state-of-the-art MVC algorithms.

Multiplicative update rules for accelerating deep learning training and increasing robustness

Even nowadays, where Deep Learning (DL) has achieved state-of-the-art performance in a wide range of research domains, accelerating training and building robust DL models remains a challenging task. To this end, generations of researchers have pursued developing robust methods for training DL architectures that can be less sensitive to weight distributions, model architectures, and loss landscapes. However, such methods are limited to adaptive learning rate optimizers, initialization schemes, and clipping gradients without investigating the fundamental rule of parameters update. Although multiplicative updates have contributed significantly to the early development of machine learning and hold strong theoretical claims, to the best of our knowledge, this is the first work that investigates them in a unified manner in the context of DL optimization. In this work, we propose an optimization framework that fits a wide range of optimization algorithms and enables one to apply alternative update rules. To this end, we propose a novel multiplicative update rule and we extend their capabilities by combining it with a traditional additive update term, under a novel hybrid update method. We claim that the proposed framework accelerates training while leading to more robust models in contrast to traditionally used additive update rule, and we experimentally demonstrate its effectiveness in a wide range of task and optimization methods. Such tasks range from convex and non-convex optimization to difficult image classification benchmarks applying a wide range of traditionally used optimization methods and Deep Neural Network (DNN) architectures, providing quantitative and qualitative experimental results.

Non-negative Tucker decomposition with double constraints for multiway dimensionality reduction

<p>Nonnegative Tucker decomposition (NTD) is one of the renowned techniques in feature extraction and representation for nonnegative high-dimensional tensor data. The main focus behind the NTD-like model was how to factorize the data to get ahold of a high quality data representation from multidimensional directions. However, existing NTD-like models do not consider relationship and properties between the factor matrix of columns while preserving the geometric structure of the data space. In this paper, we managed to capture nonlinear local features of data space and further enhance expressiveness of the NTD clustering method by syncretizing organically approximately orthogonal constraint and graph regularized constraint. First, based on the uni-side and bi-side approximate orthogonality, we flexibly proposed two novel approximately orthogonal NTD with graph regularized models, which not only in part make the factor matrix tend to be orthogonality, but also preserve the geometrical information from high-dimensional tensor data. Second, we developed the iterative updating algorithm dependent on the multiplicative update rule to solve the proposed models, and provided its convergence and computational complexity. Finally, we used numerical experimental results to demonstrate the effectiveness, robustness, and efficiency of the proposed new methods on the real-world image datasets.</p>

Alpha-divergence two-dimensional nonnegative matrix factorization for biomedical blind source separation

An alpha-divergence two-dimensional nonnegative matrix factorization (NMF2D) for biomedical signal separation is presented. NMF2D is a popular approach for retrieving low-rank approximations of nonnegative data such as image pixel, audio signal, data mining, pattern recognition and so on. In this paper, we concentrate on biomedical signal separation by using NMF2D with alpha-divergence family which decomposes a mixture into two-dimensional convolution factor matrices that represent temporal code and the spectral basis. The proposed iterative estimation algorithm (alpha-divergence algorithm) is initialized with random values, and it updated using multiplicative update rules until the values converge. Simulation experiments were carried out by comparing the original and estimated signal in term of signal-to-distortion ratio (SDR). The performances have been evaluated by including and excluding the sparseness constraint which sparseness is favored by penalizing nonzero gains. As a result, the proposed algorithm improved the iteration speed and sparseness constraints produce slight improvement of SDR.

The Generalization of Non-Negative Matrix Factorization Based on Algorithmic Stability

The Non-negative Matrix Factorization (NMF) is a popular technique for intelligent systems, which can be widely used to decompose a nonnegative matrix into two factor matrices: a basis matrix and a coefficient one, respectively. The main objective of NMF is to ensure that the operation results of the two matrices are as close to the original matrix as possible. Meanwhile, the stability and generalization ability of the algorithm should be ensured. Therefore, the generalization performance of NMF algorithms is analyzed from the perspective of algorithm stability and the generalization error bounds are given, which is named AS-NMF. Firstly, a general NMF prediction algorithm is proposed, which can predict the labels for new samples, and then the corresponding loss function is defined further. Secondly, the stability of the NMF algorithm is defined according to the loss function, and two generalization error bounds can be obtained by employing uniform stability in the case where U is fixed and it is not fixed under the multiplicative update rule. The bounds numerically show that its stability parameter depends on the upper bound on the module length of the input data, dimension of hidden matrix and Frobenius norm of the basis matrix. Finally, a general and stable framework is established, which can analyze and measure generalization error bounds for the NMF algorithm. The experimental results demonstrate the advantages of new methods on three widely used benchmark datasets, which indicate that our AS-NMF can not only achieve efficient performance, but also outperform the state-of-the-art of recommending tasks in terms of model stability.

Hyperspectral Unmixing Using Higher-Order Graph Regularized NMF With Adaptive Feature Selection

Recently, graph learning methods have attracted much research attention, which uses first-order nearest-neighbor relation between pixels to construct adjacency graphs for capturing smooth abundance. However, the first-order nearest-neighbor information neglects the shared domain structure of pixels making the higher-order nearest-neighbor relation missing resulting in limited spatial structure learning. Additionally, the feature values in hyperspectral images vary greatly in different bands, tiny value contributions are easily neglected due to the statistical properties of the model, which hinders the learning efficiency of the approach. To address these shortcomings, a higher-order graph regularizer nonnegative matrix factorization with adaptive feature selection method is proposed. Specifically, we introduce pixel second-order nearest-neighbor relation in graph learning to capture the nearest-neighbor domains of pixels with missing connection in first-order neighborhoods and enhance the aggregation of homogeneous regions. Then, the first-order and second-order nearest-neighbor relation are combined to construct higher-order graph regularizer to better preserve the global spatial structure of the image. Additionally, adaptive weight is introduced in the data reconstruction to balance the effects of different feature values by adaptively selecting spectral features to enhance the contribution of tiny features. Finally, the objective function is designed by fusing the higher-order graph regularization, adaptive feature selection and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sub> sparse regularizer into the nonnegative matrix factorization framework, and the optimization algorithm of the model is derived using a multiplicative update rule. We conducted extensive experiments on synthetic and real datasets, and the effectiveness and superiority of the proposed method were verified by comparing it with several state-of-the-art methods.

Semi-blind hyperspectral and multispectral image fusion based on a non-factorization model

Fusion of low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI) is currently the most cost-effective method for acquiring high-resolution hyperspectral images (HR-HSI). The existing fusion models like spectral unmixing, matrix-factorization, and tensor-factorization require decomposing the target HR-HSI into multiple factor matrices or tensors before fusion. Usually requires a large number of variable optimization processes. So, the above approaches do not apply to the arguably real-time scenario. Moreover, the miscellaneous intermediate estimates render poor fusion stability. This paper proposes a novel non-factorization model to address the above problems. The model firstly gets access to a rough estimate of HR-HSI through the Moore–Penrose inverse operation. Then, we acquire the further HR-HSI forecast according to the correlation between the target HR-HSI and LR-HSI singular values through the coupled singular value decomposition (SVD) of the rough HR-HSI and LR-HIS. Finally, we obtain the precise HR-HSI estimates via the multiplicative update rule. Our proposed model directly estimates the target HR-HSI, reduces the estimation process of intermediate variables, effectively saves calculation time, and enhances the stability of the fusion. We conduct experiments on remote sensing, ground-based, and real datasets to validate our proposed model’s real-time and stability.

Fast multiplicative algorithms for symmetric nonnegative tensor factorization

Symmetric nonnegative tensor factorization (SNTF) is an important tool for clustering analysis. To date, most of algorithms for SNTF are based on multiplicative update rules, which have many attractive properties, e.g., they are often simple to implement and can enforce nonnegativity without extra projection steps. However, the existing multiplicative algorithms often converge slowly due to the conservative multiplicative learning steps or the use of low-level BLAS (basic linear algebra subprograms) in their implementation. In this paper, three new multiplicative algorithms are proposed for SNTF to overcome the drawback of slow convergence. First, a parallel multiplicative algorithm, which can be implemented with high-level BLAS, is derived by auxiliary optimization. To further accelerate the convergence, two new parallel multiplicative algorithms, which enable larger learning steps for improving efficiency, are developed based on weighted geometric mean and weighted arithmetic mean, respectively. Finally, we apply the proposed algorithms to multiway probabilistic clustering, where a new hyper-stochastic normalization scheme based on Euclidean distance is developed for better data preprocessing. The experiment results on both synthetic and real-world data show that the proposed SNTF algorithms converge faster than the state-of-the-art algorithms.

Convolutive Transfer Function-Based Multichannel Nonnegative Matrix Factorization for Overdetermined Blind Source Separation

Most multichannel blind source separation (BSS) approaches rely on a spatial model to encode the transfer functions from sources to microphones and a source model to encode the source power spectral density. The rank-1 spatial model has been widely exploited in independent component analysis (ICA), independent vector analysis (IVA), and independent low-rank matrix analysis (ILRMA). The full-rank spatial model is also considered in many BSS approaches, such as full-rank spatial covariance matrix analysis (FCA), multichannel nonnegative matrix factorization (MNMF), and FastMNMF, which can improve the separation performance in the case of long reverberation times. This paper proposes a new MNMF framework based on the convolutive transfer function (CTF) for overdetermined BSS. The time-domain convolutive mixture model is approximated by a frequency-wise convolutive mixture model instead of the widely adopted frequency-wise instantaneous mixture model. The iterative projection algorithm is adopted to estimate the demixing matrix, and the multiplicative update rule is employed to estimate nonnegative matrix factorization (NMF) parameters. Finally, the source image is reconstructed using a multichannel Wiener filter. The advantages of the proposed method are twofold. First, the CTF approximation enables us to use a short window to represent long impulse responses. Second, the full-rank spatial model can be derived based on the CTF approximation and slowly time-variant source variances, and close relationships between the proposed method and ILRMA, FCA, MNMF and FastMNMF are revealed. Extensive experiments show that the proposed algorithm achieves a higher separation performance than ILRMA and FastMNMF in reverberant environments.

A convergent algorithm for bi-orthogonal nonnegative matrix tri-factorization

A convergent algorithm for nonnegative matrix factorization (NMF) with orthogonality constraints imposed on both basis and coefficient matrices is proposed in this paper. This factorization concept was first introduced by Ding et al. (Proceedings of 12th ACM SIGKDD international conference on knowledge discovery and data mining, pp 126–135, 2006) with intent to further improve clustering capability of NMF. However, as the original algorithm was developed based on multiplicative update rules, the convergence of the algorithm cannot be guaranteed. In this paper, we utilize the technique presented in our previous work Mirzal (J Comput Appl Math 260:149–166, 2014a; Proceedings of the first international conference on advanced data and information engineering (DaEng-2013). Springer, pp 177–184, 2014b; IEEE/ACM Trans Comput Biol Bioinform 11(6):1208–1217, 2014c) to develop a convergent algorithm for this problem and prove that it converges to a stationary point inside the solution space. As it is very hard to numerically show the convergence of an NMF algorithm due to the slow convergence and numerical precision issues, experiments are instead performed to evaluate whether the algorithm has the nonincreasing property (a necessary condition for the convergence) where it is shown that the algorithm has this property. Further, clustering capability of the algorithm is also inspected by using Reuters-21578 data corpus.

Hyperspectral Unmixing Based on Constrained Bilinear or Linear-Quadratic Matrix Factorization

Unsupervised hyperspectral unmixing methods aim to extract endmember spectra and infer the proportion of each of these spectra in each observed pixel when considering linear mixtures. However, the interaction between sunlight and the Earth’s surface is often very complex, so that observed spectra are then composed of nonlinear mixing terms. This nonlinearity is generally bilinear or linear quadratic. In this work, unsupervised hyperspectral unmixing methods, designed for the bilinear and linear-quadratic mixing models, are proposed. These methods are based on bilinear or linear-quadratic matrix factorization with non-negativity constraints. Two types of algorithms are considered. The first ones only use the projection of the gradient, and are therefore linked to an optimal manual choice of their learning rates, which remains the limitation of these algorithms. The second developed algorithms, which overcome the above drawback, employ multiplicative projective update rules with automatically chosen learning rates. In addition, the endmember proportions estimation, with three alternative approaches, constitutes another contribution of this work. Besides, the reduction of the number of manipulated variables in the optimization processes is also an originality of the proposed methods. Experiments based on realistic synthetic hyperspectral data, generated according to the two considered nonlinear mixing models, and also on two real hyperspectral images, are carried out to evaluate the performance of the proposed approaches. The obtained results show that the best proposed approaches yield a much better performance than various tested literature methods.

Total Variation Constrained Non-Negative Matrix Factorization for Medical Image Registration

This paper presents a novel medical image registration algorithm named total variation constrained graph-regularization for non-negative matrix factorization (TV-GNMF). The method utilizes non-negative matrix factorization by total variation constraint and graph regularization. The main contributions of our work are the following. First, total variation is incorporated into NMF to control the diffusion speed. The purpose is to denoise in smooth regions and preserve features or details of the data in edge regions by using a diffusion coefficient based on gradient information. Second, we add graph regularization into NMF to reveal intrinsic geometry and structure information of features to enhance the discrimination power. Third, the multiplicative update rules and proof of convergence of the TV-GNMF algorithm are given. Experiments conducted on datasets show that the proposed TV-GNMF method outperforms other state-of-the-art algorithms.

Multi-view data clustering via non-negative matrix factorization with manifold regularization

Nowadays, non-negative matrix factorization (NMF) based cluster analysis for multi-view data shows impressive behavior in machine learning. Usually, multi-view data have complementary information from various views. The main concern behind the NMF is how to factorize the data to achieve a significant clustering solution from these complementary views. However, NMF does not focus to conserve the geometrical structures of the data space. In this article, we intensify on the above issue and evolve a new NMF clustering method with manifold regularization for multi-view data. The manifold regularization factor is exploited to retain the locally geometrical structure of the data space and gives extensively common clustering solution from multiple views. The weight control term is adopted to handle the distribution of each view weight. An iterative optimization strategy depended on multiplicative update rule is applied on the objective function to achieve optimization. Experimental analysis on the real-world datasets are exhibited that the proposed approach achieves better clustering performance than some state-of-the-art algorithms.

Multi-agent solver for non-negative matrix factorization based on optimization

This paper investigates a distributed solver for non-negative matrix factorization (NMF) over a multi-agent network. After reformulating the problem into the standard distributed optimization form, we design our distributed algorithm (DisNMF) based on the primal-dual method and in the form of multiplicative update rule. With the help of auxiliary functions, we provide monotonic convergence analysis. Furthermore, we show by computational complexity analysis and numerical examples that our distributed NMF algorithm performs well in comparison with the centralized NMF algorithm.

Incomplete Multi-View Clustering with Reconstructed Views

As one category of important incomplete multi-view clustering methods, subspace based methods seek the common latent representation of incomplete multi-view data by matrix factorization and then partition the latent representation to get clustering results. However, these methods ignore missing views in the process of matrix factorization, which makes the connection of different views be exploited inadequately. This paper proposes Incomplete Multi-view Clustering with Reconstructed Views (IMCRV), which utilizes the incomplete examples sufficiently. In IMCRV, the missing views of incomplete examples are reconstructed and the reconstructed views are also used to seek the common latent representation. IMCRV also involves the Laplacian regularization to preserve the global property of the latent representation. A novel gradient descent method with the multiplicative update rule is designed to solve the objective function of IMCRV. The corresponding iterative algorithm is developed and the convergence of the algorithm is proved. IMCRV is compared with many state-of-the-art incomplete multi-view clustering methods under different Incomplete Example Rates (IER) on public multi-view datasets. The experimental results demonstrate the superior effectiveness of IMCRV.

Total Variation Constrained Graph-Regularized Convex Non-Negative Matrix Factorization for Data Representation

We propose a novel NMF algorithm, named Total Variation constrained Graph-regularized Convex Non-negative Matrix Factorization (TV-GCNMF), to incorporate total variation and graph Laplacian with convex NMF. In this model, the feature details of the data are preserved by a diffusion coefficient based on the gradient information. The graph regularization and convex constraints reveal the intrinsic geometry and structure information of the features; thereby, obtaining sparse and parts-based representations. Furthermore, we give the multiplicative update rules and prove convergence of the proposed algorithm. The results of clustering experiments on multiple datasets, under various noise conditions, show the effectiveness and robustness of the proposed method compared to state-of-the-art clustering methods and other related work.