Multiplicative Update Algorithm Research Articles

Hypergraph regularized nonnegative triple decomposition for multiway data analysis

Tucker decomposition is widely used for image representation, data reconstruction, and machine learning tasks, but the calculation cost for updating the Tucker core is high. Bilevel form of triple decomposition (TriD) overcomes this issue by decomposing the Tucker core into three low-dimensional third-order factor tensors and plays an important role in the dimension reduction of data representation. TriD, on the other hand, is incapable of precisely encoding similarity relationships for tensor data with a complex manifold structure. To address this shortcoming, we take advantage of hypergraph learning and propose a novel hypergraph regularized nonnegative triple decomposition for multiway data analysis that employs the hypergraph to model the complex relationships among the raw data. Furthermore, we develop a multiplicative update algorithm to solve our optimization problem and theoretically prove its convergence. Finally, we perform extensive numerical tests on six real-world datasets, and the results show that our proposed algorithm outperforms some state-of-the-art methods.

Deep Multiplicative Update Algorithm for Nonnegative Matrix Factorization and Its Application to Audio Signals

Deep Multiplicative Update Algorithm for Nonnegative Matrix Factorization and Its Application to Audio Signals

Scalable NMF via linearly optimized data compression.

Orthonormal projective non-negative matrix factorization (opNMF) has been widely used in neuroimaging and clinical neuroscience applications to derive representations of the brain in health and disease. The non-negativity and orthonormality constraints of opNMF result in intuitive and well-localized factors. However, the advantages of opNMF come at a steep computational cost that prohibits its use in large-scale data. In this work, we propose novel and scalable optimization schemes for orthonormal projective non-negative matrix factorization that enable the use of the method in large-scale neuroimaging settings. We replace the high-dimensional data matrix with its corresponding singular value decomposition (SVD) and QR decompositions and combine the decompositions with opNMF multiplicative update algorithm. Empirical validation of the proposed methods demonstrated significant speed-up in computation time while keeping memory consumption low without compromising the accuracy of the solution.

Fast and Accurate Non-Negative Latent Factor Analysis of High-Dimensional and Sparse Matrices in Recommender Systems

A fast non-negative latent factor (FNLF) model for a high-dimensional and sparse (HiDS) matrix adopts a Single Latent Factor-dependent, Non-negative, Multiplicative and Momentum-incorporated Update (SLF-NM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> U) algorithm, which enables its fast convergence. It is crucial to achieve a rigorously theoretical proof regarding its fast convergence, which has not been provided in prior research. Aiming at addressing this critical issue, this work theoretically proves that with an appropriately chosen momentum coefficient, SLF-NM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> U enables the fast convergence of an FNLF model in both continuous and discrete time cases. Empirical analysis of HiDS matrices generated by representative industrial applications provides empirical evidences for the theoretical proof. Hence, this study represents an important milestone in the field of HiDS matrix analysis.

Semi-supervised graph regularized concept factorization with the class-driven constraint for image representation

&lt;abstract&gt;&lt;p&gt;As a popular dimensionality reduction technique, concept factorization (CF) has been widely applied in image clustering. However, CF fails to extract the intrinsic structure of data space and does not utilize the label information. In this paper, a new semi-supervised graph regularized CF (SGCF) method is proposed, which makes full use of the limited label information and the graph regularization to improve the algorithm of clustering performance. Particularly, SGCF associates the class label information of data points with their new representations by using the class-driven constraint, and this constraint forces the new representations of data points to be more similar within the same class while different between classes. Furthermore, SGCF extracts the geometric structure of the data space by incorporating graph regularization. SGCF not only reveals the geometrical structure of the data space, but also takes into the limited label information account. We drive an efficient multiplicative update algorithm for SGCF to solve the optimization, and analyze the proposed SGCF method in terms of the convergence and computational complexity. Clustering experiments show the effectiveness of the SGCF method in comparison to other state-of-the-art methods.&lt;/p&gt;&lt;/abstract&gt;

High-dimensional sparse portfolio selection with nonnegative constraint

Portfolio selection is a fundamental problem in finance with challenges of dimensionality and market complexities. This paper focuses on the prevalent strategy of passive portfolio management, called index tracking, considering the no-short sales, volatility, transaction costs, and the limited set of effective samples. An effective method is proposed for the high-dimensional sparse portfolio selection by using the nonconcave penalty SCAD and the nonnegative constraint. Oracle statistical properties are studied, and the Multiplicative Updates algorithm is applied for the method. The detailed comparisons of the proposed method with other existing nonnegative methods are shown in simulations and empirical analysis, which demonstrate that the proposed method has better performance.

LNS-Madam: Low-Precision Training in Logarithmic Number System Using Multiplicative Weight Update

Representing deep neural networks (DNNs) in low-precision is a promising approach to enable efficient acceleration and memory reduction. Previous methods that train DNNs in low-precision typically keep a copy of weights in high-precision during the weight updates. Directly training with low-precision weights leads to accuracy degradation due to complex interactions between the low-precision number systems and the learning algorithms. To address this issue, we develop a co-designed low-precision training framework, termed LNS-Madam, in which we jointly design a logarithmic number system (LNS) and a multiplicative weight update algorithm (Madam). We prove that LNS-Madam results in low quantization error during weight updates, leading to stable performance even if the precision is limited. We further propose a hardware design of LNS-Madam that resolves practical challenges in implementing an efficient datapath for LNS computations. Our implementation effectively reduces energy overhead incurred by LNS-to-integer conversion and partial sum accumulation. Experimental results show that LNS-Madam achieves comparable accuracy to full-precision counterparts with only 8 bits on popular computer vision and natural language tasks. Compared to FP32 and FP8, LNS-Madam reduces the energy consumption by over 90% and 55%, respectively.

Nonnegative Latent Factor Analysis-Incorporated and Feature-Weighted Fuzzy Double $c$-Means Clustering for Incomplete Data

Fuzzy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> -means (FCM) clustering is a promising method to handle uncertainties in data clustering. However, the traditional FCM and most of its variants cannot address incomplete inputs. To this aim, a novel fuzzy clustering framework is put forward to perform highly accurate clustering on incomplete data. It adopts twofold ideas: 1) Utilizing a nonnegative latent factor model to prefill the missing data in the inputs by rigidly extracting involved entities’ latent features, where the principle of a minibatch gradient descent algorithm is incorporated into a single latent factor-dependent, nonnegative and multiplicative update algorithm to accelerate the convergence rate; and 2) integrating the distribution of inputs and the weights of local features into the objective function through sparse self-representation and weighting allocation to focus on crucial features. In this way, a NLF analysis-incorporated and feature-weighted fuzzy double <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> -means clustering (NF <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^2$</tex-math></inline-formula> D) method is achieved, where the data distribution and instance correlation are simultaneously considered with care. Experiments on 12 real-world datasets including both data and images with different missing rates show that the proposed NF <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^2$</tex-math></inline-formula> D method has a significant superiority over state-of-the-art fuzzy clustering methods.

An α-β-Divergence-Generalized Recommender for Highly Accurate Predictions of Missing User Preferences.

To quantify user-item preferences, a recommender system (RS) commonly adopts a high-dimensional and sparse (HiDS) matrix. Such a matrix can be represented by a non-negative latent factor analysis model relying on a single latent factor (LF)-dependent, non-negative, and multiplicative update algorithm. However, existing models' representative abilities are limited due to their specialized learning objective. To address this issue, this study proposes an α- β -divergence-generalized model that enjoys fast convergence. Its ideas are three-fold: 1) generalizing its learning objective with α- β -divergence to achieve highly accurate representation of HiDS data; 2) incorporating a generalized momentum method into parameter learning for fast convergence; and 3) implementing self-adaptation of controllable hyperparameters for excellent practicability. Empirical studies on six HiDS matrices from real RSs demonstrate that compared with state-of-the-art LF models, the proposed one achieves significant accuracy and efficiency gain to estimate huge missing data in an HiDS matrix.

A kind of new time-weighted nonnegative lasso index-tracking model and its application

This study describes improved index-tracking methods to replicate the target index’s market performance in a high-dimensional sparse linear regression with nonnegative constraints on the coefficients. The main objective of this study is to construct a sparse portfolio with a better prediction effect and robustness. Considering the influence of time factors on index tracking, we propose a time-weighted nonnegative lasso index tracking model under different market constraints and define two new time-weighted construction methods. This index tracking model is an extension of Lasso and has variable selection consistency and estimation consistency under time-weighted nonnegative irrepresentable conditions similar to the irrepresentable condition in Lasso. We use the multiplicative updates algorithm to obtain the model’s solution since it is faster and simpler. The constrained index tracking problem in the stock market without short sales is studied in the latter part. The empirical results indicate that the optimized time-weighted nonnegative lasso index tracking model can obtain a smaller out-of-sample tracking error. The constructed portfolio has a better prediction effect and robustness, and we find that the exponential time-weighted method is better than the linear time-weighted method in capturing time information.

Graph matching based on fast normalized cut and multiplicative update mapping

Point correspondence is a fundamental problem in pattern recognition and computer vision, which can be tackled by graph matching. Since graph matching is basically an NP-complete problem, some approximate methods are proposed to solve it. Continuous relaxation offers an effective approximate method for graph matching problem. However, the discrete constraint is not taken into consideration in the optimization step. In this paper, a fast normalized cut based graph matching method is proposed, where the discrete constraint is introduced into the optimization step. Specifically, first a semidefinite positive affinity matrix based form objective function is constructed by introducing a regularization term which is related to the discrete constraint. Then the fast normalized cut algorithm is utilized to find the continuous solution. Last, the discrete solution of graph matching is obtained by a multiplicative update algorithm. Experiments on both synthetic points and real-world images validate the effectiveness of the proposed method by comparing it with the state-of-the-art methods.

Direction Matters: On Influence-Preserving Graph Summarization and Max-Cut Principle for Directed Graphs

Summarizing large-scale directed graphs into small-scale representations is a useful but less-studied problem setting. Conventional clustering approaches, based on Min-Cut-style criteria, compress both the vertices and edges of the graph into the communities, which lead to a loss of directed edge information. On the other hand, compressing the vertices while preserving the directed-edge information provides a way to learn the small-scale representation of a directed graph. The reconstruction error, which measures the edge information preserved by the summarized graph, can be used to learn such representation. Compared to the original graphs, the summarized graphs are easier to analyze and are capable of extracting group-level features, useful for efficient interventions of population behavior. In this letter, we present a model, based on minimizing reconstruction error with nonnegative constraints, which relates to a Max-Cut criterion that simultaneously identifies the compressed nodes and the directed compressed relations between these nodes. A multiplicative update algorithm with column-wise normalization is proposed. We further provide theoretical results on the identifiability of the model and the convergence of the proposed algorithms. Experiments are conducted to demonstrate the accuracy and robustness of the proposed method.

Generalized Nesterov's Acceleration-Incorporated, Non-Negative and Adaptive Latent Factor Analysis

A non-negative latent factor (NLF) model with a single latent factor-dependent, non-negative and multiplicative update (SLF-NMU) algorithm is frequently adopted to extract useful knowledge from non-negative data represented by high-dimensional and sparse (HiDS) matrices arising from various service-oriented applications. However, its convergence rate is slow. To address this issue, this study proposes a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</u> eneralized Nesterov's acceleration-incorporated, <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</u> on-negative and <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</u> daptive <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</u> atent <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</u> actor (GNALF) model. It results from a) incorporating a generalized Nesterov's accelerated gradient (NAG) method into an SLF-NMU algorithm, thereby achieving an <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</u> AG-incorporated and <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</u> lement-oriented <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</u> on-negative (NEN) algorithm to perform efficient parameter update; and b) making its regularization and acceleration parameters self-adaptive via incorporating the principle of a particle swarm optimization algorithm into the training process, thereby implementing a highly adaptive and practical model. Empirical studies on six large sparse matrices from different recommendation service applications show that a GNALF model achieves very high convergence rate without the need of hyper-parameter tuning, making its computational efficiency significantly higher than state-of-the-art models. Meanwhile, such efficiency gain does not result in accuracy loss, since its prediction accuracy is comparable with its peers. Hence, it can better serve practical service applications with real-time demands.

A Fast Non-Negative Latent Factor Model Based on Generalized Momentum Method

Non-negative latent factor (NLF) models can efficiently acquire useful knowledge from high-dimensional and sparse (HiDS) matrices filled with non-negative data. Single latent factor-dependent, non-negative and multiplicative update (SLF-NMU) is an efficient algorithm for building an NLF model on an HiDS matrix, yet it suffers slow convergence. A momentum method is frequently adopted to accelerate a learning algorithm, but it is incompatible with those implicitly adopting gradients like SLF-NMU. To build a fast NLF (FNLF) model, we propose a generalized momentum method compatible with SLF-NMU. With it, we further propose a single latent factor-dependent non-negative, multiplicative and momentum-incorporated update algorithm, thereby achieving an FNLF model. Empirical studies on six HiDS matrices from industrial application indicate that an FNLF model outperforms an NLF model in terms of both convergence rate and prediction accuracy for missing data. Hence, compared with an NLF model, an FNLF model is more practical in industrial applications.

Family of chaotic maps from game theory

ABSTRACT From a two-agent, two-strategy congestion game where both agents apply the multiplicative weights update algorithm, we obtain a two-parameter family of maps of the unit square to itself. Interesting dynamics arise on the invariant diagonal, on which a two-parameter family of bimodal interval maps exhibits periodic orbits and chaos. While the fixed point b corresponding to a Nash equilibrium of such map f is usually repelling, it is globally Cesàro attracting on the diagonal, that is, for every . This solves a known open question whether there exists a ‘natural’ nontrivial smooth map other than with centres of mass of all periodic orbits coinciding. We also study the dependence of the dynamics on the two parameters.

Momentum-Incorporated Symmetric Non-Negative Latent Factor Models

Symmetric high-dimensional and sparse (SHiDS) networks are frequently found in various industrial applications. A symmetric non-negative latent factor (SNLF) model can acquire essential features from them precisely, yet it suffers from slow convergence. To address this issue, this article integrates a generalized momentum method into a symmetric, single latent factor-dependent, non-negative and multiplicative update (S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> LF-NMU) algorithm, thereby achieving a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</u> omentum-incorporated, <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</u> ymmetric, <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</u> ingle-latent-factor-dependent <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</u> on-negative-multiplicative-update (MS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> N) algorithm. Based on an MS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> N algorithm, momentum-incorporated symmetric non-negative latent factor (MSNLF) models are proposed for an SHiDS network, which ensures fast convergence as well as high representative learning ability. Empirical studies on four SHiDS networks from industrial applications demonstrate that compared with state-of-the-art models, the proposed MSNLF models have significantly higher computational efficiency and representative learning ability.

An Accelerated Symmetric Nonnegative Matrix Factorization Algorithm Using Extrapolation

Symmetric nonnegative matrix factorization (SNMF) approximates a symmetric nonnegative matrix by the product of a nonnegative low-rank matrix and its transpose. SNMF has been successfully used in many real-world applications such as clustering. In this paper, we propose an accelerated variant of the multiplicative update (MU) algorithm of He et al. designed to solve the SNMF problem. The accelerated algorithm is derived by using the extrapolation scheme of Nesterov and a restart strategy. The extrapolation scheme plays a leading role in accelerating the MU algorithm of He et al. and the restart strategy ensures that the objective function of SNMF is monotonically decreasing. We apply the accelerated algorithm to clustering problems and symmetric nonnegative tensor factorization (SNTF). The experiment results on both synthetic and real-world data show that it is more than four times faster than the MU algorithm of He et al. and performs favorably compared to recent state-of-the-art algorithms.

Joint Intelligence Ranking by Federated Multiplicative Update

The joint intelligence ranking of intelligent systems like autonomous driving is of great importance for building a more general, extensive, and universally accepted intelligence evaluation scheme. However, due to issues such as privacy security and industry or area competition, the integration of isolated test results may face large unimaginable difficulty in information security and encrypted model training. To address this, we derive the federated multiplicative update (FMU) algorithm with boundary constraints to solve the nonnegative matrix factorization based joint intelligence ranking. The encrypted learning process is developed to alternate original computation steps in multiplicative update algorithms. Owning feasible property for the fast convergence and secure exchange of variables, the proposed framework outperforms the previous work on both real and simulated data. Further experimental analysis reveals that the introduced federated mechanism does not harm the overall time efficiency.

Spectral unmixing applied to fast identification of γ-emitting radionuclides using NaI(Tl) detectors

Spectral unmixing was investigated for fast spectroscopic identification in γ-emitter mixtures at low-statistics in the case of measurements performed to prevent illegal nuclear material trafficking or for in situ environmental analysis following a radiological or nuclear accident. For that purpose, a multiplicative update algorithm based on full-spectrum analysis was tested in the case of a 3″x3″ NaI(Tl) detector. Automatic decision-making was addressed using Monte Carlo calculations of decision thresholds and detection limits. The first results obtained with a portable instrument equipped with a 3″x3″ NaI(Tl) detector designed for the control of food samples by non-expert users following a radiological or nuclear accident, are also presented.

Majorization-Minimization Algorithm for Discriminative Non-Negative Matrix Factorization

This paper proposes a basis training algorithm for discriminative non-negative matrix factorization (NMF) with applications to single-channel audio source separation. With an NMF-based approach to supervised audio source separation, NMF is first applied to train the basis spectra of each source using training examples and then applied to the spectrogram of a mixture signal using the pretrained basis spectra at test time. The source signals can then be separated out using a Wiener filter. Here, a typical way to train the basis spectra is to minimize the dissimilarity measure between the observed spectrogram and the NMF model. However, obtaining the basis spectra in this way does not ensure that the separated signal will be optimal at test time due to the inconsistency between the objective functions for training and separation (Wiener filtering). To address this mismatch, a framework called discriminative NMF (DNMF) has recently been proposed. While this framework is noteworthy in that it uses a common objective function for training and separation, the objective function becomes more analytically complex than that of regular NMF. In the original DNMF work, a multiplicative update algorithm was proposed for the basis training; however, the convergence of the algorithm is not guaranteed and can be very slow. To overcome this weakness, this paper proposes a convergence-guaranteed algorithm for DNMF based on a majorization-minimization principle. Experimental results show that the proposed algorithm outperform the conventional DNMF algorithm as well as the regular NMF algorithm in terms of both the signal-to-distortion and signal-to-interference ratios.