Hierarchical Alternating Least Squares Research Articles

Convergence of a Fast Hierarchical Alternating Least Squares Algorithm for Nonnegative Matrix Factorization

The hierarchical alternating least squares (HALS) algorithms are powerful tools for nonnegative matrix factorization (NMF), among which the Fast-HALS, proposed in [A. Cichocki and A.-H. Phan, 2009], is one of the most efficient. This paper investigates the convergence of Fast-HALS. First, a more general weak convergence (converged subsequences exist and converge to the stationary point set) is established without any assumption, while most existing results assume all the columns of iterates are strictly away from the origin. Then, a simplified strong convergence (the entire sequence converges to a stationary point) proof is provided. The existing strong convergence is attributed to the block prox-linear (BPL) method, which is a more general framework including Fast-HALS as a special case. So, the convergence proof under BPL is quite complex. Our simplified proof explores the structure of Fast-HALS and can be regarded as a complement to the results under BPL. In addition, some numerical verifications are presented.

A novel update rule of HALS algorithm for nonnegative matrix factorization and Zangwill’s global convergence

Nonnegative Matrix Factorization (NMF) has attracted a great deal of attention as an effective technique for dimensionality reduction of large-scale nonnegative data. Given a nonnegative matrix, NMF aims to obtain two low-rank nonnegative factor matrices by solving a constrained optimization problem. The Hierarchical Alternating Least Squares (HALS) algorithm is a well-known and widely-used iterative method for solving such optimization problems. However, the original update rule used in the HALS algorithm is not well defined. In this paper, we propose a novel well-defined update rule of the HALS algorithm, and prove its global convergence in the sense of Zangwill. Unlike conventional globally-convergent update rules, the proposed one allows variables to take the value of zero and hence can obtain sparse factor matrices. We also present two stopping conditions that guarantee the finite termination of the HALS algorithm. The practical usefulness of the proposed update rule is shown through experiments using real-world datasets.

An Alternating Rank-k Nonnegative Least Squares Framework (ARkNLS) for Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) is a prominent technique for data dimensionality reduction that has been widely used for text mining, computer vision, pattern discovery, and bioinformatics. In this paper, a framework called ARkNLS (Alternating Rank-k Nonnegativity constrained Least Squares) is proposed for computing NMF. First, a recursive formula for the solution of the rank-k nonnegativity-constrained least squares (NLS) is established. This recursive formula can be used to derive the closed-form solution for the Rank-k NLS problem for any integer k $\ge$ 1. As a result, each subproblem for an alternating rank-k nonnegative least squares framework can be obtained based on this closed form solution. Assuming that all matrices involved in rank-k NLS in the context of NMF computation are of full rank, two of the currently best NMF algorithms HALS (hierarchical alternating least squares) and ANLS-BPP (Alternating NLS based on Block Principal Pivoting) can be considered as special cases of ARkNLS with k = 1 and k = r for rank r NMF, respectively. This paper is then focused on the framework with k = 3, which leads to a new algorithm for NMF via the closed-form solution of the rank-3 NLS problem. Furthermore, a new strategy that efficiently overcomes the potential singularity problem in rank-3 NLS within the context of NMF computation is also presented. Extensive numerical comparisons using real and synthetic data sets demonstrate that the proposed algorithm provides state-of-the-art performance in terms of computational accuracy and cpu time.

Assessment of nonnegative matrix factorization algorithms for electroencephalography spectral analysis

BackgroundNonnegative matrix factorization (NMF) has been successfully used for electroencephalography (EEG) spectral analysis. Since NMF was proposed in the 1990s, many adaptive algorithms have been developed. However, the performance of their use in EEG data analysis has not been fully compared. Here, we provide a comparison of four NMF algorithms in terms of accuracy of estimation, stability (repeatability of the results) and time complexity of algorithms with simulated data. In the practical application of NMF algorithms, stability plays an important role, which was an emphasis in the comparison. A Hierarchical clustering algorithm was implemented to evaluate the stability of NMF algorithms.ResultsIn simulation-based comprehensive analysis of fit, stability, accuracy of estimation and time complexity, hierarchical alternating least squares (HALS) low-rank NMF algorithm (lraNMF_HALS) outperformed the other three NMF algorithms. In the application of lraNMF_HALS for real resting-state EEG data analysis, stable and interpretable features were extracted.ConclusionBased on the results of assessment, our recommendation is to use lraNMF_HALS, providing the most accurate and robust estimation.

Two fast vector-wise update algorithms for orthogonal nonnegative matrix factorization with sparsity constraint

Recently, orthogonal nonnegative matrix factorization (ONMF) has been introduced and shown to work remarkably well for clustering tasks. Because of the nonnegativity and the orthogonality constraints, the orthogonal factor matrix of ONMF is naturally sparse. Based on this fact, by introducing sparsity constraints on the orthogonal coefficient matrix, we propose two vector-wise algorithms based on Hierarchical Alternating Least Squares (HALS) and Block Prox-linear (BPL) methods to the approximately sparse orthogonal nonnegative matrix factorization (SONMF). Some global convergence results are established under the mild conditions. Numerical results including synthetic and real-world datasets are given to illustrate that the proposed algorithms compute highly accurate values and perform better than the other testing ONMF methods.

Linked CP tensor decomposition algorithms for shared and individual feature extraction

Tensor decomposition methods are widely used in various areas of science for multilinear feature extraction and dimensionality reduction of multi-way arrays. Linked CANDECOM/PARAFAC (CP) tensor decomposition (LCPTD) can be used for extraction of shared and individual multilinear features from a set of observed multi-way arrays. In many real-world applications, observations are gathered by multiple sensors that might have multiple functionalities. In this study, we propose various algorithmic approaches to estimate shared and individual latent factors in the LCPTD model. Two of them are based on the hierarchical alternating least-squares (HALS) and alternating direction method of multipliers (ADMM) approaches, assuming that both observed arrays can be uploaded to a shared memory. For processing large-scale tensors, we propose a geometry-based computational strategy that aims to distribute computing according to the MapReduce paradigm. The proposed algorithms are validated with synthetically generated and real data. They are applied to feature extraction from the spectrograms of electromyography (EMG) and mechanomyography (MMG) signals that are registered simultaneously by an array of sensors with double functionality. The extracted features are then used for classification of grasping movements for various objects. The HALS-based algorithm is also successfully applied to separation of a moving foreground from a nearly static background in a color video sequence. All the tests demonstrate high efficiency of the proposed approaches, and their usefulness for processing a set of multi-way arrays.

Randomized nonnegative matrix factorization

Nonnegative matrix factorization (NMF) is a powerful tool for data mining. However, the emergence of ‘big data’ has severely challenged our ability to compute this fundamental decomposition using deterministic algorithms. This paper presents a randomized hierarchical alternating least squares (HALS) algorithm to compute the NMF. By deriving a smaller matrix from the nonnegative input data, a more efficient nonnegative decomposition can be computed. Our algorithm scales to big data applications while attaining a near-optimal factorization, i.e., the algorithm scales with the target rank of the data rather than the ambient dimension of measurement space. The proposed algorithm is evaluated using synthetic and real world data and shows substantial speedups compared to deterministic HALS.

Spatially-weighted nonnegative matrix factorization with application to temporal psychovisual modulation

Nonnegative Matrix Factorization (NMF), which decomposes a target matrix into the product of two matrices with nonnegative elements, has been widely used in various fields of signal processing. In visual signal processing, the spatially nonuniformed distribution of perceptually meaningful information in image and video frames calls for a kind of Spatially-Weighted NMF (swNMF) that applies location dependent weights into the decomposition problem. In this paper we introduce swNMF solution based on the hierarchical alternating least squares (HALS) approach. Then we exemplify its application to a new information display diagram named temporal psychovisual modulation (TPVM) with comparison with traditional HALS method and baseline algorithm of multiplicative update (MU).

Factorization Algorithms for Temporal Psychovisual Modulation Display

Temporal psychovisual modulation (TPVM) is a new information display technology which aims to generate multiple visual percepts for different viewers on a single display simultaneously. In a TPVM system, the viewers wearing different active liquid crystal (LC) glasses with varying transparency levels can see different images (called personal views). The viewers without LC glasses can also see a semantically meaningful image (called shared view). The display frames and weights for the LC glasses in the TPVM system can be computed through nonnegative matrix factorization (NMF) with three additional constrains: 1) the values of images and modulation weights should have upper bound (i.e., limited luminance of the display and transparency level of the LC); 2) the shared view without using viewing devices should be considered (i.e., the sum of all basis images should be a meaningful image); and 3) the sparsity of modulation weights should be considered due to the material property of LC. In this paper, we proposed to solve the constrained NMF problem by a modified version of hierarchical alternating least squares (HALS) algorithms. Through experiments, we analyze the choice of parameters in the setup of TPVM system. This work serves as a guideline for practical implementation of TPVM display system.

Two Efficient Algorithms for Approximately Orthogonal Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) with orthogonality constraints is quite important due to its close relation with the K-means clustering. While existing algorithms for orthogonal NMF impose strict orthogonality constraints, in this letter we propose a penalty method with the aim of performing approximately orthogonal NMF, together with two efficient algorithms respectively based on the Hierarchical Alternating Least Squares (HALS) and the Accelerated Proximate Gradient (APG) approaches. Experimental evidence was provided to show their high efficiency and flexibility by using synthetic and real-world data.

Identifying the muscle synergy pattern during human grasping

In this work, a methodology has been proposed and evaluated for identification the muscle synergy patterns during human grasping.The proposed approach is based on decomposition analysis of involved muscle activation profiles utilizing the Hierarchical Alternating Least Squares (HALS) algorithm. The surface EMG signals of Flexor Digital Superfacialis and Flexor Pollicis Longus muscles were recorded during grasping an cylindrical object. EMG signals were full-wave rectified and smoothed through a low pass filter. Then the HALS algorithm was utilized for decomposition of muscle activation profiles. The HALS algorithm can be efficiently used instead of NNMF (non-negative matrix factorization) method. The HALS method not only provides a very good convergence property but also there is not the non-negativity constraint for the decomposed factors. The results of evaluations are interesting and promising.

Evaluation of Different Algorithms of Nonnegative Matrix Factorization in Temporal Psychovisual Modulation

Temporal psychovisual modulation (TPVM) is a newly proposed information display paradigm, which can be implemented by nonnegative matrix factorization (NMF) with additional upper bound constraints on the variables. In this paper, we study all the state-of-the-art algorithms in NMF, extend them to incorporate the upper bounds and discuss their potential use in TPVM. By comparing all the NMF algorithms with their extended versions, we find that: 1) the factorization error of the truncated alternating least squares algorithm always fluctuates throughout the iterations, 2) the alternating nonnegative least squares based algorithms may slow down dramatically under the upper bound constraints, and 3) the hierarchical alternating least squares (HALS) algorithm converges the fastest and its final factorization error is often the smallest among all the algorithms. Based on the experimental results of the HALS, we propose a guideline of determining the parameter setting of TPVM, that is, the number of viewers to support and the scaling factor for adjusting the light intensity of the images formed by TPVM. This paper will facilitate the applications of TPVM.

HALS-based NMF with flexible constraints for hyperspectral unmixing

In this article, the hyperspectral unmixing problem is solved with the nonnegative matrix factorization (NMF) algorithm. The regularized criterion is minimized with a hierarchical alternating least squares (HALS) scheme. Under the HALS framework, four constraints are introduced to improve the unmixing accuracy, including the sum-to-unity constraint, the constraints for minimum spectral dispersion and maximum spatial dispersion, and the minimum volume constraint. The derived algorithm is called F-NMF, for NMF with flexible constraints. We experimentally compare F-NMF with different constraints and combined ones. We test the sensitivity and robustness of F-NMF to many parameters such as the purity level of endmembers, the number of endmembers and pixels, the SNR, the sparsity level of abundances, and the overestimation of endmembers. The proposed algorithm improves the results estimated by vertex component analysis. A comparative analysis on real data is included. The unmixing results given by a geometrical method, the simplex identification via split augmented Lagrangian and the F-NMF algorithms with combined constraints are compared, which shows the relative stability of F-NMF.

Extended HALS algorithm for nonnegative Tucker decomposition and its applications for multiway analysis and classification

Analysis of high dimensional data in modern applications, such as neuroscience, text mining, spectral analysis, chemometrices naturally requires tensor decomposition methods. The Tucker decompositions allow us to extract hidden factors (component matrices) with different dimension in each mode, and investigate interactions among various modalities. The alternating least squares (ALS) algorithms have been confirmed effective and efficient in most of tensor decompositions, especially Tucker with orthogonality constraints. However, for nonnegative Tucker decomposition (NTD), standard ALS algorithms suffer from unstable convergence properties, demand high computational cost for large scale problems due to matrix inverse, and often return suboptimal solutions. Moreover they are quite sensitive with respect to noise, and can be relatively slow in the special case when data are nearly collinear. In this paper, we propose a new algorithm for nonnegative Tucker decomposition based on constrained minimization of a set of local cost functions and hierarchical alternating least squares (HALS). The developed NTD-HALS algorithm sequentially updates components, hence avoids matrix inverse, and is suitable for large-scale problems. The proposed algorithm is also regularized with additional constraint terms such as sparseness, orthogonality, smoothness, and especially discriminant. Extensive experiments confirm the validity and higher performance of the developed algorithm in comparison with other existing algorithms.

Extraction of functional domains in optical imaging data using regularized nonnegative matrix factorization

Event Abstract Back to Event Extraction of functional domains in optical imaging data using regularized nonnegative matrix factorization Jan Soelter1*, Jan Schumacher2, Antonia Strutz3, Silke Sachse3, Hartwig Spors2 and Michael Schmuker1, 4 1 Freie Universität Berlin, Institute of Biology, Germany 2 Max Planck Institute of Biophysics, Department of Molecular Neurogenetics, Germany 3 Max Planck Institute for Chemical Ecology, Department of Evolutionary Neuroethology, Germany 4 Bernstein Center for Computational Neuroscience Berlin, Germany Functional optical imaging is widely used to measure neuronal activity in the living brain. In many cases, the neuronal activity exhibits spatial structure which reflects a distinct physiological organization of the tissue. However, exact data on that organization is often not available in functional imaging. This poses the challenge of automatically estimating distinct spatial domains and their corresponding stimulus response based on functional data alone. The decomposition of observations into their generating sources is an instance of the Blind Source Separation problem. Previous approaches to this problem were based on Principal and Independent Component Analysis. However, both methods reach their limits if the number of sources lies within the range of the number of images, or even exceeds them. In this contribution, we introduce a regularized non-negative matrix factorization algorithm to overcome this problem. Using a toy dataset, we compare its performance to established approaches. In addition, we demonstrate its applicability in two natural datasets where the non-negativity condition is met: to discover (a) glomeruli, the functional domains of the olfactory bulb in mice, in intrinsic optical images and (b) inhibitory olfactory projection neurons in calcium imaging data of the lateral horn of Drosophila. To cope with the low observation to source ratio, we restricted the generative model in the introduced algorithm in several ways. The non-negative matrix factorization itself imposed the constraint of only positive contributions of the sources. Moreover, a sparseness constraint enforced localized sources. Additionally, we introduced a new modification in the Hierarchical Alternating Least Squares (HALS) algorithm to promote spatially decorrelated sources. Acknowledgements Funded by DFG grant no. SCHM2474/1-1 to M.S. (SPP 1392, principal source of funding), BMBF grant no. 01GQ1001D (BCCN Berlin) to M.S., BMBF grant no. 01GQ0807 to S.S., DFG grant no. SP 1134/2-1to H.S. (SPP 1392). Keywords: calcium imaging, data analysis and machine learning, Drosophila lateral horn, Intrinsic signal imaging, mouse olfactory bulb Conference: BC11 : Computational Neuroscience & Neurotechnology Bernstein Conference & Neurex Annual Meeting 2011, Freiburg, Germany, 4 Oct - 6 Oct, 2011. Presentation Type: Poster Topic: data analysis and machine learning (please use "data analysis and machine learning" as keyword) Citation: Soelter J, Schumacher J, Strutz A, Sachse S, Spors H and Schmuker M (2011). Extraction of functional domains in optical imaging data using regularized nonnegative matrix factorization. Front. Comput. Neurosci. Conference Abstract: BC11 : Computational Neuroscience & Neurotechnology Bernstein Conference & Neurex Annual Meeting 2011. doi: 10.3389/conf.fncom.2011.53.00077 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 23 Aug 2011; Published Online: 04 Oct 2011. * Correspondence: Mr. Jan Soelter, Freie Universität Berlin, Institute of Biology, Berlin, 14195, Germany, j.soelter@fu-berlin.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Jan Soelter Jan Schumacher Antonia Strutz Silke Sachse Hartwig Spors Michael Schmuker Google Jan Soelter Jan Schumacher Antonia Strutz Silke Sachse Hartwig Spors Michael Schmuker Google Scholar Jan Soelter Jan Schumacher Antonia Strutz Silke Sachse Hartwig Spors Michael Schmuker PubMed Jan Soelter Jan Schumacher Antonia Strutz Silke Sachse Hartwig Spors Michael Schmuker Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations

Nonnegative matrix factorization (NMF) and its extensions such as Nonnegative Tensor Factorization (NTF) have become prominent techniques for blind sources separation (BSS), analysis of image databases, data mining and other information retrieval and clustering applications. In this paper we propose a family of efficient algorithms for NMF/NTF, as well as sparse nonnegative coding and representation, that has many potential applications in computational neuroscience, multi-sensory processing, compressed sensing and multidimensional data analysis. We have developed a class of optimized local algorithms which are referred to as Hierarchical Alternating Least Squares (HALS) algorithms. For these purposes, we have performed sequential constrained minimization on a set of squared Euclidean distances. We then extend this approach to robust cost functions using the alpha and beta divergences and derive flexible update rules. Our algorithms are locally stable and work well for NMF-based blind source separation (BSS) not only for the over-determined case but also for an under-determined (over-complete) case (i.e., for a system which has less sensors than sources) if data are sufficiently sparse. The NMF learning rules are extended and generalized for N-th order nonnegative tensor factorization (NTF). Moreover, these algorithms can be tuned to different noise statistics by adjusting a single parameter. Extensive experimental results confirm the accuracy and computational performance of the developed algorithms, especially, with usage of multi-layer hierarchical NMF approach [3].