Sparse Matrix Factorization Research Articles

Collective sparse symmetric non-negative matrix factorization for identifying overlapping communities in resting-state brain functional networks

Resting-state functional magnetic resonance imaging (rs-fMRI) provides a valuable tool to study spontaneous brain activity. Using rs-fMRI, researchers have extensively studied the organization of the brain functional network and found several consistent communities consisting of functionally connected but spatially separated brain regions across subjects. However, increasing evidence in many disciplines has shown that most realistic complex networks have overlapping community structure. Only recently has the overlapping community structure drawn increasing interest in the domain of brain network studies. Another issue is that the inter-subject variability is often not directly reflected in the process of community detection at the group level. In this paper, we propose a novel method called collective sparse symmetric non-negative matrix factorization (cssNMF) to address these issues. The cssNMF approach identifies the group-level overlapping communities across subjects and in the meantime preserves the information of individual variation in brain functional network organization. To comprehensively validate cssNMF, a simulated fMRI dataset with ground-truth, a real rs-fMRI dataset with two repeated sessions and another different real rs-fMRI dataset have been used for performance comparison in the experiment. Experimental results show that the proposed cssNMF method accurately and stably identifies group-level overlapping communities across subjects as well as individual differences in network organization with neurophysiologically meaningful interpretations. This research extends our understanding of the common underlying community structures and individual differences in community strengths in brain functional network organization.

Adaptive total-variation for non-negative matrix factorization on manifold

Non-negative matrix factorization (NMF) has been widely applied in information retrieval and computer vision. However, its performance has been restricted due to its limited tolerance to data noise, as well as its inflexibility in setting regularization parameters. In this paper, we propose a novel sparse matrix factorization method for data representation to solve these problems, termed Adaptive Total-Variation Constrained based Non-Negative Matrix Factorization on Manifold (ATV-NMF). The proposed ATV can adaptively choose the anisotropic smoothing scheme based on the gradient information of data to denoise or preserve feature details by incorporating adaptive total variation into the factorization process. Notably, the manifold graph regularization is also incorporated into NMF, which can discover intrinsic geometrical structure of data to enhance the discriminability. Experimental results demonstrate that the proposed method is very effective for data clustering in comparison to the state-of-the-art algorithms on several standard benchmarks.

Algorithm 980

Sparse matrix factorization involves a mix of regular and irregular computation, which is a particular challenge when trying to obtain high-performance on the highly parallel general-purpose computing cores available on graphics processing units (GPUs). We present a sparse multifrontal QR factorization method that meets this challenge and is significantly faster than a highly optimized method on a multicore CPU. Our method factorizes many frontal matrices in parallel and keeps all the data transmitted between frontal matrices on the GPU. A novel bucket scheduler algorithm extends the communication-avoiding QR factorization for dense matrices by exploiting more parallelism and by exploiting the staircase form present in the frontal matrices of a sparse multifrontal method.

Semiring Rank Matrix Factorization

Rank data, in which each row is a complete or partial ranking of available items (columns), is ubiquitous. Among others, it can be used to represent preferences of users, levels of gene expression, and outcomes of sports events. It can have many types of patterns, among which consistent rankings of a subset of the items in multiple rows, and multiple rows that rank the same subset of the items highly. In this article, we show that the problems of finding such patterns can be formulated within a single generic framework that is based on the concept of semiring matrix factorization. In this framework, we employ the max-product semiring rather than the plus-product semiring common in traditional linear algebra. We apply this semiring matrix factorization framework on two tasks: sparse rank matrix factorization and rank matrix tiling. Experiments on both synthetic and real world datasets show that the framework is capable of discovering different types of structure as well as obtaining high quality solutions.

Spike and slab biclustering

Biclustering refers to the problem of simultaneously clustering the rows and columns of a given data matrix, with the goal of obtaining submatrices where the selected rows present a coherent behaviour in the selected columns, and vice-versa. To face this intrinsically difficult problem, we propose a novel generative model, where biclustering is approached from a sparse low-rank matrix factorization perspective. The main idea is to design a probabilistic model describing the factorization of a given data matrix in two other matrices, from which information about rows and columns belonging to the sought for biclusters can be obtained. One crucial ingredient in the proposed model is the use of a spike and slab sparsity-inducing prior, thus we term the approach spike and slab biclustering (SSBi). To estimate the parameters of the SSBi model, we propose an expectation-maximization (EM) algorithm, termed SSBiEM, which solves a low-rank factorization problem at each iteration, using a recently proposed augmented Lagrangian algorithm. Experiments with both synthetic and real data show that the SSBi approach compares favorably with the state-of-the-art.

New speedup algorithms for nonlinear dynamic time history analysis of supertall building structures under strong earthquakes

SummarySome speedup algorithms oriented to improving computational efficiency of nonlinear dynamic time history analysis of supertall buildings structures excited by strong earthquakes are proposed. In order to minimize Jacobian factorizations, the inexact Newton algorithm combined with the sparse Cholesky matrix factorization (INC) is suggested with an unbalance‐independent relation for the determination of a forcing term in the INC. Further, some shared memory parallel computing techniques are incorporated into the state transformation procedures developed previously (parallel state transformation procedure) and the matrix factorization (parallelized factorization) for full utilization of the resources of an ordinary personal computer. All the algorithms are integrated in a finite element program specialized in time history analysis of aseismic supertall building structures, with some features from OpenSees. Computational efficiency, as well as accuracy, of the proposed speedup algorithms is demonstrated on one 12‐story reinforced concrete frame (K1) and 4 frame‐core‐tube supertall buildings (S1–S4). The results from the demonstration indicate that, as the number of degrees of freedom increases, the proportion of time consumed in Jacobian factorization becomes relatively more dominant. The combination of the INC and the parallelized factorization can achieve better acceleration performance in this case. Factors associated with the forcing term of the INC have much influence on computational efficiency and should be selected based on building scale. Combination of all the algorithms in the biggest model (S4) yields an average of 28.72 of acceleration ratio even with peak ground acceleration equal to 3.2 g. In all cases considered, a desirable agreement in structural response is always reached between conventional time history analysis method and the method using the proposed speedup algorithms.

Large Sparse Cone Non-negative Matrix Factorization for Image Annotation

Image annotation assigns relevant tags to query images based on their semantic contents. Since Non-negative Matrix Factorization (NMF) has the strong ability to learn parts-based representations, recently, a number of algorithms based on NMF have been proposed for image annotation and have achieved good performance. However, most of the efforts have focused on the representations of images and annotations. The properties of the semantic parts have not been well studied. In this article, we revisit the sparseness-constrained NMF (sNMF) proposed by Hoyer [2004]. By endowing the sparseness constraint with a geometric interpretation and sNMF with theoretical analyses of the generalization ability, we show that NMF with such a sparseness constraint has three advantages for image annotation tasks: (i) The sparseness constraint is more ℓ 0 -norm oriented than the ℓ 1 -norm-based sparseness, which significantly enhances the ability of NMF to robustly learn semantic parts. (ii) The sparseness constraint has a large cone interpretation and thus allows the reconstruction error of NMF to be smaller, which means that the learned semantic parts are more powerful to represent images for tagging. (iii) The learned semantic parts are less correlated, which increases the discriminative ability for annotating images. Moreover, we present a new efficient large sparse cone NMF (LsCNMF) algorithm to optimize the sNMF problem by employing the Nesterov’s optimal gradient method. We conducted experiments on the PASCAL VOC07 dataset and demonstrated the effectiveness of LsCNMF for image annotation.

ECG based Atrial Fibrillation detection using Sequency Ordered Complex Hadamard Transform and Hybrid Firefly Algorithm

Electrocardiogram (ECG), a non-invasive diagnostic technique, used for detecting cardiac arrhythmia. From last decade industry dealing with biomedical instrumentation and research, demanding an advancement in its ability to distinguish different cardiac arrhythmia. Atrial Fibrillation (AF) is an irregular rhythm of the human heart. During AF, the atrial moments are quicker than the normal rate. As blood is not completely ejected out of atria, chances for the formation of blood clots in atrium. These abnormalities in the heart can be identified by the changes in the morphology of the ECG. The first step in the detection of AF is preprocessing of ECG, which removes noise using filters. Feature extraction is the next key process in this research. Recent feature extraction methods, such as Auto Regressive (AR) modeling, Magnitude Squared Coherence (MSC) and Wavelet Coherence (WTC) using standard database (MIT-BIH), yielded a lot of features. Many of these features might be insignificant containing some redundant and non-discriminatory features that introduce computational burden and loss of performance. This paper presents fast Conjugate Symmetric Sequency Ordered Complex Hadamard Transform (CS-SCHT) for extracting relevant features from the ECG signal. The sparse matrix factorization method is used for developing fast and efficient CS-SCHT algorithm and its computational performance is examined and compared to that of the HT and NCHT. The applications of the CS-SCHT in the ECG-based AF detection is also discussed. These fast CS-SCHT features are optimized using Hybrid Firefly and Particle Swarm Optimization (FFPSO) to increase the performance of the classifier.

Sparse Matrix Factorizations for Fast Linear Solvers with Application to Laplacian Systems

In solving a linear system with iterative methods, one is usually confronted with the dilemma of having to choose between cheap, inefficient iterates over sparse search directions (e.g., coordinate descent), or expensive iterates in well-chosen search directions (e.g., conjugate gradients). In this paper, we propose to interpolate between these two extremes, and show how to perform cheap iterations along nonsparse search directions, provided that these directions can be extracted from a new kind of sparse factorization. For example, if the search directions are the columns of a hierarchical matrix, then the cost of each iteration is typically logarithmic in the number of variables. Using some graph-theoretical results on low-stretch spanning trees, we deduce as a special case a nearly linear time algorithm to approximate the minimal norm solution of a linear system $Bx=b$ where $B$ is the incidence matrix of a graph. We thereby can connect our results to recently proposed nearly linear time solvers for Laplacian systems, which emerge here as a particular application of our sparse matrix factorization.

Subspace clustering by (<i>k</i>,<i>k</i>)-sparse matrix factorization

High-dimensional data often lie in low-dimensional subspaces instead of the whole space. Subspace clustering is a problem to analyze data that are from multiple low-dimensional subspaces and cluster them into the corresponding subspaces. In this work, we propose a \begin{document}$(k,k)$\end{document} -sparse matrix factorization method for subspace clustering. In this method, data itself is considered as the dictionary, and each data point is represented as a linear combination of the basis of its cluster in the dictionary. Thus, the coefficient matrix is low-rank and sparse. With an appropriate permutation, it is also blockwise with each block corresponding to a cluster. With an assumption that each block is no more than \begin{document}$k$\end{document} -by- \begin{document}$k$\end{document} in matrix recovery, we seek a low-rank and \begin{document}$(k,k)$\end{document} -sparse coefficient matrix, which will be used for the construction of affinity matrix in spectral clustering. The advantage of our proposed method is that we recover a coefficient matrix with \begin{document}$(k,k)$\end{document} -sparse and low-rank simultaneously, which is better fit for subspace clustering. Numerical results illustrate the effectiveness that it is better than SSC and LRR in real-world classification problems such as face clustering and motion segmentation.

Trade the tweet: Social media text mining and sparse matrix factorization for stock market prediction

We investigate the potential use of textual information from user-generated microblogs to predict the stock market. Utilizing the latent space model proposed by Wong et al. (2014), we correlate the movements of both stock prices and social media content. This study differs from models in prior studies in two significant ways: (1) it leverages market information contained in high-volume social media data rather than news articles and (2) it does not evaluate sentiment. We test this model on data spanning from 2011 to 2015 on a majority of stocks listed in the S&P 500 Index and find that our model outperforms a baseline regression. We conclude by providing a trading strategy that produces an attractive annual return and Sharpe ratio.

Prediction of Soil Salinity Using Near-Infrared Reflectance Spectroscopy with Nonnegative Matrix Factorization.

As a key, yet difficult, issue currently in the quantitative remote sensing analysis of soil, the accurate and stable monitoring of soil salinity content (SSC) in situ should be studied and improved. The purpose of this study is to explore the method of fusing spectra outdoors with spectra indoors and improve the estimation precision of SSC based on near-infrared (NIR) reflectance hyper-spectra. First, samples of saline soil from the Yellow River delta of China were collected and analyzed. We measured three groups of sample spectra using a spectrometer: (1) situ-spectra, measured at sampling points in situ; (2) out-spectra, measured outdoors on air-dried samples; and, (3) lab-spectra, measured in a dark laboratory with the above air-dried samples. Second, four algorithms (multiplicative update, alternating least-squares, sparse affine non-negative matrix factorization (NMF), and gradient projection algorithms) of NMF were used to fuse the situ-spectra or out-spectra with the lab-spectra for the calibration of SSC. Finally, estimation models of SSC were built using the multiple linear regression method based on the first derivatives of the un-fused and fused spectra. The results indicate that using the NMF method to fuse the situ-spectra or out-spectra with the lab-spectra can heighten the correlation between SSC and the outdoor spectra in most wavelength ranges and improve the accuracy of the prediction model. The gradient projection algorithm shows the best performance with fewer variables and highest accuracy of the SSC model based on the NIR spectra.

Fault diagnosis of rolling element bearing compound faults based on sparse no-negative matrix factorization-support vector data description

The bispectrum of rolling element bearing compound faults contains abundant fault characteristic information, and how to extract the fault feature effectively is a key problem. The fault diagnosis method of rolling element bearing compound faults based on Sparse No-Negative Matrix Factorization (SNMF)-Support Vector Data Description (SVDD) is proposed in the paper. The figure handling method SNMF is used firstly in fault feature extraction of the bispectrums of rolling element bearing different kinds of compound faults and the sparse coefficient matrices of the bispectrums are obtained. The sparse coefficient matrices are used as training and test input vectors of SVDD. At last, the three kinds of rolling element bearing compound faults (inner race outer race compound faults, outer race rolling element compound faults and inner race outer race rolling element compound faults) are classified correctly. In order to verify the advantages of the proposed method, the diagnosis results of the same three kinds of rolling element bearing compound faults based on No-Negative Matrix Factorization (NMF)-SVDD is used as comparison. The proposed method provides a new idea for fault diagnosis of rolling element bearing compound faults.

Sparse matrix factorization in the implicit finite element method on petascale architecture

The performance of the massively parallel direct multifrontal solver Watson Sparse Matrix Package (WSMP) for solving large sparse systems of linear equations arising in implicit finite element method on unstructured (free) meshes in solid mechanics was evaluated on one of the most powerful supercomputers currently available to the open science community—the sustained petascale high performance computing system of Blue Waters. We have performed full-scale benchmarking tests up to 65,536 cores using assembled global stiffness matrices and load vectors ranging from 11 to 40 million unknowns extracted from “real-world” commercial implicit finite element analysis (FEA) applications. The results show that a direct multifrontal factorization method with a hybrid parallel implementation in WSMP performs exceedingly well on a petascale high-performance computing (HPC) system, and delivers superior factorization time and parallel scalability, thus opening the door for the high fidelity modeling of complex industrial structures and assemblies in real scale.

Bayesian Factorizations of Big Sparse Tensors

It has become routine to collect data that are structured as multiway arrays (tensors). There is an enormous literature on low rank and sparse matrix factorizations, but limited consideration of extensions to the tensor case in statistics. The most common low rank tensor factorization relies on parallel factor analysis (PARAFAC), which expresses a rank k tensor as a sum of rank one tensors. In contingency table applications in which the sample size is massively less than the number of cells in the table, the low rank assumption is not sufficient and PARAFAC has poor performance. We induce an additional layer of dimension reduction by allowing the effective rank to vary across dimensions of the table. Taking a Bayesian approach, we place priors on terms in the factorization and develop an efficient Gibbs sampler for posterior computation. Theory is provided showing posterior concentration rates in high-dimensional settings, and the methods are shown to have excellent performance in simulations and several real data applications.

Learning to Detect Anomalies in Surveillance Video

Detecting anomalies in surveillance videos, that is, finding events or objects with low probability of occurrence, is a practical and challenging research topic in computer vision community. In this paper, we put forward a novel unsupervised learning framework for anomaly detection. At feature level, we propose a Sparse Semi-nonnegative Matrix Factorization (SSMF) to learn local patterns at each pixel, and a Histogram of Nonnegative Coefficients (HNC) can be constructed as local feature which is more expressive than previously used features like Histogram of Oriented Gradients (HOG). At model level, we learn a probability model which takes the spatial and temporal contextual information into consideration. Our framework is totally unsupervised requiring no human-labeled training data. With more expressive features and more complicated model, our framework can accurately detect and localize anomalies in surveillance video. We carried out extensive experiments on several benchmark video datasets for anomaly detection, and the results demonstrate the superiority of our framework to state-of-the-art approaches, validating the effectiveness of our framework.

Speech Enhancement Under Low SNR Conditions Via Noise Estimation Using Sparse and Low-Rank NMF with Kullback–Leibler Divergence

A key stage in speech enhancement is noise estimation which usually requires prior models for speech or noise or both. However, prior models can sometimes be difficult to obtain. In this paper, without any prior knowledge of speech and noise, sparse and low-rank nonnegative matrix factorization (NMF) with Kullback-Leibler divergence is proposed to noise and speech estimation by decomposing the input noisy magnitude spectrogram into a low-rank noise part and a sparse speech-like part. This initial unsupervised speech-noise estimation allows us to set a subsequent regularized version of NMF or convolutional NMF to reconstruct the noise and speech spectrogram, either by estimating a speech dictionary on the fly (categorized as unsupervised approaches) or by using a pre-trained speech dictionary on utterances with disjoint speakers (categorized as semi-supervised approaches). Information fusion was investigated by taking the geometric mean of the outputs from multiple enhancement algorithms. The performance of the algorithms were evaluated on five metrics (PESQ, SDR, SNR, STOI, and OVERALL) by making experiments on TIMIT with 15 noise types. The geometric means of the proposed unsupervised approaches outperformed spectral subtraction (SS), minimum mean square estimation (MMSE) under low input SNR conditions. All the proposed semi-supervised approaches showed superiority over SS and MMSE and also obtained better performance than the state-of-the-art algorithms which utilized a prior noise or speech dictionary under low SNR conditions.

Parallel Scheduling of Task Trees with Limited Memory

This article investigates the execution of tree-shaped task graphs using multiple processors. Each edge of such a tree represents some large data. A task can only be executed if all input and output data fit into memory, and a data can only be removed from memory after the completion of the task that uses it as an input data. Such trees arise in the multifrontal method of sparse matrix factorization. The peak memory needed for the processing of the entire tree depends on the execution order of the tasks. With one processor, the objective of the tree traversal is to minimize the required memory. This problem was well studied, and optimal polynomial algorithms were proposed. Here, we extend the problem by considering multiple processors, which is of obvious interest in the application area of matrix factorization. With multiple processors comes the additional objective to minimize the time needed to traverse the tree—that is, to minimize the makespan. Not surprisingly, this problem proves to be much harder than the sequential one. We study the computational complexity of this problem and provide inapproximability results even for unit weight trees. We design a series of practical heuristics achieving different trade-offs between the minimization of peak memory usage and makespan. Some of these heuristics are able to process a tree while keeping the memory usage under a given memory limit. The different heuristics are evaluated in an extensive experimental evaluation using realistic trees.

$$\mathcal H$$ H -FAINV: hierarchically factored approximate inverse preconditioners

Given a sparse matrix, its LU-factors, inverse and inverse factors typically suffer from substantial fill-in, leading to non-optimal complexities in their computation as well as their storage. In the past, several computationally efficient methods have been developed to compute approximations to these otherwise rather dense matrices. Many of these approaches are based on approximations through sparse matrices, leading to well-known ILU, sparse approximate inverse or factored sparse approximate inverse techniques and their variants. A different approximation approach is based on blockwise low rank approximations and is realized, for example, through hierarchical ($$\mathcal H$$H-) matrices. While $$\mathcal H$$H-inverses and $$\mathcal H$$H-LU factors have been discussed in the literature, this paper will consider the construction of an approximation of the factored inverse through $$\mathcal H$$H-matrices ($$\mathcal H$$H-FAINV). We will describe a blockwise approach that permits to replace (exact) matrix arithmetic through approximate efficient $$\mathcal H$$H-arithmetic. We conclude with numerical results in which we use approximate factored inverses as preconditioners in the iterative solution of the discretized convection---diffusion problem.

Memory-aware tree traversals with pre-assigned tasks

We study the complexity of traversing tree-shaped workflows whose tasks require large I/O files. We target a heterogeneous architecture with two resource types, each with a different memory, such as a multicore node equipped with a dedicated accelerator (FPGA or GPU). The tasks in the workflow are colored according to their type and can be processed if all their input and output files can be stored in the corresponding memory. The amount of used memory of each type at a given execution step strongly depends upon the ordering in which the tasks are executed, and upon when communications between both memories are scheduled. The objective is to determine an efficient traversal that minimizes the maximum amount of memory of each type needed to traverse the whole tree. In this paper, we establish the complexity of this two-memory scheduling problem, and provide inapproximability results. In addition, we design several heuristics, based on both post-order and general traversals, and we evaluate them on a comprehensive set of tree graphs, including random trees as well as assembly trees arising in the context of sparse matrix factorizations.