Deep Non-negative Matrix Factorization Research Articles

Deep bidirectional hierarchical matrix factorization model for hyperspectral unmixing

Existing deep nonnegative matrix factorization-based approaches treat shallow and deep layers equally or with similar strategies, neglecting the heterogeneous physical structures between the shallow and progressively deeper layers, thus failing to explore the latent different sub-manifold structures in the hyperspectral image. In this paper, we propose a deep nonnegative matrix factorization model with bidirectional constraints to achieve hyperspectral unmixing. The sub-manifold structures in hyperspectral image are fully exploited by filtering and penalizing the shallow abundance layer with a denoised regularizer and a manifold regularizer. In contrast to the shallow abundance layer, the remaining layers are constrained by an extremely common regularizer to avoid over-denoising and maintain fidelity. In this way, the fine cues between different substances are exploited to a large extent. Additionally, the overall reconstruction error can be well controlled because the performance of the designed feedback mechanism can be fine-tuned by the inverse hierarchical constraints. Finally, we employ Nesterov's optimal gradient method to solve the optimization problem effectively. Experiment results are conducted on both synthetic datasets and real datasets, and all results show that the proposed method is superior to recent canonical unmixing methods.

Deep Nonnegative Matrix Factorization With Beta Divergences.

Deep nonnegative matrix factorization (deep NMF) has recently emerged as a valuable technique for extracting multiple layers of features across different scales. However, all existing deep NMF models and algorithms have primarily centered their evaluation on the least squares error, which may not be the most appropriate metric for assessing the quality of approximations on diverse data sets. For instance, when dealing with data types such as audio signals and documents, it is widely acknowledged that ß-divergences offer a more suitable alternative. In this article, we develop new models and algorithms for deep NMF using some ß-divergences, with a focus on the Kullback-Leibler divergence. Subsequently, we apply these techniques to the extraction of facial features, the identification of topics within document collections, and the identification of materials within hyperspectral images.

Deep embedding based tensor incomplete multi-view clustering

The majority of multi-view data are extracted from real life, which often lose information in some views. To solve this problem, existing incomplete multi-view clustering algorithms explore the valuable information from incomplete data while populating the missing information. Nevertheless, they suffer from the following two limitations: 1) The non-linear relationships of high-dimensional available data are not considered. 2) Although some methods utilize information from different views to fill in missing information, they still cannot precisely explore the missing information of each view. To this end, this article proposes a novel one-step incomplete multi-view framework, referred to as deep embedding based tensor incomplete multi-view clustering (DETIMC). Concretely, in this framework, the high-dimensional available data are projected into the low-dimensional embedding space by deep non-negative matrix factorization (NMF), which can obtain a clean space while capturing the complex non-linear relationship of available data. Moreover, a novel double-enhanced missing-view inferring strategy is developed, in which the weighted higher-order information of different views and the clustering structure information are simultaneously exploited. Comprehensive experiments on several benchmark datasets exhibit the superiority of DETIMC over conventional and state-of-the-art algorithms.

Comprehensive Multiview Representation Learning via Deep Autoencoder-Like Nonnegative Matrix Factorization.

Learning a comprehensive representation from multiview data is crucial in many real-world applications. Multiview representation learning (MRL) based on nonnegative matrix factorization (NMF) has been widely adopted by projecting high-dimensional space into a lower order dimensional space with great interpretability. However, most prior NMF-based MRL techniques are shallow models that ignore hierarchical information. Although deep matrix factorization (DMF)-based methods have been proposed recently, most of them only focus on the consistency of multiple views and have cumbersome clustering steps. To address the above issues, in this article, we propose a novel model termed deep autoencoder-like NMF for MRL (DANMF-MRL), which obtains the representation matrix through the deep encoding stage and decodes it back to the original data. In this way, through a DANMF-based framework, we can simultaneously consider the multiview consistency and complementarity, allowing for a more comprehensive representation. We further propose a one-step DANMF-MRL, which learns the latent representation and final clustering labels matrix in a unified framework. In this approach, the two steps can negotiate with each other to fully exploit the latent clustering structure, avoid previous tedious clustering steps, and achieve optimal clustering performance. Furthermore, two efficient iterative optimization algorithms are developed to solve the proposed models both with theoretical convergence analysis. Extensive experiments on five benchmark datasets demonstrate the superiority of our approaches against other state-of-the-art MRL methods.

Cluster structure augmented deep nonnegative matrix factorization with low-rank tensor learning

Clustering approaches based on deep nonnegative matrix factorization have received significant attention because it can learn hierarchical semantics from data in a layer-wise manner. However, latent representation learning and clustering are two separate processes in these models, leading to information loss and sub-optimal clusterings. Furthermore, if data contain complex structure or noise, the pre-defined graph in data space will not be precise enough for graph-regularized deep nonnegative matrix factorization models. Moreover, most of them maintain the similarity graph at the top layer, neglecting the information covered by other layers. In this paper, Cluster Structure Augmented Deep Nonnegative Matrix Factorization with Low-rank Tensor Learning is proposed. First, as representations at different layers cover different data abstractions, a learning mechanism is leveraged to generate multiple local similarity graphs based on representations from different layers, thus maintaining diversity across layers. Then, a low-rank third-order tensor is constructed by stacking these graphs to capture the high-order consistency across layers, thus capturing the intrinsic property of data. Third, clustering is integrated into the framework, allowing the cluster structure to facilitate multi-layer matrix factorization and graph structure learning, thereby generating discriminative representations. Experimental results on several datasets demonstrate that the proposed model outperforms several state-of-the-art methods.

A Comprehensive Survey on Community Detection With Deep Learning.

Detecting a community in a network is a matter of discerning the distinct features and connections of a group of members that are different from those in other communities. The ability to do this is of great significance in network analysis. However, beyond the classic spectral clustering and statistical inference methods, there have been significant developments with deep learning techniques for community detection in recent years--particularly when it comes to handling high-dimensional network data. Hence, a comprehensive review of the latest progress in community detection through deep learning is timely. To frame the survey, we have devised a new taxonomy covering different state-of-the-art methods, including deep learning models based on deep neural networks (DNNs), deep nonnegative matrix factorization, and deep sparse filtering. The main category, i.e., DNNs, is further divided into convolutional networks, graph attention networks, generative adversarial networks, and autoencoders. The popular benchmark datasets, evaluation metrics, and open-source implementations to address experimentation settings are also summarized. This is followed by a discussion on the practical applications of community detection in various domains. The survey concludes with suggestions of challenging topics that would make for fruitful future research directions in this fast-growing deep learning field.

Deep Nonnegative Matrix Factorization with Joint Global and Local Structure Preservation

Deep Non-Negative Matrix Factorization (DNMF) methods provide an efficient low-dimensional representation of given data through their layered architecture. A limitation of such methods is that they cannot effectively preserve the local and global geometric structures of the data in each layer. Consequently, a significant amount of the geometrical information within the data, present in each layer of the employed deep framework, can be overlooked by the model. This can lead to an information loss and a subsequent drop in performance. In this paper, we propose a novel deep non-negative matrix factorization method, Deep Non-Negative Matrix Factorization with Joint Global and Local Structure Preservation (dubbed Dn2MFGL), that ensures the preservation of both global and local structures within the data space. Dn2MFGL performs representation learning through a sequential embedding procedure which involves both the global data structure by accounting for the data variance, and the local data relationships by utilizing information from neighboring data points. Moreover, a regularization term that promotes sparsity by utilizing the concept of the inner product is applied to the matrices representing the lower dimensions. This aims to retain the fundamental data structure while discarding less crucial features. Simultaneously, the residual matrix of Dn2MFGL is subjected to the L2,1 norm, which ensures the robustness of the model against noisy data samples. An effective and multiplicative updating process also facilitates Dn2MFGL in solving the employed objective function. The clustering performance of the proposed deep NMF method is explored across various benchmarks of face datasets. The results point to Dn2MFGL outperforming several existing classical and state-of-the-art NMF methods. The source code is available at https://github.com/FaridSaberi/Dn2MFGO.git.

A Multi-Kernel-Based Multi-View Deep Non-Negative Matrix Factorization for Enhanced Healthcare Data Clustering

A Multi-Kernel-Based Multi-View Deep Non-Negative Matrix Factorization for Enhanced Healthcare Data Clustering

Robust Hypergraph Regularized Deep Non-Negative Matrix Factorization for Multi-View Clustering

Robust Hypergraph Regularized Deep Non-Negative Matrix Factorization for Multi-View Clustering

Deep non-negative matrix factorization with edge generator for link prediction in complex networks

Deep non-negative matrix factorization with edge generator for link prediction in complex networks

Deep asymmetric nonnegative matrix factorization for graph clustering

Graph clustering is a fundamental technique in machine learning that has widespread applications in various fields. Deep Nonnegative Matrix Factorization (DNMF) was recently emerged to cope with the extraction of several layers of features, and it has been demonstrated to achieve remarkable results on unsupervised tasks. While DNMF has been applied for analyzing graphs, the effectiveness of the current DNMF approaches for graph clustering is generally unsatisfactory: these methods are intrinsically data representation models, and their objective functions do not capture cluster structures, also ignores direction which is crucial in the directed graph clustering problems. To overcome these downsides, this paper proposes a graph-specific DNMF model based on the Asymmetric NMF which can handle undirected and directed graphs. Inspired by hierarchical graph clustering and graph summarization approaches, the Deep Asymmetric Nonnegative Matrix Factorization (DAsNMF) is introduced for the directed graph clustering problem. In a pseudo-hierarchical clustering setting, DAsNMF decomposes the input graph to extract low-level to high-level node representations and graph representations (summarized graphs). In addition, the asymmetric cosine and PageRank-based similarities are imposed on the proposed model to preserve the local and global graph structures. The learning process is formulated as a unified optimization problem to jointly train representation learning model and clustering model. The extensive experimental studies validate the effectiveness of the proposed method on directed graphs.

Learning deep representation and discriminative features for clustering of multi-layer networks

The multi-layer network consists of the interactions between different layers, where each layer of the network is depicted as a graph, providing a comprehensive way to model the underlying complex systems. The layer-specific modules of multi-layer networks are critical to understanding the structure and function of the system. However, existing methods fail to characterize and balance the connectivity and specificity of layer-specific modules in networks because of the complicated inter- and intra-coupling of various layers. To address the above issues, a joint learning graph clustering algorithm (DRDF) for detecting layer-specific modules in multi-layer networks is proposed, which simultaneously learns the deep representation and discriminative features. Specifically, DRDF learns the deep representation with deep nonnegative matrix factorization, where the high-order topology of the multi-layer network is gradually and precisely characterized. Moreover, it addresses the specificity of modules with discriminative feature learning, where the intra-class compactness and inter-class separation of pseudo-labels of clusters are explored as self-supervised information, thereby providing a more accurate method to explicitly model the specificity of the multi-layer network. Finally, DRDF balances the connectivity and specificity of layer-specific modules with joint learning, where the overall objective of the graph clustering algorithm and optimization rules are derived. The experiments on ten multi-layer networks showed that DRDF not only outperforms eight baselines on graph clustering but also enhances the robustness of algorithms.

Community detection in attributed networks via adaptive deep nonnegative matrix factorization

Community detection in attributed networks via adaptive deep nonnegative matrix factorization

Deep low-rank tensor embedding for multi-view subspace clustering

Despite the good clustering performance, most of existing multi-view subspace clustering methods fail to consider the non-linear relationships of high-dimensional data, higher-order correlations of different views, and optimal local geometric structure of the latent embedding space simultaneously. To this end, we put forward a novel multi-view subspace clustering model, named deep low-rank tensor embedding (DLTE). In DLTE, we project the high-dimensional data into the low-dimensional embedding space by utilizing deep non-negative matrix factorization (NMF), which can efficiently capture the complex non-linear relationship of data. Moreover, in the deep embedding space, we utilize the weighted low-rank tensor constraint to capture the global structure and higher-order correlations of different views while considering the contributions of different views. Additionally, we introduce an optimal graph Laplacian to learn a more reasonable local geometric structure of data in the deep embedding space. At last, comprehensive experimental results on eleven datasets indicate the superiority and effectiveness of the proposed DLTE method.

Deep Autoencoder-like non-negative matrix factorization with graph regularized for link prediction in dynamic networks

Link prediction in dynamic(temporal) networks refers to predicting future edges by analyzing the available network information. Among the existing temporal link prediction approaches, non-negative matrix factorization(NMF) is a kind of competitive algorithm and has attracted extensive attention. However, traditional NMF-based prediction methods are shallow methods and cannot fully mine the dynamic network, which may lead to a decrease in performance of algorithms. To overcome these shortcomings, inspired by deep Autoencoder, we propose two novel deep Autoencoder-like NMF with graph regularized prediction methods for dynamic networks. By fusing encoder component with deep structure into deep NMF model, our algorithms can sufficiently exploit the complex hierarchical information hidden in dynamic networks. To further extract the abundant information hidden in dynamic networks, graph regularization and PageRank are utilized to exploit the local and global topology information of each snapshot, respectively. By jointly optimizing them in deep Autoencoder-like NMF model, our model is able to preserve the local and global information hidden in dynamic networks, simultaneously. Moreover, an effective alternating iterative method with convergence guarantee is developed for minimizing the established model. Finally, we test our proposed prediction methods on several synthetic and real world datasets to demonstrate that our approaches outperform the state-of-the-art prediction approaches.

Nonnegative Matrix Factorization: A Review

Recent developments in Non-negative Matrix Factorization (NMF) have focused on addressing several challenges and advancing its applicability. New algorithmic variations, such as robust NMF, deep NMF, and graph-regularized NMF, have emerged to improve NMF's performance in various domains. These developments aim to enhance the interpretability, scalability, and robustness of NMF-based solutions. NMF is now widely used in audio source separation, text mining, recommendation systems, and image processing. However, NMF still faces challenges, including sensitivity to initialization, the determination of the appropriate rank, and computational complexity. Overlapping sources in audio and data sparsity in some applications remain challenging issues. Additionally, ensuring the consistency and stability of NMF results in noisy environments is a subject of ongoing research. The quest for more efficient and scalable NMF algorithms continues, especially for handling large datasets. While NMF has made significant strides in recent years, addressing these challenges is crucial for unlocking its full potential in diverse data analysis and source separation tasks.

Hyperspectral Unmixing Using Robust Deep Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) and its numerous variants have been extensively studied and used in hyperspectral unmixing (HU). With the aid of the designed deep structure, deep NMF-based methods demonstrate advantages in exploring the hierarchical features of complex data. However, a noise corruption problem commonly exists in hyperspectral data and severely degrades the unmixing performance of deep NMF-based methods when applied to HU. In this study, we propose an ℓ2,1 norm-based robust deep nonnegative matrix factorization (ℓ2,1-RDNMF) for HU, which incorporates an ℓ2,1 norm into the two stages of the deep structure to achieve robustness. The multiplicative updating rules of ℓ2,1-RDNMF are efficiently learned and provided. The efficiency of the presented method is verified in experiments using both synthetic and genuine data.

Deep Nonnegative Matrix Factorization Using a Variational Autoencoder With Application to Single-Cell RNA Sequencing Data.

Single-cell RNA sequencing is used to analyze the gene expression data of individual cells, thereby adding to existing knowledge of biological phenomena. Accordingly, this technology is widely used in numerous biomedical studies. Recently, the variational autoencoder has emerged and has been adopted for the analysis of single-cell data owing to its high capacity to manage large-scale data. Many different variants of the variational autoencoder have been applied, and have yielded superior results. However, because it is nonlinear, the model does not provide parameters that can be used to explain the underlying biological patterns. In this paper, we propose an interpretable nonnegative matrix factorization method that decomposes parameters into those shared across cells and those that are cell-specific. Effective nonlinear dimension reduction was achieved via a variational autoencoder applied to the cell-specific parameters. In addition to achieving nonlinear dimension reduction, our model could estimate the cell-type-specific gene expression. To improve the estimation accuracy, we introduced log-regularization, which reflects the single-cell property. Overall, our approach displayed excellent performance in a simulation study and in real data analyses, while maintaining good biological interpretability.

Elastic adversarial deep nonnegative matrix factorization for matrix completion

In recent years, matrix completion has attracted a lot of attention. Conventional matrix completions are shallow and ineffective when dealing with complex data structures. Researchers have recently tried to incorporate deep structures into matrix completion; however, considerable challenges still exist. Most matrix completion methods may fail to work effectively in the presence of limited observations. To enhance the generalization of the reconstruction, adversarial methods are proposed that attempt to fool models by providing deceptive input. The aim is to develop an adversarial training algorithm that resists attacks in a deep model, thus at the same time leading to enhancing the generalization. Therefore, in this paper, we propose an elastic adversarial training to design a high-capacity Deep Nonnegative Matrix Factorization (DNMF) model with proper discovery latent structure of the data and enhanced generalization abilities. In other words, the challenges mentioned above are addressed by perturbing the inputs in DNMF with an elastic loss which is intercalated and adapted between Frobenius and ℓ2,1 norms. This model not only dispenses with adversarial DNMF generation but also is robust towards a mixture of multiple attacks to attain improved accuracy. Extensive simulations show that the proposed approach outperforms state-of-the-art methods.

Cross-domain residual deep NMF for transfer learning between different hyperspectral image scenes

Hyperspectral image (HSI) classification has long been a hot research topic. Most previous researches concentrate on the classification task of a single HSI scene, called single-scene classification. This research focuses on two closely related HSI scenes (called source and target scenes, respectively), and the problem is named cross-scene classification. This paper aims to explore the shared feature sub-space between two HSI scenes. A transfer learning algorithm called cross-domain residual deep nonnegative matrix factorization (CDRDNMF) is proposed. CDRDNMF is a multi-layer architecture consisting of dual-dictionary nonnegative matrix factorization (DDNMF) layers. In each layer, DDNMF is performed on source and target features for domain-invariant feature extraction. Then a data recovery process is completed, and the residual components from the recovery are passed to the next layer after activation. With such a multi-layer architecture, CDRDNMF delivers knowledge transfer and multi-scale feature extraction tasks. The experimental results prove the excellent performance of CDRDNMF on cross-scene classification.