Deep Non-negative Matrix Factorization Research Articles

Deep embedding based tensor incomplete multi-view clustering

The majority of multi-view data are extracted from real life, which often lose information in some views. To solve this problem, existing incomplete multi-view clustering algorithms explore the valuable information from incomplete data while populating the missing information. Nevertheless, they suffer from the following two limitations: 1) The non-linear relationships of high-dimensional available data are not considered. 2) Although some methods utilize information from different views to fill in missing information, they still cannot precisely explore the missing information of each view. To this end, this article proposes a novel one-step incomplete multi-view framework, referred to as deep embedding based tensor incomplete multi-view clustering (DETIMC). Concretely, in this framework, the high-dimensional available data are projected into the low-dimensional embedding space by deep non-negative matrix factorization (NMF), which can obtain a clean space while capturing the complex non-linear relationship of available data. Moreover, a novel double-enhanced missing-view inferring strategy is developed, in which the weighted higher-order information of different views and the clustering structure information are simultaneously exploited. Comprehensive experiments on several benchmark datasets exhibit the superiority of DETIMC over conventional and state-of-the-art algorithms.

Comprehensive Multiview Representation Learning via Deep Autoencoder-Like Nonnegative Matrix Factorization.

Learning a comprehensive representation from multiview data is crucial in many real-world applications. Multiview representation learning (MRL) based on nonnegative matrix factorization (NMF) has been widely adopted by projecting high-dimensional space into a lower order dimensional space with great interpretability. However, most prior NMF-based MRL techniques are shallow models that ignore hierarchical information. Although deep matrix factorization (DMF)-based methods have been proposed recently, most of them only focus on the consistency of multiple views and have cumbersome clustering steps. To address the above issues, in this article, we propose a novel model termed deep autoencoder-like NMF for MRL (DANMF-MRL), which obtains the representation matrix through the deep encoding stage and decodes it back to the original data. In this way, through a DANMF-based framework, we can simultaneously consider the multiview consistency and complementarity, allowing for a more comprehensive representation. We further propose a one-step DANMF-MRL, which learns the latent representation and final clustering labels matrix in a unified framework. In this approach, the two steps can negotiate with each other to fully exploit the latent clustering structure, avoid previous tedious clustering steps, and achieve optimal clustering performance. Furthermore, two efficient iterative optimization algorithms are developed to solve the proposed models both with theoretical convergence analysis. Extensive experiments on five benchmark datasets demonstrate the superiority of our approaches against other state-of-the-art MRL methods.

Cluster structure augmented deep nonnegative matrix factorization with low-rank tensor learning

Clustering approaches based on deep nonnegative matrix factorization have received significant attention because it can learn hierarchical semantics from data in a layer-wise manner. However, latent representation learning and clustering are two separate processes in these models, leading to information loss and sub-optimal clusterings. Furthermore, if data contain complex structure or noise, the pre-defined graph in data space will not be precise enough for graph-regularized deep nonnegative matrix factorization models. Moreover, most of them maintain the similarity graph at the top layer, neglecting the information covered by other layers. In this paper, Cluster Structure Augmented Deep Nonnegative Matrix Factorization with Low-rank Tensor Learning is proposed. First, as representations at different layers cover different data abstractions, a learning mechanism is leveraged to generate multiple local similarity graphs based on representations from different layers, thus maintaining diversity across layers. Then, a low-rank third-order tensor is constructed by stacking these graphs to capture the high-order consistency across layers, thus capturing the intrinsic property of data. Third, clustering is integrated into the framework, allowing the cluster structure to facilitate multi-layer matrix factorization and graph structure learning, thereby generating discriminative representations. Experimental results on several datasets demonstrate that the proposed model outperforms several state-of-the-art methods.

A Comprehensive Survey on Community Detection With Deep Learning.

Detecting a community in a network is a matter of discerning the distinct features and connections of a group of members that are different from those in other communities. The ability to do this is of great significance in network analysis. However, beyond the classic spectral clustering and statistical inference methods, there have been significant developments with deep learning techniques for community detection in recent years--particularly when it comes to handling high-dimensional network data. Hence, a comprehensive review of the latest progress in community detection through deep learning is timely. To frame the survey, we have devised a new taxonomy covering different state-of-the-art methods, including deep learning models based on deep neural networks (DNNs), deep nonnegative matrix factorization, and deep sparse filtering. The main category, i.e., DNNs, is further divided into convolutional networks, graph attention networks, generative adversarial networks, and autoencoders. The popular benchmark datasets, evaluation metrics, and open-source implementations to address experimentation settings are also summarized. This is followed by a discussion on the practical applications of community detection in various domains. The survey concludes with suggestions of challenging topics that would make for fruitful future research directions in this fast-growing deep learning field.

Deep Nonnegative Matrix Factorization with Joint Global and Local Structure Preservation

Deep Non-Negative Matrix Factorization (DNMF) methods provide an efficient low-dimensional representation of given data through their layered architecture. A limitation of such methods is that they cannot effectively preserve the local and global geometric structures of the data in each layer. Consequently, a significant amount of the geometrical information within the data, present in each layer of the employed deep framework, can be overlooked by the model. This can lead to an information loss and a subsequent drop in performance. In this paper, we propose a novel deep non-negative matrix factorization method, Deep Non-Negative Matrix Factorization with Joint Global and Local Structure Preservation (dubbed Dn2MFGL), that ensures the preservation of both global and local structures within the data space. Dn2MFGL performs representation learning through a sequential embedding procedure which involves both the global data structure by accounting for the data variance, and the local data relationships by utilizing information from neighboring data points. Moreover, a regularization term that promotes sparsity by utilizing the concept of the inner product is applied to the matrices representing the lower dimensions. This aims to retain the fundamental data structure while discarding less crucial features. Simultaneously, the residual matrix of Dn2MFGL is subjected to the L2,1 norm, which ensures the robustness of the model against noisy data samples. An effective and multiplicative updating process also facilitates Dn2MFGL in solving the employed objective function. The clustering performance of the proposed deep NMF method is explored across various benchmarks of face datasets. The results point to Dn2MFGL outperforming several existing classical and state-of-the-art NMF methods. The source code is available at https://github.com/FaridSaberi/Dn2MFGO.git.

Deep non-negative matrix factorization with edge generator for link prediction in complex networks

Deep non-negative matrix factorization with edge generator for link prediction in complex networks

Deep asymmetric nonnegative matrix factorization for graph clustering

Graph clustering is a fundamental technique in machine learning that has widespread applications in various fields. Deep Nonnegative Matrix Factorization (DNMF) was recently emerged to cope with the extraction of several layers of features, and it has been demonstrated to achieve remarkable results on unsupervised tasks. While DNMF has been applied for analyzing graphs, the effectiveness of the current DNMF approaches for graph clustering is generally unsatisfactory: these methods are intrinsically data representation models, and their objective functions do not capture cluster structures, also ignores direction which is crucial in the directed graph clustering problems. To overcome these downsides, this paper proposes a graph-specific DNMF model based on the Asymmetric NMF which can handle undirected and directed graphs. Inspired by hierarchical graph clustering and graph summarization approaches, the Deep Asymmetric Nonnegative Matrix Factorization (DAsNMF) is introduced for the directed graph clustering problem. In a pseudo-hierarchical clustering setting, DAsNMF decomposes the input graph to extract low-level to high-level node representations and graph representations (summarized graphs). In addition, the asymmetric cosine and PageRank-based similarities are imposed on the proposed model to preserve the local and global graph structures. The learning process is formulated as a unified optimization problem to jointly train representation learning model and clustering model. The extensive experimental studies validate the effectiveness of the proposed method on directed graphs.

Learning deep representation and discriminative features for clustering of multi-layer networks

The multi-layer network consists of the interactions between different layers, where each layer of the network is depicted as a graph, providing a comprehensive way to model the underlying complex systems. The layer-specific modules of multi-layer networks are critical to understanding the structure and function of the system. However, existing methods fail to characterize and balance the connectivity and specificity of layer-specific modules in networks because of the complicated inter- and intra-coupling of various layers. To address the above issues, a joint learning graph clustering algorithm (DRDF) for detecting layer-specific modules in multi-layer networks is proposed, which simultaneously learns the deep representation and discriminative features. Specifically, DRDF learns the deep representation with deep nonnegative matrix factorization, where the high-order topology of the multi-layer network is gradually and precisely characterized. Moreover, it addresses the specificity of modules with discriminative feature learning, where the intra-class compactness and inter-class separation of pseudo-labels of clusters are explored as self-supervised information, thereby providing a more accurate method to explicitly model the specificity of the multi-layer network. Finally, DRDF balances the connectivity and specificity of layer-specific modules with joint learning, where the overall objective of the graph clustering algorithm and optimization rules are derived. The experiments on ten multi-layer networks showed that DRDF not only outperforms eight baselines on graph clustering but also enhances the robustness of algorithms.

Community detection in attributed networks via adaptive deep nonnegative matrix factorization

Community detection in attributed networks via adaptive deep nonnegative matrix factorization

Deep low-rank tensor embedding for multi-view subspace clustering

Despite the good clustering performance, most of existing multi-view subspace clustering methods fail to consider the non-linear relationships of high-dimensional data, higher-order correlations of different views, and optimal local geometric structure of the latent embedding space simultaneously. To this end, we put forward a novel multi-view subspace clustering model, named deep low-rank tensor embedding (DLTE). In DLTE, we project the high-dimensional data into the low-dimensional embedding space by utilizing deep non-negative matrix factorization (NMF), which can efficiently capture the complex non-linear relationship of data. Moreover, in the deep embedding space, we utilize the weighted low-rank tensor constraint to capture the global structure and higher-order correlations of different views while considering the contributions of different views. Additionally, we introduce an optimal graph Laplacian to learn a more reasonable local geometric structure of data in the deep embedding space. At last, comprehensive experimental results on eleven datasets indicate the superiority and effectiveness of the proposed DLTE method.

Deep Autoencoder-like non-negative matrix factorization with graph regularized for link prediction in dynamic networks

Link prediction in dynamic(temporal) networks refers to predicting future edges by analyzing the available network information. Among the existing temporal link prediction approaches, non-negative matrix factorization(NMF) is a kind of competitive algorithm and has attracted extensive attention. However, traditional NMF-based prediction methods are shallow methods and cannot fully mine the dynamic network, which may lead to a decrease in performance of algorithms. To overcome these shortcomings, inspired by deep Autoencoder, we propose two novel deep Autoencoder-like NMF with graph regularized prediction methods for dynamic networks. By fusing encoder component with deep structure into deep NMF model, our algorithms can sufficiently exploit the complex hierarchical information hidden in dynamic networks. To further extract the abundant information hidden in dynamic networks, graph regularization and PageRank are utilized to exploit the local and global topology information of each snapshot, respectively. By jointly optimizing them in deep Autoencoder-like NMF model, our model is able to preserve the local and global information hidden in dynamic networks, simultaneously. Moreover, an effective alternating iterative method with convergence guarantee is developed for minimizing the established model. Finally, we test our proposed prediction methods on several synthetic and real world datasets to demonstrate that our approaches outperform the state-of-the-art prediction approaches.

Nonnegative Matrix Factorization: A Review

Recent developments in Non-negative Matrix Factorization (NMF) have focused on addressing several challenges and advancing its applicability. New algorithmic variations, such as robust NMF, deep NMF, and graph-regularized NMF, have emerged to improve NMF's performance in various domains. These developments aim to enhance the interpretability, scalability, and robustness of NMF-based solutions. NMF is now widely used in audio source separation, text mining, recommendation systems, and image processing. However, NMF still faces challenges, including sensitivity to initialization, the determination of the appropriate rank, and computational complexity. Overlapping sources in audio and data sparsity in some applications remain challenging issues. Additionally, ensuring the consistency and stability of NMF results in noisy environments is a subject of ongoing research. The quest for more efficient and scalable NMF algorithms continues, especially for handling large datasets. While NMF has made significant strides in recent years, addressing these challenges is crucial for unlocking its full potential in diverse data analysis and source separation tasks.

Hyperspectral Unmixing Using Robust Deep Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) and its numerous variants have been extensively studied and used in hyperspectral unmixing (HU). With the aid of the designed deep structure, deep NMF-based methods demonstrate advantages in exploring the hierarchical features of complex data. However, a noise corruption problem commonly exists in hyperspectral data and severely degrades the unmixing performance of deep NMF-based methods when applied to HU. In this study, we propose an ℓ2,1 norm-based robust deep nonnegative matrix factorization (ℓ2,1-RDNMF) for HU, which incorporates an ℓ2,1 norm into the two stages of the deep structure to achieve robustness. The multiplicative updating rules of ℓ2,1-RDNMF are efficiently learned and provided. The efficiency of the presented method is verified in experiments using both synthetic and genuine data.

Deep Nonnegative Matrix Factorization Using a Variational Autoencoder With Application to Single-Cell RNA Sequencing Data.

Single-cell RNA sequencing is used to analyze the gene expression data of individual cells, thereby adding to existing knowledge of biological phenomena. Accordingly, this technology is widely used in numerous biomedical studies. Recently, the variational autoencoder has emerged and has been adopted for the analysis of single-cell data owing to its high capacity to manage large-scale data. Many different variants of the variational autoencoder have been applied, and have yielded superior results. However, because it is nonlinear, the model does not provide parameters that can be used to explain the underlying biological patterns. In this paper, we propose an interpretable nonnegative matrix factorization method that decomposes parameters into those shared across cells and those that are cell-specific. Effective nonlinear dimension reduction was achieved via a variational autoencoder applied to the cell-specific parameters. In addition to achieving nonlinear dimension reduction, our model could estimate the cell-type-specific gene expression. To improve the estimation accuracy, we introduced log-regularization, which reflects the single-cell property. Overall, our approach displayed excellent performance in a simulation study and in real data analyses, while maintaining good biological interpretability.

Elastic adversarial deep nonnegative matrix factorization for matrix completion

In recent years, matrix completion has attracted a lot of attention. Conventional matrix completions are shallow and ineffective when dealing with complex data structures. Researchers have recently tried to incorporate deep structures into matrix completion; however, considerable challenges still exist. Most matrix completion methods may fail to work effectively in the presence of limited observations. To enhance the generalization of the reconstruction, adversarial methods are proposed that attempt to fool models by providing deceptive input. The aim is to develop an adversarial training algorithm that resists attacks in a deep model, thus at the same time leading to enhancing the generalization. Therefore, in this paper, we propose an elastic adversarial training to design a high-capacity Deep Nonnegative Matrix Factorization (DNMF) model with proper discovery latent structure of the data and enhanced generalization abilities. In other words, the challenges mentioned above are addressed by perturbing the inputs in DNMF with an elastic loss which is intercalated and adapted between Frobenius and ℓ2,1 norms. This model not only dispenses with adversarial DNMF generation but also is robust towards a mixture of multiple attacks to attain improved accuracy. Extensive simulations show that the proposed approach outperforms state-of-the-art methods.

Cross-domain residual deep NMF for transfer learning between different hyperspectral image scenes

Hyperspectral image (HSI) classification has long been a hot research topic. Most previous researches concentrate on the classification task of a single HSI scene, called single-scene classification. This research focuses on two closely related HSI scenes (called source and target scenes, respectively), and the problem is named cross-scene classification. This paper aims to explore the shared feature sub-space between two HSI scenes. A transfer learning algorithm called cross-domain residual deep nonnegative matrix factorization (CDRDNMF) is proposed. CDRDNMF is a multi-layer architecture consisting of dual-dictionary nonnegative matrix factorization (DDNMF) layers. In each layer, DDNMF is performed on source and target features for domain-invariant feature extraction. Then a data recovery process is completed, and the residual components from the recovery are passed to the next layer after activation. With such a multi-layer architecture, CDRDNMF delivers knowledge transfer and multi-scale feature extraction tasks. The experimental results prove the excellent performance of CDRDNMF on cross-scene classification.

Visual-Tactile Fused Graph Learning for Object Clustering.

Object clustering has received considerable research attention most recently. However, 1) most existing object clustering methods utilize visual information while ignoring important tactile modality, which would inevitably lead to model performance degradation and 2) simply concatenating visual and tactile information via multiview clustering method can make complementary information to not be fully explored, since there are many differences between vision and touch. To address these issues, we put forward a graph-based visual-tactile fused object clustering framework with two modules: 1) a modality-specific representation learning module MR and 2) a unified affinity graph learning module MU . Specifically, MR focuses on learning modality-specific representations for visual-tactile data, where deep non-negative matrix factorization (NMF) is adopted to extract the hidden information behind each modality. Meanwhile, we employ an autoencoder-like structure to enhance the robustness of the learned representations, and two graphs to improve its compactness. Furthermore, MU highlights how to mitigate the differences between vision and touch, and further maximize the mutual information, which adopts a minimizing disagreement scheme to guide the modality-specific representations toward a unified affinity graph. To achieve ideal clustering performance, a Laplacian rank constraint is imposed to regularize the learned graph with ideal connected components, where noises that caused wrong connections are removed and clustering labels can be obtained directly. Finally, we propose an efficient alternating iterative minimization updating strategy, followed by a theoretical proof to prove framework convergence. Comprehensive experiments on five public datasets demonstrate the superiority of the proposed framework.

Label consistency-based deep semisupervised NMF for tumor recognition

Tumor has become a hot topic in the field of image processing and pattern recognition. Gene expression data is an important way to study tumor. Since gene expression data is characterized by high-dimensional small samples, it is one of the key steps to extract discriminative gene features to distinguish different tumor types. Nonnegative matrix factorization (NMF) is an unsupervised feature representation method that does not depend on the label information of data. NMF can achieve nonlinear dimension reduction and is widely used in tumor recognition. Considering that some data may carry label information, models’ feature representation capability will be improved if the label information can be used effectively. Therefore, this paper proposes a semisupervised NMF model based on label consistency (LC-NMF), which uses both labeled and unlabeled data to obtain feature representations. Furthermore, to alleviate the sensitivity of the NMF model to initial values and excavate the deep features of data, a label consistency-based deep semisupervised NMF model (LC-DNMF) is constructed, which combines the LC-NMF model with the layer-by-layer pretraining and multilayer representation strategy in deep learning. The performance of the proposed models (i.e., LC-NMF and LC-DNMF) is verified by applying them to the tumor recognition tasks. The experimental results on seven datasets show that the two models achieve good results and can obtain competitive recognition accuracies compared with the state-of-the-art methods. Furthermore, the performance of the LC-DNMF model outperforms that of the LC-NMF model, which verifies the effectiveness of introducing the layer-by-layer pretraining and multilayer representation strategy.

Deep Autoencoder-like NMF with Contrastive Regularization and Feature Relationship Preservation

Nonnegative Matrix Factorization is a data analysis method to discover parts-based, linear representations of data. It has been successfully used in a great variety of applications. Deep Nonnegative Matrix Factorization (deep NMF) was recently established to cope with the extraction of hierarchical latent feature representation, and it has been demonstrated to achieve outstanding results in unsupervised representation learning. However, defining a suitable regularization for the deep models is a key challenge, and the existing Deep NMF approaches lack a well-suited regularization. In this paper, we propose the Deep Autoencoder-like NMF with Contrastive Regularization and Feature Relationship preservation (DANMF-CRFR) to address the above problem. Inspired by contrastive learning, this deep model is able to learn discriminative and instructive deep features while adequately enforcing the local and global structures of the data to its decoder and encoder components. Meanwhile, DANMF-CRFR also imposes feature correlations on the basis matrices during feature learning to improve part-based learning capabilities. Multiplicative updating rules and convergence guarantees are also provided. Extensive experimental results demonstrate the advantages of the proposed model. The source code for reproducing our results can be found at https://github.com/NavidSalahian/DANMF_CRFR.

Deep NMF topic modeling

Nonnegative matrix factorization (NMF) based topic modeling methods do not rely on model- or data-assumptions much. However, they are usually formulated as difficult optimization problems, which may suffer from bad local minima and high computational complexity. In this paper, we propose a deep NMF (DNMF) topic modeling framework to alleviate the aforementioned problems. It first applies an unsupervised deep learning method to learn latent hierarchical structures of documents, under the assumption that if we could learn a good representation of documents by, e.g. a deep model, then the topic word discovery problem can be boosted. Then, it takes the output of the deep model to constrain a topic-document distribution for the discovery of the discriminant topic words, which not only improves the efficacy but also reduces the computational complexity over conventional unsupervised NMF methods. We constrain the topic-document distribution in three ways, which takes the advantages of the three major sub-categories of NMF—basic NMF, structured NMF, and constrained NMF respectively. To overcome the weaknesses of deep neural networks in unsupervised topic modeling, we adopt a non-neural-network deep model—multilayer bootstrap network. To our knowledge, this is the first time that a deep NMF model is used for unsupervised topic modeling. We have compared the proposed method with a number of representative references covering major branches of topic modeling on a variety of real-world text corpora. Experimental results illustrate the effectiveness of the proposed method under various evaluation metrics.