Local Structure Preservation Research Articles

Knowledge distillation via adaptive meta-learning for graph neural network

With the ever-increasing scale of graph-structured data, the computational resource requirements of large-scale graph neural networks (GNNs) can impede their deployment on devices with limited resources. Through knowledge distillation (KD), the expertise of an expansive, previously trained model (the teacher) can be transferred to a smaller architecture (the student) while maintaining comparable performance. However, existing KD approaches for GNNs often fix the teacher model during student learning. Without considering the students' learning feedback, there is a performance reduction after KD is achieved. With this objective, this research proposed an expertise transfer method via adaptive meta-learning for GNNs. The teacher can continuously update its parameters according to the student's optimal gradient direction in each KD step. Thus, the teacher learns to teach appropriate knowledge to the student. To maintain the structural features of each node and further avoid over-smoothing, we also introduced local structure preservation loss. Comprehensive experiments across four benchmarks demonstrate the effectiveness of our methodology.

Mesoscopic structure graphs for interpreting uncertainty in non-linear embeddings

Probabilistic-based non-linear dimensionality reduction (PB-NL-DR) methods, such as t-SNE and UMAP, are effective in unfolding complex high-dimensional manifolds, allowing users to explore and understand the structural patterns of data. However, due to the trade-off between global and local structure preservation and the randomness during computation, these methods may introduce false neighborhood relationships, known as distortion errors and misleading visualizations. To address this issue, we first conduct a detailed survey to illustrate the design space of prior layout enrichment visualizations for interpreting DR results, and then propose a node-link visualization technique, ManiGraph. This technique rethinks the neighborhood fidelity between the high- and low-dimensional spaces by constructing dynamic mesoscopic structure graphs and measuring region-adapted trustworthiness. ManiGraph also addresses the overplotting issue in scatterplot visualization for large-scale datasets and supports examining in unsupervised scenarios. We demonstrate the effectiveness of ManiGraph in different analytical cases, including generic machine learning using 3D toy data illustrations and fashion-MNIST, a computational biology study using a single-cell RNA sequencing dataset, and a deep learning-enabled colorectal cancer study with histopathology-MNIST.

Joint Projected Fuzzy Neighborhood Preserving C-means Clustering with Local Adaptive Learning

Graph-based dimensionality reduction approaches are extensively applied for preserving the intrinsic structures of high-dimensional data. To capture discriminative information from data clusters that lack a clear manifold structure, previous studies have combined clustering algorithms with local manifold structure preservation. These methods assume that two neighbor data points share the same cluster label. However, the assumption may not hold for Fuzzy C-Means which may assign two nearby data points to different clusters depending on the distance between data points and cluster centroids. This inconsistency degrades the performance in clustering and dimensionality reduction. To tackle this issue, we propose a novel approach for dimensionality reduction, termed joint Projected Fuzzy Neighborhood Preserving C-means Clustering with Local Adaptive Learning (PFNPCM). PFNPCM learns cluster centroids, sparse membership grades, embedded subspaces, and structured graphs simultaneously and extracts both global and local structures of data. Through clustering data that preserves the local neighborhood structure in a low-dimensional space instead of the original one, PFNPCM can effectively capture both global and local information from the inherent manifold structure and cluster structure. The employment of ℓ2,1-norm as the distance metric in the clustering and the preservation of neighborhood structure further ensure the robustness of clustering results against noise or outliers. To achieve the consistency of the captured global and local discriminative information, PFNPCM adopts a novel fuzzy local similarity measure based on the manifold structure, and the manifold assumption thus holds when partitioning unlabeled samples into subgroups. Massive experiments on diverse well-known datasets demonstrate that PFNPCM is superior to some state-of-the-art methods.

C2FResMorph: A high-performance framework for unsupervised 2D medical image registration

Deformable medical image registration is an important precursor task for surgical automation, while enhancing the registration performance of 2D medical images remains a challenging work. Existing methods primarily minimize the similarity loss between image pairs as the main optimization objective, leading to limited registration accuracy and a lack of pixel matching. Moreover, the scarcity of informative features in 2D images often results in overfitting on the training set, hampering generalization. To address these issues, we propose C2FResMorph, a learning-based deformable registration algorithm specifically designed for 2D medical images. C2FResMorph employs a two-stage framework that improves registration accuracy and preserves topology during deformation in a coarse-to-fine manner. Inside the framework, by leveraging the convolutional neural network’s locality and the multi-head self-attention mechanism’s globality, a ResMorph registration network is designed. Additionally, the integration of residual image knowledge addresses deformation folding in 2D image registration, enhancing the preservation of local structures and improving generalization. Experimental evaluations on three datasets demonstrate that C2FResMorph outperforms existing learning-based methods in terms of accuracy, generalization ability for 2D medical image registration, and also retains the efficiency advantages.

A Survey of Multiple Kernel Graph Clustering Algorithms

In recent years, algorithms based on multi-kernel learning and graph clustering have garnered significant attention in the field of data mining. This article reviews four advanced multi-kernel graph clustering algorithms: nearest neighbor linear kernel weighted multi-kernel graph clustering, multi-kernel graph clustering based on simultaneous global and local structure preservation, multi-kernel graph clustering based on direct consensus relationship graph learning, and custom non-negative matrix factorization multi-kernel graph tensor clustering. These algorithms have their own characteristics and, by combining multi-kernel learning, graph clustering, and other advanced technologies, they improve the accuracy and efficiency of clustering. However, they also face challenges such as high computational complexity and parameter settings, necessitating further research and optimization. These algorithms have wide application prospects in fields such as image recognition, text classification, and bioinformatics, providing new solutions for handling complex data structures.

Heterogeneous domain adaptation by class centroid matching and local discriminative structure preservation

Heterogeneous domain adaptation by class centroid matching and local discriminative structure preservation

On Representation Knowledge Distillation for Graph Neural Networks.

Knowledge distillation (KD) is a learning paradigm for boosting resource-efficient graph neural networks (GNNs) using more expressive yet cumbersome teacher models. Past work on distillation for GNNs proposed the local structure preserving (LSP) loss, which matches local structural relationships defined over edges across the student and teacher's node embeddings. This article studies whether preserving the global topology of how the teacher embeds graph data can be a more effective distillation objective for GNNs, as real-world graphs often contain latent interactions and noisy edges. We propose graph contrastive representation distillation (G-CRD), which uses contrastive learning to implicitly preserve global topology by aligning the student node embeddings to those of the teacher in a shared representation space. Additionally, we introduce an expanded set of benchmarks on large-scale real-world datasets where the performance gap between teacher and student GNNs is non-negligible. Experiments across four datasets and 14 heterogeneous GNN architectures show that G-CRD consistently boosts the performance and robustness of lightweight GNNs, outperforming LSP (and a global structure preserving (GSP) variant of LSP) as well as baselines from 2-D computer vision. An analysis of the representational similarity among teacher and student embedding spaces reveals that G-CRD balances preserving local and global relationships, while structure preserving approaches are best at preserving one or the other.

Reverse Graph Learning for Graph Neural Network.

Graph neural networks (GNNs) conduct feature learning by taking into account the local structure preservation of the data to produce discriminative features, but need to address the following issues, i.e., 1) the initial graph containing faulty and missing edges often affect feature learning and 2) most GNN methods suffer from the issue of out-of-example since their training processes do not directly generate a prediction model to predict unseen data points. In this work, we propose a reverse GNN model to learn the graph from the intrinsic space of the original data points as well as to investigate a new out-of-sample extension method. As a result, the proposed method can output a high-quality graph to improve the quality of feature learning, while the new method of out-of-sample extension makes our reverse GNN method available for conducting supervised learning and semi-supervised learning. Experimental results on real-world datasets show that our method outputs competitive classification performance, compared to state-of-the-art methods, in terms of semi-supervised node classification, out-of-sample extension, random edge attack, link prediction, and image retrieval.

Graph embedding orthogonal decomposition: A synchronous feature selection technique based on collaborative particle swarm optimization

In unsupervised feature selection, the clustering label matrix has the ability to distinguish between projection clusters. However, the latent geometric structure of the clustering labels is often ignored. In addition, the optimal sub-feature selection performance of feature selection techniques relies greatly on the choice of balanced parameters, and the selection range of most technical parameters is limited and fixed. To solve the above-mentioned problems, this paper proposes a synchronous feature selection technique based on graph-embedded cluster label orthogonal decomposition and collaborative particle swarm optimization (GOD-cPSO). First, GOD-cPSO extends the feature selection framework of clustering label orthogonal decomposition by graph embedding to retain the latent geometric structure of clustering labels, thus maintaining the correlation between clustered sample labels. Then, the l2,1-2-norm with strong global convergence is extended to the graph embedding clustering label orthogonal decomposition framework. By imposing this non-convex constraint, GOD-cPSO can achieve low-dimensional sparse and low-redundant sub-features. In addition, the local structure preserving of low-dimensional manifolds is integrated into the graph-embedded clustering label orthogonal decomposition framework to obtain good cluster separation and effectively maintain the latent local structure of the data. Finally, to ensure the adaptive parameter selection over a large range, GOD-cPSO synchronously guides the graph-embedding clustering labeling orthogonal decomposition framework for feature selection through collaborative particle swarm optimization. GOD-cPSO has synchronous parameter optimization and feature selection and picks parameters in a larger range. Comprehensive numerical experiments are performed on nine datasets to test the validity of the GOD-cPSO. The experimental results demonstrate that the sub-features selected by the GOD-cPSO have stronger discriminative power and are superior to other techniques in the clustering assignments.

Discriminatively Fuzzy Multi-View K-means Clustering with Local Structure Preserving

Multi-view K-means clustering successfully generalizes K-means from single-view to multi-view, and obtains excellent clustering performance. In every view, it makes each data point close to the center of the corresponding cluster. However, multi-view K-means only considers the compactness of each cluster, but ignores the separability of different clusters, which is of great importance to producing a good clustering result. In this paper, we propose Discriminatively Fuzzy Multi-view K-means clustering with Local Structure Preserving (DFMKLS). On the basis of minimizing the distance between each data point and the center of the corresponding cluster, DFMKLS separates clusters by maximizing the distance between the centers of pairwise clusters. DFMKLS also relaxes its objective by introducing the idea of fuzzy clustering, which calculates the probability that a data point belongs to each cluster. Considering multi-view K-means mainly focuses on the global information of the data, to efficiently use the local information, we integrate the local structure preserving into the framework of DFMKLS. The effectiveness of DFMKLS is evaluated on benchmark multi-view datasets. It obtains superior performances than state-of-the-art multi-view clustering methods, including multi-view K-means.

Deep Nonnegative Matrix Factorization with Joint Global and Local Structure Preservation

Deep Non-Negative Matrix Factorization (DNMF) methods provide an efficient low-dimensional representation of given data through their layered architecture. A limitation of such methods is that they cannot effectively preserve the local and global geometric structures of the data in each layer. Consequently, a significant amount of the geometrical information within the data, present in each layer of the employed deep framework, can be overlooked by the model. This can lead to an information loss and a subsequent drop in performance. In this paper, we propose a novel deep non-negative matrix factorization method, Deep Non-Negative Matrix Factorization with Joint Global and Local Structure Preservation (dubbed Dn2MFGL), that ensures the preservation of both global and local structures within the data space. Dn2MFGL performs representation learning through a sequential embedding procedure which involves both the global data structure by accounting for the data variance, and the local data relationships by utilizing information from neighboring data points. Moreover, a regularization term that promotes sparsity by utilizing the concept of the inner product is applied to the matrices representing the lower dimensions. This aims to retain the fundamental data structure while discarding less crucial features. Simultaneously, the residual matrix of Dn2MFGL is subjected to the L2,1 norm, which ensures the robustness of the model against noisy data samples. An effective and multiplicative updating process also facilitates Dn2MFGL in solving the employed objective function. The clustering performance of the proposed deep NMF method is explored across various benchmarks of face datasets. The results point to Dn2MFGL outperforming several existing classical and state-of-the-art NMF methods. The source code is available at https://github.com/FaridSaberi/Dn2MFGO.git.

Graph manifold learning with non-gradient decision layer

Generally, Graph convolution network (GCN) utilizes the graph convolution operators and the softmax to extract the deep representation and make the prediction, respectively. Although GCN successfully represents the connectivity relationship among the nodes by aggregating the information on the graph, the softmax-based decision layer may result in suboptimal performance in semi-supervised learning with less label support due to ignoring the inner distribution of the graph nodes. Besides, the gradient descent will take thousands of interaction for optimization. To address the referred issues, we propose a novel graph deep model with a non-gradient decision layer for graph mining. Firstly, manifold learning is unified with label local-structure preservation to capture the topological information and make accurate predictions with limited label support. Moreover, it is theoretically proven to have analytical solutions and acts as a non-gradient decision layer in graph convolution networks. Particularly, a joint optimization method is designed for this graph model, which extremely accelerates the convergence of the model. Finally, extensive experiments show that the proposed model has achieved excellent performance compared to the current models.

Discriminative and robust least squares regression for semi-supervised image classification

Due to the ability to leverage information from both unlabeled and labeled data, semi-supervised classification has found extensive applications in various practical scenarios. However, there are two major drawbacks: (1) Traditional graph construction leads to high algorithmic complexity, which limits efficiency. (2) The decision boundary might be blurred by boundary points in common cases. To cope with these issues, inspired by classical Least Squares Regression (LSR), we present a novel semi-supervised classification algorithm termed as Discriminative and Robust LSR (DRLSR) for semi-supervised image classification. First of all, a manifold regularization term is designed and introduced to an LSR-based semi-supervised method to preserve local manifold structures and smooth structures of the subspace, which strengthens the discrimination ability. Meanwhile, preservation of local manifold structures also contributes to restrain decision boundaries from being blurred by boundary points, which strengthens robustness of the algorithm. After that, an efficient alternative optimization method is applied to our algorithm. Evidence of the effectiveness of DRLSR are compelled by extensive experimental results.

Untrained neural network embedded Fourier phase retrieval from few measurements

Fourier phase retrieval (FPR) is a challenging task used in various applications to recover an unknown signal from its Fourier phaseless measurements. FPR with few measurements helps reduce time and hardware costs but suffers from severe ill-posedness. Recently, untrained neural networks have offered new approaches by introducing learned priors to alleviate ill-posedness without requiring external data. However, they are computationally expensive and not ideal for reconstructing high-frequency structures in images. This paper proposes an untrained neural network (NN) embedded FPR algorithm, based on the alternating direction method of multipliers. Specifically, we use a generative network to represent the image to be recovered, which confines the image to the space defined by the network structure. Additionally, total variation regularization is imposed to facilitate the preservation of image edges and local structures. Furthermore, to reduce the computational cost mainly caused by the updates of NN’s parameters, we develop an accelerated algorithm that adaptively trades off between explicit and implicit regularization. We theoretically analyze that the proposed algorithm can converge to a stationary point of the objective function under mild conditions. Experimental results indicate that the proposed algorithm outperforms existing untrained-NN-based algorithms with fewer computational resources and even performs competitively against trained-NN-based algorithms.

Unsupervised domain adaptation with local structure preservation for colon histopathological image classification

The histopathological image classification method, based on deep learning, can be used to assist pathologists in cancer recognition in colon histopathology. The popularization of automatic and accurate histopathological image classification methods in this way is of great significance. However, smaller medical institutions with limited medical resources may lack colon histopathology image training sets with reliable labeled information; thus they may be unable to meet the needs of deep learning for many labeled training samples. Therefore, in this paper, the colon histopathological image set with rich label information from a certain medical institution is taken as the source domain; the colon histopathological image set from a smaller medical institution with limited medical resources is taken as the target domain. Considering the potential differences between histopathological images obtained by different institutions, this paper proposes a classification learning framework, namely unsupervised domain adaptation with local structure preservation for colon histopathological image classification, which can learn an adaptive classifier by performing distribution alignment and preserving intra-domain local structure to predict the labels of the colon histopathological images from institutions with lower medical resources. Extensive experiments demonstrate that the proposed framework shows significant improvement in accuracy and specificity of colon histopathological images without reliable labeled information compared to models without unsupervised domain adaptation. Specifically, in an affiliated hospital in Fuyang City, Anhui Province, the classification accuracy of benign and malignant colon histopathological images reaches 96.21%. The results of comparative experiments also show promising classification performance of our method in comparison with other unsupervised domain adaptation methods.

NeuroDAVIS: A neural network model for data visualization

The task of dimensionality reduction and visualization of high-dimensional datasets remains a challenging problem since long. Modern high-throughput technologies produce large datasets having multiple views with relatively new data types. Visualization of these datasets require efficient algorithms that can uncover hidden patterns in the data without affecting the local and global structures, and bring out the inherent non-linearity within the data. To this end, however, very few such methodologies exist, which can realize this task. In this work, we have introduced a novel unsupervised deep neural network model, called NeuroDAVIS, for data visualization. NeuroDAVIS is capable of extracting important features from the data, without assuming any data distribution, visualize effectively in low dimension, and preserve both local and global structures simultaneously. It has been shown theoretically that neighbourhood relationship of the data in high dimension remains preserved in lower dimension. The performance of NeuroDAVIS has been evaluated on a wide variety of synthetic and real high-dimensional datasets including numeric, textual, image and biological data. NeuroDAVIS has been highly competitive against t-Distributed Stochastic Neighbor Embedding (t-SNE), Uniform Manifold Approximation and Projection (UMAP), Principal Component Analysis (PCA) and Random Projection (RP) with respect to visualization quality, preservation of data size, shape, and both local and global structure. It has outperformed Fast interpolation-based t-SNE (Fit-SNE), a variant of t-SNE, for most of the datasets. For the biological datasets, besides t-SNE, UMAP, Fit-SNE, PCA and RP, NeuroDAVIS has also performed well as compared to other state-of-the-art algorithms, like Potential of Heat-diffusion for Affinity-based Trajectory Embedding (PHATE) and the siamese neural network-based method, called IVIS. Downstream classification and clustering analyses have also revealed favourable results for NeuroDAVIS-generated embeddings.

Learning an Interpretable Stylized Subspace for 3D-aware Animatable Artforms.

Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the pose and content of a given art painting. This anchor serves as a reliable indicator of the original latent space local structure, therefore sharing the same editable predefined expression vectors. In the second stage, we train a customized 3D-aware GAN specific to the input artform, while enforcing the preservation of the original latent local structure through a meticulous style-directional difference loss. This approach ensures the creation of a stylized sub-space that remains interpretable and retains 3D control. The effectiveness and versatility of 3DArtmator are validated through extensive experiments across a diverse range of art styles. With the ability to generate 3D reconstruction and editing for artforms while maintaining interpretability, 3DArtmator opens up new possibilities for artistic exploration and engagement.

Deep Fuzzy Variable C-Means Clustering Incorporated With Curriculum Learning

End-to-end deep clustering method utilizes deep neural networks to jointly learn representation features and clustering assignments. Although many k-means-friendly deep clustering models have been explored, the existing division-based methods tend to directly implement clustering with a specific number of clusters, which suffers from poor performance resulted from indistinguishable clusters, and contributes to bad local optimum. At the same time, the representation learning of fuzzy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> -means clustering in the feature space still needs more research. In this paper, a Deep Fuzzy Curriculum Clustering (DFC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) method with the learning strategy of clustering from easy to complex automatically is proposed to tackle above issues. Firstly, considering the soft flexible allocation of fuzzy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> -means and the preservation of local structure of original data, the fuzzy clustering loss and the autoencoder's reconstruction loss are constructed to learn the embedded features and clustering centers simultaneously. Second, curriculum loss is introduced into the constraint to make clusters successively merge in line with implementing clustering from easy to complex, and realize the bottom-up deep aggregative clustering automatically. In addition, novel curriculum information is proposed as constraint to guide the merging of clusters belonging to the same class. Experimental results on four real-world datasets show the superiority of the proposal.

Sparse low-redundancy multilabel feature selection based on dynamic local structure preservation and triple graphs exploration

Much semantic information is involved in multilabel data due to more than one label associated with each instance. The redundant features and noise challenge knowledge mining in multilabel data. Constructing a learning model with discriminative features is essential for multilabel learning. Sparse graph-based methods simultaneously consider the topological structure, complex relations between features and labels, and the significance of features. However, three challenges exist. First, they either consider local label correlation or local label relevance and are complementary in the feature selection process. Second, existing methods use low-quality static graphs to explore local label correlations that result in degraded performance. Finally, only some methods deal with redundant features. A ridge regression-based sparse multilabel learning is proposed in this study to address these problems. The global and local label correlation are explored by preserving the instance-level graph structure to obtain a robust low-dimensional pseudo-label matrix to construct a high-quality dynamic label-level graph. Meanwhile, it preserves the feature-level graph structure to select low-redundant features. In addition, a new ℓ2,1/2−2-norm is designed to maintain the high-row sparsity of the model. The above items are embedded into a unified multilabel learning framework. A simple and effective optimization solution is finally designed and compared with eight relevant algorithms on twelve public benchmark data sets. The results demonstrate that the algorithm can improve classification performance.

Unsupervised learning: Local and global structure preservation in industrial data

This article examines the use of data mining and machine learning algorithms for knowledge discovery in industrial processes, with a focus on understanding how these methods distinguish and represent the global and local structures underlying data. The effectiveness of alternative dimensionality reduction algorithms in characterizing process operations is also explored, with an emphasis on a novel dimensionality reduction method, PaCMAP, compared to the dependable historical method of PCA and the more recent t-SNE. The article demonstrates a visualization method for identifying the variables responsible for the separation between selected clusters and studies the effects of key elements of the algorithms on cluster analysis in light of domain knowledge. Results from an industrial case study of a pyrolysis reactor validates the applicability of these methods in real-life scenarios. Overall, this article provides insights into the use of machine learning and dimensionality reduction algorithms for knowledge discovery in industrial processes.