Graph-based Semi-supervised Learning Method Research Articles

Almost exact recovery in noisy semi-supervised learning

Abstract Graph-based semi-supervised learning methods combine the graph structure and labeled data to classify unlabeled data. In this work, we study the effect of a noisy oracle on classification. In particular, we derive the maximum a posteriori (MAP) estimator for clustering a degree corrected stochastic block model when a noisy oracle reveals a fraction of the labels. We then propose an algorithm derived from a continuous relaxation of the MAP, and we establish its consistency. Numerical experiments show that our approach achieves promising performance on synthetic and real data sets, even in the case of very noisy labeled data.

Graph-Based Semi-Supervised Learning with Bipartite Graph for Large-Scale Data and Prediction of Unseen Data

Recently, considerable attention has been directed toward graph-based semi-supervised learning (GSSL) as an effective approach for data labeling. Despite the progress achieved by current methodologies, several limitations persist. Firstly, many studies treat all samples equally in terms of weight and influence, disregarding the potential increased importance of samples near decision boundaries. Secondly, the detection of outlier-labeled data is crucial, as it can significantly impact model performance. Thirdly, existing models often struggle with predicting labels for unseen test data, restricting their utility in practical applications. Lastly, most graph-based algorithms rely on affinity matrices that capture pairwise similarities across all data points, thus limiting their scalability to large-scale databases. In this paper, we propose a novel GSSL algorithm tailored for large-scale databases, leveraging anchor points to mitigate the challenges posed by large affinity matrices. Additionally, our method enhances the influence of nodes near decision boundaries by assigning different weights based on their importance and using a mapping function from feature space to label space. Leveraging this mapping function enables direct label prediction for test samples without requiring iterative learning processes. Experimental evaluations on two extensive datasets (Norb and Covtype) demonstrate that our approach is scalable and outperforms existing GSSL methods in terms of performance metrics.

Scalable and Inductive Semi-supervised Classifier with Sample Weighting Based on Graph Topology

Recently, graph-based semi-supervised learning (GSSL) has garnered significant interest in the realms of machine learning and pattern recognition. Although some of the proposed methods have made some progress, there are still some shortcomings that need to be overcome. There are three main limitations. First, the graphs used in these approaches are usually predefined regardless of the task at hand. Second, due to the use of graphs, almost all approaches are unable to process and consider data with a very large number of unlabeled samples. Thirdly, the imbalance of the topology of the samples is very often not taken into account. In particular, processing large datasets with GSSL might pose challenges in terms of computational resource feasibility. In this article, we present a scalable and inductive GSSL method. We broaden the scope of the graph topology imbalance paradigm to extensive databases. Second, we employ the calculated weights of the labeled sample for the label-matching term in the global objective function. This leads to a unified, scalable, semi-supervised learning model that allows simultaneous labeling of unlabeled data, projection of the feature space onto the labeling space, along with the graph matrix of anchors. In the proposed scheme, the integration of labels and features from anchors is applied for the adaptive construction of the anchor graph. Experimental results were performed on four large databases: NORB, RCV1, Covtype, and MNIST. These experiments demonstrate that the proposed method exhibits superior performance when compared to existing scalable semi-supervised learning models.

Model Change Active Learning in Graph-Based Semi-supervised Learning

Active learning in semi-supervised classification involves introducing additional labels for unlabelled data to improve the accuracy of the underlying classifier. A challenge is to identify which points to label to best improve performance while limiting the number of new labels. “Model Change” active learning quantifies the resulting change incurred in the classifier by introducing the additional label(s). We pair this idea with graph-based semi-supervised learning (SSL) methods, that use the spectrum of the graph Laplacian matrix, which can be truncated to avoid prohibitively large computational and storage costs. We consider a family of convex loss functions for which the acquisition function can be efficiently approximated using the Laplace approximation of the posterior distribution. We show a variety of multiclass examples that illustrate improved performance over prior state-of-art.

A novel adaptive safe semi-supervised learning framework for pattern extraction and classification

<p>Manifold regularization semi-supervised learning is a powerful graph-based semi-supervised learning method. However, the performance of semi-supervised learning methods based on manifold regularization depends to some extent on the quality of the manifold graph and unlabeled samples. Intuitively speaking, the quality of the graph directly affects the final classification performance of the model. In response to the above problems, this paper first proposed an adaptive safety semi-supervised learning framework. The framework implements the weight assignment of the self-similarity graph during the model learning process. In order to adapt to the learning needs, accelerate the learning speed, and avoid the impact of the curse of dimensionality, the framework also optimizes the features of each sample point through an automatic weighting mechanism to extract effective features and eliminate redundant information in the learning task. In addition, the framework defines an adaptive risk measurement mechanism for the uncertainty and potential risks of unlabeled samples to determine the degree of risk of unlabeled samples. Finally, a new adaptive safe semi-supervised extreme learning machine was proposed. Comprehensive experimental results across various class imbalance scenarios demonstrated that our proposed method outperforms other methods in terms of classification accuracy, and other critical performance metrics.</p>

From Cluster Assumption to Graph Convolution: Graph-Based Semi-Supervised Learning Revisited.

Graph-based semi-supervised learning (GSSL) has long been a research focus. Traditional methods are generally shallow learners, based on the cluster assumption. Recently, graph convolutional networks (GCNs) have become the predominant techniques for their promising performance. However, a critical question remains largely unanswered: why do deep GCNs encounter the oversmoothing problem, while traditional shallow GSSL methods do not, despite both progressing through the graph in a similar iterative manner? In this article, we theoretically discuss the relationship between these two types of methods in a unified optimization framework. One of the most intriguing findings is that, unlike traditional ones, typical GCNs may not effectively incorporate both graph structure and label information at each layer. Motivated by this, we propose three simple but powerful graph convolution methods. The first, optimized simple graph convolution (), is a supervised method, which guides the graph convolution process with labels. The others are two "no-learning" unsupervised methods: graph structure preserving graph convolution () and its multiscale version GGCM, both aiming to preserve the graph structure information during the convolution process. Finally, we conduct extensive experiments to show the effectiveness of our methods.

Graph-Based Semi-Supervised Learning: A Comprehensive Review.

Semi-supervised learning (SSL) has tremendous value in practice due to the utilization of both labeled and unlabelled data. An essential class of SSL methods, referred to as graph-based semi-supervised learning (GSSL) methods in the literature, is to first represent each sample as a node in an affinity graph, and then, the label information of unlabeled samples can be inferred based on the structure of the constructed graph. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large-scale data. Focusing on GSSL methods only, this work aims to provide both researchers and practitioners with a solid and systematic understanding of relevant advances as well as the underlying connections among them. The concentration on one class of SSL makes this article distinct from recent surveys that cover a more general and broader picture of SSL methods yet often neglect the fundamental understanding of GSSL methods. In particular, a significant contribution of this article lies in a newly generalized taxonomy for GSSL under the unified framework, with the most up-to-date references and valuable resources such as codes, datasets, and applications. Furthermore, we present several potential research directions as future work with our insights into this rapidly growing field.

A unified semi-supervised model with joint estimation of graph, soft labels and latent subspace

Since manually labeling images is expensive and labor intensive, in practice we often do not have enough labeled images to train an effective classifier for the new image classification tasks. The graph-based SSL methods have received more attention in practice due to their convexity, scalability and efficiency. In this paper, we propose a novel graph-based semi-supervised learning method that takes full advantage of a small set of labeled graphs and a large set of unlabeled graph data. First, we explain the concept of graph-based semi-supervised learning. The core idea of these models is to jointly estimate a low-rank graph with soft labels and a latent subspace. The proposed scheme leverages the synergy between the graph structure and the data representation in terms of soft labels and latent features. This improves the monitoring information and leads to better discriminative linear transformation. Several experiments were conducted on five image datasets using state-of-the-art methods. These experiments show the effectiveness of the proposed semi-supervised method.

A multi-sensor fusion-based prognostic model for systems with partially observable failure modes

With the rapid development of sensor and communication technology, multi-sensor data is available to monitor the degradation of complex systems and predict the failure modes. However, two huge challenges remain to be resolved: (i) how to predict the failure modes with limited failure mode labeled systems to alleviate the heavy dependence on expert experience; (ii) how to effectively fuze the useful information from the multi-sensor data to achieve an accurate estimation of the degradation status automatically. To address these issues, we propose a novel semi-supervised prognostic model for the systems with partially observable failure modes, where only a small fraction of the systems in the training set are known for their failure modes. First, we develop a graph-based semi-supervised learning method to extract features characterizing the failure modes. Then, we input these features as well as the multi-sensor streams into an elastic net functional regression model to predict the residual useful lifetime. The proposed model is validated by extensive simulation studies and a case study of aircraft turbofan engines available from the NASA repository.

Robust deep semi-supervised learning with label propagation and differential privacy

Semi-supervised learning (SSL) methods provide a powerful tool for utilizing abundant unlabeled data to strengthen standard supervised learning. Traditional graph-based SSL methods prevail in classical SSL problems for their intuitional implementation and effective performance. However, they encounter troubles when applying to image classification followed by modern deep learning, since the diffusion algorithms face the curse of dimensionality. In this study, we propose a simple and efficient SSL method, combining a graph-based SSL paradigm with differential privacy. We aim at developing coherent latent feature space of deep neural networks so that the diffusion algorithm in the latent space can give more precise predictions for unlabeled data. Our approach achieves state-of-the-art performance on the Cifar10, Cifar100, and Mini-imagenet benchmark datasets and obtains an error rate of 18.56% on Cifar10 using only 1% of all labels. Furthermore, our approach inherits the benefits of graph-based SSL methods with a simple training process and can be easily combined with any network architecture.

Robust semi-supervised multi-view graph learning with sharable and individual structure

The construction of a high-quality multi-view consensus graph is key to graph-based semi-supervised multi-view learning (GSSMvL) methods. However, most existing GSSMvL methods explore sample relationships in the original multi-view feature space, which obtains a contaminated graph that cannot reveal the underlying manifold structure of the samples. Moreover, traditional GSSMvL methods fail to explore the diverse structures of multi-view features, which may lose their complementary information and lead to a suboptimal graph. In this paper, we propose a novel unified robust semi-supervised multi-view graph learning framework based on the sharable and individual structure (RSSMvSI), which can eliminate the influence of noise and exploit the knowledge of multi-view data in a reasonable manner. Specifically, we first learn clean data by manipulating sparse noise with l2,1 norm. We then simultaneously explore the sharable and individual self-representation subspace on the learned clean multi-view data. The key point is that noisy data does not participate in subspace learning, which improves the robustness of the proposed method. By constructing the optimal consensus graph with the learned sharable and individual subspace, RSSMvSI can better utilize the complementary information of multi-view data and approximate the manifold structure of samples. To the best of our knowledge, this is the first attempt to learn the self-representation subspace on recovered multi-view clean data. Extensive experiments on various real-world multi-view datasets demonstrate the superiority and robustness against state-of-the-art methods.

Graph Learning on Millions of Data in Seconds: Label Propagation Acceleration on Graph Using Data Distribution.

Graph-based semi-supervised learning methods have been used in a wide range of real-world applications, e.g., from social relationship mining to multimedia classification and retrieval. However, existing methods are limited along with high computational complexity or not facilitating incremental learning, which may not be powerful to deal with large-scale data, whose scale may continuously increase, in real world. This paper proposes a new method called Data Distribution Based Graph Learning (DDGL) for semi-supervised learning on large-scale data. This method can achieve a fast and effective label propagation and supports incremental learning. The key motivation is to propagate the labels along smaller-scale data distribution model parameters, rather than directly dealing with the raw data as previous methods, which accelerate the data propagation significantly. It also improves the prediction accuracy since the loss of structure information can be alleviated in this way. To enable incremental learning, we propose an adaptive graph updating strategy which can update the model when there is distribution bias between new data and the already seen data. We have conducted comprehensive experiments on multiple datasets with sample sizes increasing from seven thousand to five million. Experimental results on the classification task on large-scale data demonstrate that our proposed DDGL method improves the classification accuracy by a large margin while consuming much less time compared to state-of-the-art methods.

Uniform convergence rates for Lipschitz learning on graphs

Abstract Lipschitz learning is a graph-based semisupervised learning method where one extends labels from a labeled to an unlabeled data set by solving the infinity Laplace equation on a weighted graph. In this work we prove uniform convergence rates for solutions of the graph infinity Laplace equation as the number of vertices grows to infinity. Their continuum limits are absolutely minimizing Lipschitz extensions (AMLEs) with respect to the geodesic metric of the domain where the graph vertices are sampled from. We work under very general assumptions on the graph weights, the set of labeled vertices and the continuum domain. Our main contribution is that we obtain quantitative convergence rates even for very sparsely connected graphs, as they typically appear in applications like semisupervised learning. In particular, our framework allows for graph bandwidths down to the connectivity radius. For proving this we first show a quantitative convergence statement for graph distance functions to geodesic distance functions in the continuum. Using the ‘comparison with distance functions’ principle, we can pass these convergence statements to infinity harmonic functions and AMLEs.

Multi-Model Adaptation Learning With Possibilistic Clustering Assumption for EEG-Based Emotion Recognition.

In machine learning community, graph-based semi-supervised learning (GSSL) approaches have attracted more extensive research due to their elegant mathematical formulation and good performance. However, one of the reasons affecting the performance of the GSSL method is that the training data and test data need to be independently identically distributed (IID); any individual user may show a completely different encephalogram (EEG) data in the same situation. The EEG data may be non-IID. In addition, noise/outlier sensitiveness still exist in GSSL approaches. To these ends, we propose in this paper a novel clustering method based on structure risk minimization model, called multi-model adaptation learning with possibilistic clustering assumption for EEG-based emotion recognition (MA-PCA). It can effectively minimize the influence from the noise/outlier samples based on different EEG-based data distribution in some reproduced kernel Hilbert space. Our main ideas are as follows: (1) reducing the negative impact of noise/outlier patterns through fuzzy entropy regularization, (2) considering the training data and test data are IID and non-IID to obtain a better performance by multi-model adaptation learning, and (3) the algorithm implementation and convergence theorem are also given. A large number of experiments and deep analysis on real DEAP datasets and SEED datasets was carried out. The results show that the MA-PCA method has superior or comparable robustness and generalization performance to EEG-based emotion recognition.

A safe semi-supervised graph convolution network.

In the semi-supervised learning field, Graph Convolution Network (GCN), as a variant model of GNN, has achieved promising results for non-Euclidean data by introducing convolution into GNN. However, GCN and its variant models fail to safely use the information of risk unlabeled data, which will degrade the performance of semi-supervised learning. Therefore, we propose a Safe GCN framework (Safe-GCN) to improve the learning performance. In the Safe-GCN, we design an iterative process to label the unlabeled data. In each iteration, a GCN and its supervised version (S-GCN) are learned to find the unlabeled data with high confidence. The high-confidence unlabeled data and their pseudo labels are then added to the label set. Finally, both added unlabeled data and labeled ones are used to train a S-GCN which can achieve the safe exploration of the risk unlabeled data and enable safe use of large numbers of unlabeled data. The performance of Safe-GCN is evaluated on three well-known citation network datasets and the obtained results demonstrate the effectiveness of the proposed framework over several graph-based semi-supervised learning methods.

Semi-supervised hierarchical attribute representation learning via multi-layer matrix factorization for machinery fault diagnosis

Data-driven fault diagnosis methods have become a research hotspot recently. However, the following two problems are still barring them from the application: (1) Most of the existing models rely deeply on sufficient labeled samples and neglect the high cost of labeled data collection in reality; (2) The existing models usually focus on the single-level attribute of the sample and ignore the latent hierarchical fault attributes. To address these issues, a novel semi-supervised multi-layer non-negative matrix factorization (SMNMF) method is proposed in this study. The fault pattern and severity identification problems are converted into a hierarchical fault attribute representation task, which can reduce the complexity of the classification task and improve the fault diagnosis accuracy. The hierarchical attribute representations of different fault locations and sizes are learned from the time-frequency distribution (TFD) of signals by a newly constructed two-layer non-negative matrix factorization model. The graph-based semi-supervised learning method is adopted to lead the attributes of the hierarchy structure and carry out label propagation from labeled samples to unlabeled samples for more accurate fault diagnosis. The fault diagnosis experiments executed in the aeroengine bearings and a diesel engine demonstrated the feasibility and superiority of the proposed method.

Fast Semi-Supervised Learning With Optimal Bipartite Graph

Recently, with the explosive increase in Internet data, the traditional Graph-based Semi-Supervised Learning (GSSL) model is not suitable to deal with large scale data as the high computation complexity. Besides, GSSL models perform classification on a fixed input data graph. The quality of initialized graph has a great effect on the classification result. To solve this problem, in this paper, we propose a novel approach, named optimal bipartite graph-based SSL (OBGSSL). Instead of fixing the input data graph, we learn a new bipartite graph to make the result more robust. Based on the learned bipartite graph, the labels of the original data and anchors can be calculated simultaneously, which solves co-classification problem in SSL. Then, we use the label of anchor to handle out-of-sample problem, which preserves well classification performance and saves much time. The computational complexity of OBGSSL is O(ndmt+nm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), which is a significant improvement compared with traditional GSSL methods that need O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> d+n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ), where n, d, m and t are the number of samples, features anchors and iterations, respectively. Experimental results demonstrate the effectiveness and efficiency of our OBGSSL model.

A Graph-based Semi-supervised Multi-label Learning Method Based on Label Correlation Consistency

A Graph-based Semi-supervised Multi-label Learning Method Based on Label Correlation Consistency

Possibilistic Clustering-Promoting Semi-Supervised Learning for EEG-Based Emotion Recognition.

The purpose of the latest brain computer interface is to perform accurate emotion recognition through the customization of their recognizers to each subject. In the field of machine learning, graph-based semi-supervised learning (GSSL) has attracted more and more attention due to its intuitive and good learning performance for emotion recognition. However, the existing GSSL methods are sensitive or not robust enough to noise or outlier electroencephalogram (EEG)-based data since each individual subject may present noise or outlier EEG patterns in the same scenario. To address the problem, in this paper, we invent a Possibilistic Clustering-Promoting semi-supervised learning method for EEG-based Emotion Recognition. Specifically, it constrains each instance to have the same label membership value with its local weighted mean to improve the reliability of the recognition method. In addition, a regularization term about fuzzy entropy is introduced into the objective function, and the generalization ability of membership function is enhanced by increasing the amount of sample discrimination information, which improves the robustness of the method to noise and the outlier. A large number of experimental results on the three real datasets (i.e., DEAP, SEED, and SEED-IV) show that the proposed method improves the reliability and robustness of the EEG-based emotion recognition.

A simple graph-based semi-supervised learning approach for imbalanced classification

Graph-based Semi-Supervised Learning (GSSL) methods aim to classify unlabeled data by learning the graph structure and labeled data jointly. In this work, we propose a simple GSSL approach, which can deal with various degrees of class imbalance in given datasets. The key idea is to estimate the class proportion of input data in order to enhance the discriminative power of learned smooth classification function on the graph. Moreover, it has interesting connections to the regularization framework, the Markov stability for graph partition and the group inverse of normalized Laplacain matrix. For classification problems, experimental results demonstrate our approach can achieve promising performance on several datasets with varying class imbalance.