Graph-based Semi-supervised Learning Research Articles

Building semi-supervised decision trees with semi-cart algorithm

AbstractDecision trees are a fundamental statistical learning tool for addressing classification and regression problems through a recursive partitioning approach that effectively accommodates numerical and categorical data [1, 2]. The Classification and regression tree (CART) algorithm underlies modern Boosting methodologies such as Gradient boosting machine (GBM), Extreme gradient boosting (XGBoost), and Light gradient boosting machine (LightGBM). However, the standard CART algorithm may require improvement due to its inability to learn from unlabeled data. This study proposes several modifications to incorporate test data into the training phase. Specifically, we introduce a method based on Graph-based semi-supervised learning called “Distance-based Weighting,” which calculates and removes irrelevant records from the training set to accelerate the training process and improve performance. We present Semi-supervised classification and regression tree (Semi-Cart), a new implementation of CART that constructs a decision tree using weighted training data. We evaluated its performance on thirteen datasets from various domains. Our results demonstrate that Semi-Cart outperforms standard CART methods and contributes to statistical learning.

Efficient dynamic IC design analysis using graph-based semi-supervised learning

Efficient dynamic IC design analysis using graph-based semi-supervised learning

GUIDE: Training Deep Graph Neural Networks via Guided Dropout Over Edges.

Graph neural networks (GNNs) have made great progress in graph-based semi-supervised learning (GSSL). However, most existing GNNs are confronted with the oversmoothing issue that limits their expressive ability. A key factor that leads to this problem is the excessive aggregation of information from other classes when updating the node representation. To alleviate this limitation, we propose an effective method called GUIded Dropout over Edges (GUIDE) for training deep GNNs. The core of the method is to reduce the influence of nodes from other classes by removing a certain number of inter-class edges. In GUIDE, we drop edges according to the edge strength, which is defined as the time an edge acts as a bridge along the shortest path between node pairs. We find that the stronger the edge strength, the more likely it is to be an inter-class edge. In this way, GUIDE can drop more inter-class edges and keep more intra-class edges. Therefore, nodes in the same community or class are more similar, whereas different classes are more separated in the embedded space. In addition, we perform some theoretical analysis of the proposed method, which explains why it is effective in alleviating the oversmoothing problem. To validate its rationality and effectiveness, we conduct experiments on six public benchmarks with different GNNs backbones. Experimental results demonstrate that GUIDE consistently outperforms state-of-the-art methods in both shallow and deep GNNs.

Scalable and Inductive Semi-supervised Classifier with Sample Weighting Based on Graph Topology

Recently, graph-based semi-supervised learning (GSSL) has garnered significant interest in the realms of machine learning and pattern recognition. Although some of the proposed methods have made some progress, there are still some shortcomings that need to be overcome. There are three main limitations. First, the graphs used in these approaches are usually predefined regardless of the task at hand. Second, due to the use of graphs, almost all approaches are unable to process and consider data with a very large number of unlabeled samples. Thirdly, the imbalance of the topology of the samples is very often not taken into account. In particular, processing large datasets with GSSL might pose challenges in terms of computational resource feasibility. In this article, we present a scalable and inductive GSSL method. We broaden the scope of the graph topology imbalance paradigm to extensive databases. Second, we employ the calculated weights of the labeled sample for the label-matching term in the global objective function. This leads to a unified, scalable, semi-supervised learning model that allows simultaneous labeling of unlabeled data, projection of the feature space onto the labeling space, along with the graph matrix of anchors. In the proposed scheme, the integration of labels and features from anchors is applied for the adaptive construction of the anchor graph. Experimental results were performed on four large databases: NORB, RCV1, Covtype, and MNIST. These experiments demonstrate that the proposed method exhibits superior performance when compared to existing scalable semi-supervised learning models.

Robust Structure-Aware Graph-based Semi-Supervised Learning: Batch and Recursive Processing

Graph-based semi-supervised learning plays an important role in large scale image classification tasks. However, the problem becomes very challenging in the presence of noisy labels and outliers. Moreover, traditional robust semi-supervised learning solutions suffers from prohibitive computational burdens thus cannot be computed for streaming data. Motivated by that, we present a novel unified framework robust structure-aware semi-supervised learning called Unified RSSL (URSSL) for batch processing and recursive processing robust to both outliers and noisy labels. Particularly, URSSL applies joint semi-supervised dimensionality reduction with robust estimators and network sparse regularization simultaneously on the graph Laplacian matrix iteratively to preserve the intrinsic graph structure and ensure robustness to the compound noise. First, in order to relieve the influence from outliers, a novel semi-supervised robust dimensionality reduction is applied relying on robust estimators to suppress outliers. Meanwhile, to tackle noisy labels, the denoised graph similarity information is encoded into the network regularization. Moreover, by identifying strong relevance of dimensionality reduction and network regularization in the context of robust semi-supervised learning (RSSL), a two-step alternative optimization is derived to compute optimal solutions with guaranteed convergence. We further derive our framework to adapt to large scale semi-supervised learning particularly suitable for large scale image classification and demonstrate the model robustness under different adversarial attacks. For recursive processing, we rely on reparameterization to transform the formulation to unlock the challenging problem of robust streaming-based semi-supervised learning. Last but not least, we extend our solution into distributed solutions to resolve the challenging issue of distributed robust semi-supervised learning when images are captured by multiple cameras at different locations. Extensive experimental results demonstrate the promising performance of this framework when applied to multiple benchmark datasets with respect to state-of-the-art approaches for important applications in the areas of image classification and spam data analysis.

Model Change Active Learning in Graph-Based Semi-supervised Learning

AbstractActive learning in semi-supervised classification involves introducing additional labels for unlabelled data to improve the accuracy of the underlying classifier. A challenge is to identify which points to label to best improve performance while limiting the number of new labels. “Model Change” active learning quantifies the resulting change incurred in the classifier by introducing the additional label(s). We pair this idea with graph-based semi-supervised learning (SSL) methods, that use the spectrum of the graph Laplacian matrix, which can be truncated to avoid prohibitively large computational and storage costs. We consider a family of convex loss functions for which the acquisition function can be efficiently approximated using the Laplace approximation of the posterior distribution. We show a variety of multiclass examples that illustrate improved performance over prior state-of-art.

Semisupervised Subspace Learning With Adaptive Pairwise Graph Embedding.

Graph-based semisupervised learning can explore the graph topology information behind the samples, becoming one of the most attractive research areas in machine learning in recent years. Nevertheless, existing graph-based methods also suffer from two shortcomings. On the one hand, the existing methods generate graphs in the original high-dimensional space, which are easily disturbed by noisy and redundancy features, resulting in low-quality constructed graphs that cannot accurately portray the relationships between data. On the other hand, most of the existing models are based on the Gaussian assumption, which cannot capture the local submanifold structure information of the data, thus reducing the discriminativeness of the learned low-dimensional representations. This article proposes a semisupervised subspace learning with adaptive pairwise graph embedding (APGE), which first builds a k1 -nearest neighbor graph on the labeled data to learn local discriminant embeddings for exploring the intrinsic structure of the non-Gaussian labeled data, i.e., the submanifold structure. Then, a k2 -nearest neighbor graph is constructed on all samples and mapped to GE learning to adaptively explore the global structure of all samples. Clustering unlabeled data and its corresponding labeled neighbors into the same submanifold, sharing the same label information, improves embedded data's discriminative ability. And the adaptive neighborhood learning method is used to learn the graph structure in the continuously optimized subspace to ensure that the optimal graph matrix and projection matrix are finally learned, which has strong robustness. Meanwhile, the rank constraint is added to the Laplacian matrix of the similarity matrix of all samples so that the connected components in the obtained similarity matrix are precisely equal to the number of classes in the sample, which makes the structure of the graph clearer and the relationship between the near-neighbor sample points more explicit. Finally, multiple experiments on several synthetic and real-world datasets show that the method performs well in exploring local structure and classification tasks.

Overcoming graph topology imbalance for inductive and scalable semi-supervised learning

Graph-based semi-supervised learning (GSSL) has received much attention recently. Despite some progress made in this area by some recent methods, some limitations still need to be addressed. Namely, there are two main shortcomings. First, the graphs used are very often built in advance and independently of the task at hand, using a heuristic, and generally do not represent the true topology of the data. The second shortcoming is the ability of the model to handle a very large number of unlabeled samples. This can make the GSSL solution impractical from a computational resource perspective. In this paper, we propose the Weighted Simultaneous Graph Construction and Reduced Flexible Manifold Embedding (W-SGRFME) method, which is a scalable and inductive GSSL framework. The main contributions are as follows. First, we extend the concept of graph topology imbalance to large datasets. Second, we integrate the computed weights of the labeled samples into the unified semi-supervised model. The latter jointly estimates the labels of the unlabeled samples, the mapping of the feature space to the label space, and the graph matrix of the anchor graph. Moreover, the fusion of labels and features of anchors is used to adaptively construct the graph. Experimental results on three large datasets from semi-supervised learning show the effectiveness of the proposed scalable method. These datasets are NORB, RCV1, and Covtype. Experimental results on large datasets show the superiority of the proposed method over existing scalable models.

Sample-weighted fused graph-based semi-supervised learning on multi-view data

Research in semi-supervised learning on graphs has attracted more and more attention in recent years, as learning on graphs is applied in more and more domains and labeling data is expensive and time-consuming. Some scenarios have inherent graph structures in their data, such as the relationships between people in social scenarios or the relationships between objects that are mutually referenced. However, there are also many data types without inherent graph structures, such as image data, and each image can be described with different features, which is a typical type of multi-view data. For image data and other non-graph data, there are significantly fewer deep learning approaches that target multi-view graph-based semi-supervised learning. This paper attempts to fill this gap. Based on the Graph Convolutional Network (GCN) architecture, we propose a Sample-weighted Fused Graph-based Semi-supervised Classification (WFGSC) method for multi-view data in this paper. It (i) constructs a semi-supervised graph in each view using a flexible model for joint graph and label estimation, (ii) obtains an additional graph based on the representation of nodes provided by the joint estimator, and then obtains a fused graph between all views, (iii) gives higher weights to hard-to-classify samples, (iv) proposes a loss function to train the GCN on the fused features and the consensus graph that integrates graph auto-encoder loss and label smoothing over the consensus graph. The results of our experiments on six multi-view datasets show that our WFGSC performs well on both fused graph construction and semi-supervised classification, and generally outperforms traditional GCNs and other multi-view semi-supervised multi-view classification methods.11Source code: https://github.com/BiJingjun/WFGSC.

Graph-Based Semi-Supervised Learning: A Comprehensive Review.

Semi-supervised learning (SSL) has tremendous value in practice due to the utilization of both labeled and unlabelled data. An essential class of SSL methods, referred to as graph-based semi-supervised learning (GSSL) methods in the literature, is to first represent each sample as a node in an affinity graph, and then, the label information of unlabeled samples can be inferred based on the structure of the constructed graph. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large-scale data. Focusing on GSSL methods only, this work aims to provide both researchers and practitioners with a solid and systematic understanding of relevant advances as well as the underlying connections among them. The concentration on one class of SSL makes this article distinct from recent surveys that cover a more general and broader picture of SSL methods yet often neglect the fundamental understanding of GSSL methods. In particular, a significant contribution of this article lies in a newly generalized taxonomy for GSSL under the unified framework, with the most up-to-date references and valuable resources such as codes, datasets, and applications. Furthermore, we present several potential research directions as future work with our insights into this rapidly growing field.

Non-Intrusive Load Monitoring Method for Multi-Energy Coupling Appliances Considering Spatio-Temporal Coupling

The multi-energy coupling (MEC) appliances couple electricity, gas, heat, and cold, of which the gas, heat, and cold have long transient processes. However, these transient features have never been considered in past non-intrusive load monitoring (NILM) methods. This paper proposes a novel NILM method for MEC appliances considering spatio-temporal coupling, which combines semi-supervised learning and weakly supervised learning. Firstly, the unlabeled historical events are labeled by graph-based semi-supervised learning (GBSSL) and distinguished into three categories, i.e., reliable, uncertain, and unreliable samples. Then, to fully use of spatio-temporal coupling information, the unreliable samples are discarded and the feature vectors of other samples are expanded into spatio-temporal coupling feature vectors. To improve the accuracy of load identification, this paper proposes a two-stage deep learning framework “Teaching and Mutual Learning" (TML). In the teaching stage, two convolutional neural networks (CNN) are pre-trained with reliable samples. In the mutual learning stage, the noise samples are filtered by using the different learning abilities of the two networks. The effectiveness and superiority of the proposed method are verified on five types of MEC appliances in a park IES.

A unified semi-supervised model with joint estimation of graph, soft labels and latent subspace

Since manually labeling images is expensive and labor intensive, in practice we often do not have enough labeled images to train an effective classifier for the new image classification tasks. The graph-based SSL methods have received more attention in practice due to their convexity, scalability and efficiency. In this paper, we propose a novel graph-based semi-supervised learning method that takes full advantage of a small set of labeled graphs and a large set of unlabeled graph data. First, we explain the concept of graph-based semi-supervised learning. The core idea of these models is to jointly estimate a low-rank graph with soft labels and a latent subspace. The proposed scheme leverages the synergy between the graph structure and the data representation in terms of soft labels and latent features. This improves the monitoring information and leads to better discriminative linear transformation. Several experiments were conducted on five image datasets using state-of-the-art methods. These experiments show the effectiveness of the proposed semi-supervised method.

Intelligent traffic light systems using edge flow predictions

In this paper, we propose a novel graph-based semi-supervised learning approach for traffic light management in multiple intersections. Specifically, the basic premise behind our paper is that if we know some of the occupied roads and predict which roads will be congested, we can dynamically change traffic lights at the intersections that are connected to the roads anticipated to be congested. Comparative performance evaluations show that the proposed approach can produce comparable average vehicle waiting time and reduce the training/learning time of learning adequate traffic light configurations for all intersections within a few seconds, while a deep learning-based approach can be trained in a few days for learning similar light configurations.

Machine Learning Methods for Small Data Challenges in Molecular Science.

Small data are often used in scientific and engineering research due to the presence of various constraints, such as time, cost, ethics, privacy, security, and technical limitations in data acquisition. However, big data have been the focus for the past decade, small data and their challenges have received little attention, even though they are technically more severe in machine learning (ML) and deep learning (DL) studies. Overall, the small data challenge is often compounded by issues, such as data diversity, imputation, noise, imbalance, and high-dimensionality. Fortunately, the current big data era is characterized by technological breakthroughs in ML, DL, and artificial intelligence (AI), which enable data-driven scientific discovery, and many advanced ML and DL technologies developed for big data have inadvertently provided solutions for small data problems. As a result, significant progress has been made in ML and DL for small data challenges in the past decade. In this review, we summarize and analyze several emerging potential solutions to small data challenges in molecular science, including chemical and biological sciences. We review both basic machine learning algorithms, such as linear regression, logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), kernel learning (KL), random forest (RF), and gradient boosting trees (GBT), and more advanced techniques, including artificial neural network (ANN), convolutional neural network (CNN), U-Net, graph neural network (GNN), Generative Adversarial Network (GAN), long short-term memory (LSTM), autoencoder, transformer, transfer learning, active learning, graph-based semi-supervised learning, combining deep learning with traditional machine learning, and physical model-based data augmentation. We also briefly discuss the latest advances in these methods. Finally, we conclude the survey with a discussion of promising trends in small data challenges in molecular science.

A multi-sensor fusion-based prognostic model for systems with partially observable failure modes

With the rapid development of sensor and communication technology, multi-sensor data is available to monitor the degradation of complex systems and predict the failure modes. However, two huge challenges remain to be resolved: (i) how to predict the failure modes with limited failure mode labeled systems to alleviate the heavy dependence on expert experience; (ii) how to effectively fuze the useful information from the multi-sensor data to achieve an accurate estimation of the degradation status automatically. To address these issues, we propose a novel semi-supervised prognostic model for the systems with partially observable failure modes, where only a small fraction of the systems in the training set are known for their failure modes. First, we develop a graph-based semi-supervised learning method to extract features characterizing the failure modes. Then, we input these features as well as the multi-sensor streams into an elastic net functional regression model to predict the residual useful lifetime. The proposed model is validated by extensive simulation studies and a case study of aircraft turbofan engines available from the NASA repository.

Development of complemented comprehensive networks for rapid screening of repurposable drugs applicable to new emerging disease outbreaks

BackgroundComputational drug repurposing is crucial for identifying candidate therapeutic medications to address the urgent need for developing treatments for newly emerging infectious diseases. The recent COVID-19 pandemic has taught us the importance of rapidly discovering candidate drugs and providing them to medical and pharmaceutical experts for further investigation. Network-based approaches can provide repurposable drugs quickly by leveraging comprehensive relationships among biological components. However, in a case of newly emerging disease, applying a repurposing methods with only pre-existing knowledge networks may prove inadequate due to the insufficiency of information flow caused by the novel nature of the disease.MethodsWe proposed a network-based complementary linkage method for drug repurposing to solve the lack of incoming new disease-specific information in knowledge networks. We simulate our method under the controlled repurposing scenario that we faced in the early stage of the COVID-19 pandemic. First, the disease-gene-drug multi-layered network was constructed as the backbone network by fusing comprehensive knowledge database. Then, complementary information for COVID-19, containing data on 18 comorbid diseases and 17 relevant proteins, was collected from publications or preprint servers as of May 2020. We estimated connections between the novel COVID-19 node and the backbone network to construct a complemented network. Network-based drug scoring for COVID-19 was performed by applying graph-based semi-supervised learning, and the resulting scores were used to validate prioritized drugs for population-scale electronic health records-based medication analyses.ResultsThe backbone networks consisted of 591 diseases, 26,681 proteins, and 2,173 drug nodes based on pre-pandemic knowledge. After incorporating the 35 entities comprised of complemented information into the backbone network, drug scoring screened top 30 potential repurposable drugs for COVID-19. The prioritized drugs were subsequently analyzed in electronic health records obtained from patients in the Penn Medicine COVID-19 Registry as of October 2021 and 8 of these were found to be statistically associated with a COVID-19 phenotype.ConclusionWe found that 8 of the 30 drugs identified by graph-based scoring on complemented networks as potential candidates for COVID-19 repurposing were additionally supported by real-world patient data in follow-up analyses. These results show that our network-based complementary linkage method and drug scoring algorithm are promising strategies for identifying candidate repurposable drugs when new emerging disease outbreaks.

Joint Latent Space and Label Inference Estimation with Adaptive Fused Data and Label Graphs

Recently, structured computing has become an interesting topic in the world of artificial intelligence, especially in the field of machine learning, as most researchers focus on the development of graph-based semi-supervised learning models. In this article, we present a new framework for graph-based semi-supervised learning. We present a powerful method for simultaneous label inference and linear transform estimation. The targeted linear transformation is used to obtain a discriminant subspace. To improve semi-supervised learning, our framework focuses on exploiting the data structure and soft labels of the available unlabeled samples. In the iterative optimization scheme used, the prior estimation of the label increases the supervision information indirectly through an introduced informative matrix called the label graph, thus avoiding the use of hard confidence-based decisions as used in self-supervised methods. In addition, the estimation of labels and projected data is made more robust by using smoothing concepts based on hybrid graphs. For each type of smoothing, the hybrid graph is an adaptive fusion of the two graphs encoding the similarity of the data and the similarity of the labels. The proposed method leads to an improved discriminative linear transformation. Several experimental results on real image datasets confirm the effectiveness of the proposed method. They also show superior performance compared to semi-supervised methods that use integration and label inference simultaneously.

Improving an Electronic Health Record-Based Clinical Prediction Model Under Label Deficiency: Network-Based Generative Adversarial Semisupervised Approach.

Observational biomedical studies facilitate a new strategy for large-scale electronic health record (EHR) utilization to support precision medicine. However, data label inaccessibility is an increasingly important issue in clinical prediction, despite the use of synthetic and semisupervised learning from data. Little research has aimed to uncover the underlying graphical structure of EHRs. A network-based generative adversarial semisupervised method is proposed. The objective is to train clinical prediction models on label-deficient EHRs to achieve comparable learning performance to supervised methods. Three public data sets and one colorectal cancer data set gathered from the Second Affiliated Hospital of Zhejiang University were selected as benchmarks. The proposed models were trained on 5% to 25% labeled data and evaluated on classification metrics against conventional semisupervised and supervised methods. The data quality, model security, and memory scalability were also evaluated. The proposed method for semisupervised classification outperforms related semisupervised methods under the same setup, with the average area under the receiver operating characteristics curve (AUC) reaching 0.945, 0.673, 0.611, and 0.588 for the four data sets, respectively, followed by graph-based semisupervised learning (0.450, 0.454, 0.425, and 0.5676, respectively) and label propagation (0.475,0.344, 0.440, and 0.477, respectively). The average classification AUCs with 10% labeled data were 0.929, 0.719, 0.652, and 0.650, respectively, comparable to that of the supervised learning methods logistic regression (0.601, 0.670, 0.731, and 0.710, respectively), support vector machines (0.733, 0.720, 0.720, and 0.721, respectively), and random forests (0.982, 0.750, 0.758, and 0.740, respectively). The concerns regarding the secondary use of data and data security are alleviated by realistic data synthesis and robust privacy preservation. Training clinical prediction models on label-deficient EHRs is indispensable in data-driven research. The proposed method has great potential to exploit the intrinsic structure of EHRs and achieve comparable learning performance to supervised methods.

Robust deep semi-supervised learning with label propagation and differential privacy

Semi-supervised learning (SSL) methods provide a powerful tool for utilizing abundant unlabeled data to strengthen standard supervised learning. Traditional graph-based SSL methods prevail in classical SSL problems for their intuitional implementation and effective performance. However, they encounter troubles when applying to image classification followed by modern deep learning, since the diffusion algorithms face the curse of dimensionality. In this study, we propose a simple and efficient SSL method, combining a graph-based SSL paradigm with differential privacy. We aim at developing coherent latent feature space of deep neural networks so that the diffusion algorithm in the latent space can give more precise predictions for unlabeled data. Our approach achieves state-of-the-art performance on the Cifar10, Cifar100, and Mini-imagenet benchmark datasets and obtains an error rate of 18.56% on Cifar10 using only 1% of all labels. Furthermore, our approach inherits the benefits of graph-based SSL methods with a simple training process and can be easily combined with any network architecture.

LoyalDE: Improving the performance of Graph Neural Networks with loyal node discovery and emphasis

Recent years have witnessed an increasing focus on graph-based semi-supervised learning with Graph Neural Networks (GNNs). Despite existing GNNs having achieved remarkable accuracy, research on the quality of graph supervision information has inadvertently been ignored. In fact, there are significant differences in the quality of supervision information provided by different labeled nodes, and treating supervision information with different qualities equally may lead to sub-optimal performance of GNNs. We refer to this as the graph supervision loyalty problem, which is a new perspective for improving the performance of GNNs. In this paper, we devise FT-Score to quantify node loyalty by considering both the local feature similarity and the local topology similarity, and nodes with higher loyalty are more likely to provide higher-quality supervision. Based on this, we propose LoyalDE (Loyal Node Discovery and Emphasis), a model-agnostic hot-plugging training strategy, which can discover potential nodes with high loyalty to expand the training set, and then emphasize nodes with high loyalty during model training to improve performance. Experiments demonstrate that the graph supervision loyalty problem will fail most existing GNNs. In contrast, LoyalDE brings about at most 9.1% performance improvement to vanilla GNNs and consistently outperforms several state-of-the-art training strategies for semi-supervised node classification.