Semi-supervised Learning Framework Research Articles

A Dual-channel Semi-supervised Learning Framework on Graphs via Knowledge Transfer and Meta-learning

This article studies the problem of semi-supervised learning on graphs, which aims to incorporate ubiquitous unlabeled knowledge (e.g., graph topology, node attributes) with few-available labeled knowledge (e.g., node class) to alleviate the scarcity issue of supervised information on node classification. While promising results are achieved, existing works for this problem usually suffer from the poor balance of generalization and fitting ability due to the heavy reliance on labels or task-agnostic unsupervised information. To address the challenge, we propose a dual-channel framework for semi-supervised learning on G raphs via K nowledge T ransfer between independent supervised and unsupervised embedding spaces, namely, GKT. Specifically, we devise a dual-channel framework including a supervised model for learning the label probability of nodes and an unsupervised model for extracting information from massive unlabeled graph data. A knowledge transfer head is proposed to bridge the gap between the generalization and fitting capability of the two models. We use the unsupervised information to reconstruct batch-graphs to smooth the label probability distribution on the graphs to improve the generalization of prediction. We also adaptively adjust the reconstructed graphs by encouraging the label-related connections to solidify the fitting ability. Since the optimization of the supervised channel with knowledge transfer contains that of the unsupervised channel as a constraint and vice versa, we then propose a meta-learning-based method to solve the bi-level optimization problem, which avoids the negative transfer and further improves the model’s performance. Finally, extensive experiments validate the effectiveness of our proposed framework by comparing state-of-the-art algorithms.

DAHGN: Degree-Aware Heterogeneous Graph Neural Network

In recent years, Graph Neural Networks (GNNs), an emerging technology for learning from graph-structured data, have attracted much attention. Despite the widespread applications of GNNs across diverse domains, recent studies have shown that insufficient information aggregation for low-degree nodes (i.e., tail nodes) leads to inferior performance of GNNs. While numerous recent studies have aimed to address the issue of degree bias in homogeneous graphs, there are limited studies on analyzing such unfairness for heterogeneous information networks. In this article, we propose a novel semi-supervised contrastive learning framework named Degree-Aware Heterogeneous Graph Neural Network (DAHGN), which achieves promising performance by alleviating the degree bias. In brief, we first construct two contrastive graph views, one from a heterogeneous graph that focuses on capturing complementary information to enhance the node representations and one from an extracted homogeneous subgraph that focuses on applying different strategies to low- and high- degree nodes to eliminate the degree bias. We then leverage the label learned from the heterogeneous view to reconstruct the positive node pairs in the contrastive learning framework. Finally, we combine the task loss (semi-supervised) and the contrastive loss to optimize the model performance. We conduct extensive experiments on benchmark datasets to validate the effectiveness of our model, and the results demonstrate that the DAHGN outperforms a wide range of state-of-the-art methods in the semi-supervised node classification problem.

Deep temporal clustering features for speech emotion recognition

Deep clustering is a popular unsupervised technique for feature representation learning. We recently proposed the chunk-based DeepEmoCluster framework for speech emotion recognition (SER) to adopt the concept of deep clustering as a novel semi-supervised learning (SSL) framework, which achieved improved recognition performances over conventional reconstruction-based approaches. However, the vanilla DeepEmoCluster lacks critical sentence-level temporal information that is useful for SER tasks. This study builds upon the DeepEmoCluster framework, creating a powerful SSL approach that leverages temporal information within a sentence. We propose two sentence-level temporal modeling alternatives using either the temporal-net or the triplet loss function, resulting in a novel temporal-enhanced DeepEmoCluster framework to capture essential temporal information. The key contribution to achieving this goal is the proposed sentence-level uniform sampling strategy, which preserves the original temporal order of the data for the clustering process. An extra network module (e.g., gated recurrent unit) is utilized for the temporal-net option to encode temporal information across the data chunks. Alternatively, we can impose additional temporal constraints by using the triplet loss function while training the DeepEmoCluster framework, which does not increase model complexity. Our experimental results based on the MSP-Podcast corpus demonstrate that the proposed temporal-enhanced framework significantly outperforms the vanilla DeepEmoCluster framework and other existing SSL approaches in regression tasks for the emotional attributes arousal, dominance, and valence. The improvements are observed in fully-supervised learning or SSL implementations. Further analyses validate the effectiveness of the proposed temporal modeling, showing (1) high temporal consistency in the cluster assignment, and (2) well-separated emotional patterns in the generated clusters.

Semi-Supervised Learning With Heterogeneous Distribution Consistency for Visible Infrared Person Re-Identification.

Visible infrared person re-identification (VI-ReID) exposes considerable challenges because of the modality gaps between the person images captured by daytime visible cameras and nighttime infrared cameras. Several fully-supervised VI-ReID methods have improved the performance with extensive labeled heterogeneous images. However, the identity of the person is difficult to obtain in real-world situations, especially at night. Limited known identities and large modality discrepancies impede the effectiveness of the model to a great extent. In this paper, we propose a novel Semi-Supervised Learning framework with Heterogeneous Distribution Consistency (HDC-SSL) for VI-ReID. Specifically, through investigating the confidence distribution of heterogeneous images, we introduce a Gaussian Mixture Model-based Pseudo Labeling (GMM-PL) method, which adaptively adjusts different thresholds for each modality to label the identity. Moreover, to facilitate the representation learning of unutilized data whose prediction is lower than the threshold, Modality Consistency Regularization (MCR) is proposed to ensure the prediction consistency of the cross-modality pedestrian images and handle the modality variance. Extensive experiments with different label settings on two VI-ReID datasets demonstrate the effectiveness of our method. Particularly, HDC-SSL achieves competitive performance with state-of-the-art fully-supervised VI-ReID methods on RegDB dataset with only 1 visible label and 1 infrared label per class.

OTAMatch: Optimal Transport Assignment with PseudoNCE for Semi-supervised Learning.

In semi-supervised learning (SSL), many approaches follow the effective self-training paradigm with consistency regularization, utilizing threshold heuristics to alleviate label noise. However, such threshold heuristics lead to the underutilization of crucial discriminative information from the excluded data. In this paper, we present OTAMatch, a novel SSL framework that reformulates pseudo-labeling as an optimal transport (OT) assignment problem and simultaneously exploits data with high confidence to mitigate the confirmation bias. Firstly, OTAMatch models the pseudo-label allocation task as a convex minimization problem, facilitating end-to-end optimization with all pseudo-labels and employing the Sinkhorn-Knopp algorithm for efficient approximation. Meanwhile, we incorporate epsilon-greedy posterior regularization and curriculum bias correction strategies to constrain the distribution of OT assignments, improving the robustness with noisy pseudo-labels. Secondly, we propose PseudoNCE, which explicitly exploits pseudo-label consistency with threshold heuristics to maximize mutual information within self-training, significantly boosting the balance of convergence speed and performance. Consequently, our proposed approach achieves competitive performance on various SSL benchmarks. Specifically, OTAMatch substantially outperforms the previous state-of-the-art SSL algorithms in realistic and challenging scenarios, exemplified by a notable 9.45% error rate reduction over SoftMatch on ImageNet with 100K-label split, underlining its robustness and effectiveness.

SSwsrNet: A Semi-Supervised Few-Shot Learning Framework for Wireless Signal Recognition

SSwsrNet: A Semi-Supervised Few-Shot Learning Framework for Wireless Signal Recognition

Machine learning-based ransomware classification of Bitcoin transactions

Ransomware presents a significant threat to the security and integrity of cryptocurrency transactions. This research paper explores the intricacies of ransomware detection in cryptocurrency transactions using the Bitcoinheist dataset. The dataset encompasses 28 distinct families classified into three ransomware categories: Princeton, Montreal, and Padua, along with a white category representing legitimate transactions. We propose a novel hybrid supervised and semi-supervised multistage machine learning framework to tackle this challenge. Our framework effectively classifies known ransomware families by leveraging ensemble learning techniques such as Decision Tree, Random Forest, XGBoost, and Stacking. Additionally, we introduce a novel semi-supervised approach to accurately identify previously unseen ransomware instances within the dataset. Through rigorous evaluation employing comprehensive classification metrics, including accuracy, precision, recall, F1 score, RoC score, and prediction time, our proposed approach demonstrates promising results in ransomware detection within cryptocurrency transactions.

Deep Semi-Supervised Learning Method for False Data Detection Against Forgery and Concealing of Faults in Cyber-Physical Power Systems

This paper proposes a deep semi-supervised numerical false data detection (DSS-NFDD) method, where hidden data in labeled and unlabeled datasets are leveraged simultaneously, to detect the false data in CPPS. Two types of false data are considered in this study: one is the forged fault data, which makes operators mistakenly believe that there is a fault in their operating system; the other is the false data used to conceal actual faults. First, a data dimension reduction method is proposed based on the PageRank algorithm to avoid the excessive noise caused by high-dimension data. Then, a semi-supervised deep learning framework is established to detect false data samples, which consists of two parts: one is the priori estimation module, and the other is a false data scoring network. A novel concentration loss function is presented for training the false data scoring network, which minimizes the impacts of noise pollution and sample bias. Next, a false feature location method is proposed to help human operators eliminate anomalies. Finally, the effectiveness and the superiorities of the proposed DSS-NFDD method are verified by analyzing the simulation results for the IEEE-39 bus and IEEE-118 bus systems.

Robust object detection for autonomous driving based on semi-supervised learning

Deep learning based on labeled data has brought massive success in computer vision, speech recognition, and natural language processing. Nevertheless, labeled data is just a drop in the ocean compared with unlabeled data. How can people utilize the unlabeled data effectively? Research has focused on unsupervised and semi-supervised learning to solve such a problem. Some theoretical and empirical studies have proved that unlabeled data can help boost the generalization ability and robustness under adversarial attacks. However, current theoretical research on the relationship between robustness and unlabeled data limits its scope to toy datasets. Meanwhile, the visual models in autonomous driving need a significant improvement in robustness to guarantee security and safety. This paper proposes a semi-supervised learning framework for object detection in autonomous vehicles, improving the robustness with unlabeled data. Firstly, we build a baseline with the transfer learning of an unsupervised contrastive learning method—Momentum Contrast (MoCo). Secondly, we propose a semi-supervised co-training method to label the unlabeled data for retraining, which improves generalization on the autonomous driving dataset. Thirdly, we apply the unsupervised Bounding Box data augmentation (BBAug) method based on a search algorithm, which uses reinforcement learning to improve the robustness of object detection for autonomous driving. We present an empirical study on the KITTI dataset with diverse adversarial attack methods. Our proposed method realizes the state-of-the-art generalization and robustness under white-box attacks (DPatch and Contextual Patch) and black-box attacks (Gaussian noise, Rain, Fog, and so on). Our proposed method and empirical study show that using more unlabeled data benefits the robustness of perception systems in autonomous driving.

A Review of Semi-Supervised Learning Frameworks for Semantic Segmentation of Remote Sensing Images

A Review of Semi-Supervised Learning Frameworks for Semantic Segmentation of Remote Sensing Images

RSSL: A Unified Reliability-Aware Semi-Supervised Learning Framework for Medical Image Classification

RSSL: A Unified Reliability-Aware Semi-Supervised Learning Framework for Medical Image Classification

Interactive semi-supervised learning for microarray analysis

Microarray technology has generated vast amounts of gene expression data with distinct patterns. Based on the premise that genes of correlated functions tend to exhibit similar expression patterns, various machine learning methods have been applied to capture these specific patterns in microarray data. However, the discrepancy between the rich expression profiles and the limited knowledge of gene functions has been a major hurdle to the understanding of cellular networks. To bridge this gap so as to properly comprehend and interpret expression data, we introduce Relevance Feedback to microarray analysis and propose an interactive learning framework to incorporate the expert knowledge into the decision module. In order to find a good learning method and solve two intrinsic problems in microarray data: high dimensionality and small sample size, we also propose a semi-supervised learning algorithm: Kernel Discriminant-EM (KDEM). This algorithm efficiently utilizes a large set of unlabeled data to compensate for the insufficiency of a small set of labeled data and it extends linear algorithm in Discriminant-EM (DEM) to kernel algorithm to handle nonlinearly separable data in a lower dimensional space. Relevance Feedback technique and KDEM together construct an efficient and effective interactive semi-supervised learning framework for microarray analysis. Extensive experiments on the yeast cell cycle regulation data set and Plasmodium falciparum red blood cell cycle data set show the promise of this approach.

Semi-Supervised Learning Under General Causal Models.

Semi-supervised learning (SSL) aims to train a machine learning (ML) model using both labeled and unlabeled data. While the unlabeled data have been used in various ways to improve the prediction accuracy, the reason why unlabeled data could help is not fully understood. One interesting and promising direction is to understand SSL from a causal perspective. In light of the independent causal mechanisms (ICM) principle, the unlabeled data can be helpful when the label causes the features but not vice versa. However, the causal relations between the features and labels can be complex in real world applications. In this article, we propose an SSL framework that works with general causal models in which the variables have flexible causal relations. More specifically, we explore the causal graph structures and design corresponding causal generative models which can be learned with the help of unlabeled data. The learned causal generative model can generate synthetic labeled data for training a more accurate predictive model. We verify the effectiveness of our proposed method by empirical studies on both simulated and real data.

DSST: A dual student model guided student–teacher framework for semi-supervised medical image segmentation

Numerous semi-supervised learning methods are emerging in medical image segmentation to reduce the dependency of deep learning models on pixel-level annotation data. Consistency regularization methods based on the Student–Teacher structure have achieved brilliant results in this domain. However, the current structures are unable to resolve the tight weight coupling satisfactorily between the teacher and student model, which leads to a decrease in the segmentation performance. In this paper, we propose a novel and practical semi-supervised learning framework, Dual-Student-Single-Teacher (DSST), to alleviate this problem. Particularly, the DSST framework consists of three segmentation models with the identical structure but different initial parameters, one serves as the teacher model and others as the student models, which employs an alternating manner to update the teacher model parameters. For the DSST framework, we present different supervised modes to sufficiently explore the enhancement of consistency regularization for model segmentation performance. Furthermore, we also introduce abundant and efficient input and feature perturbations for the proposed method. Finally, we evaluate our framework in three public medical image segmentation tasks, including Pancreas-CT, LA dataset, and cardiac segmentation on the ACDC dataset. Extensive experiments demonstrate that, compared with eight other superior semi-supervised methods, the DSST method obtains state-of-the-art segmentation performance and is an effective and generalizable framework. Code is available at https://github.com/LBL0704/DSST.

Semi-supervised Learning for Automatic Modulation Recognition Using Haar Time–Frequency Mask and Positional–Spatial Attention

Automatic modulation recognition plays an important role in many military and civilian applications, including cognitive radio, spectrum sensing, signal surveillance, and interference identification. Due to the powerful ability of deep learning to extract hidden features and perform classification, it can extract highly separative features from massive signal samples. Considering the condition of limited training samples, we propose a semi-supervised learning framework based on Haar time–frequency (HTF) mask data augmentation and the positional–spatial attention (PSA) mechanism. Specifically, the HTF mask is designed to increase data diversity, and the PSA is designed to address the limited receptive field of the convolutional layer and enhance the feature extraction capability of the constructed network. Extensive experimental results obtained on the public RML2016.10a dataset show that the proposed semi-supervised framework utilizes 1% of the given labeled data and reaches a recognition accuracy of 92.09% under 6 dB signals.

A review on semi-supervised learning for EEG-based emotion recognition

Semisupervised learning holds significant academic and practical importance in the realm of EEG-based emotion recognition. Currently, a multitude of research endeavors are dedicated to addressing the challenge of emotion recognition, especially in light of the growing popularity of deep learning. Within this context, semisupervised learning has emerged as an innovative and promising methodological trend. This article seeks to contribute a comprehensive review that captures the research landscape, distills essential technologies, and projects the trajectory of semisupervised learning for EEG-based emotion recognition. The review encompasses various critical aspects. Firstly, it introduces EEG-based emotion recognition and underscores the pivotal role of semisupervised learning in enhancing its performance. Secondly, the semisupervised learning framework is analyzed, drawing comparisons with conventional methodologies to highlight its unique advantages. Thirdly, the article delves into the intricacies of the emotion recognition challenge, shedding light on how semisupervised learning can effectively address these complexities. Furthermore, the review synthesizes, categorizes, and provides exemplifications of representative semisupervised learning approaches that are specifically tailored for EEG-based emotion recognition. Finally, the methodological strengths of semisupervised learning are elucidated, accompanied by a discussion of the encountered challenges. The article concludes by projecting the future developmental trajectory of semisupervised learning within the domain of EEG-based emotion recognition, underlining its potential for inspiring innovation and advancements in this field.

Joint data-driven and physics-driven prestack amplitude-variation-with-angle elastic parameter inversion

Elastic parameters such as P-wave and S-wave velocity and density are of great significance for subsurface quantitative interpretation and reservoir prediction. Current prestack amplitude-variation-with-angle (AVA) inversion methods have been widely used in the industry to obtain subsurface elastic parameters. Conventional AVA inversion methods are theoretically based on a linearized amplitude-variation-with-offset model that formulates the relationship between prestack seismic reflection coefficients and the subsurface model of elastic parameters, which is called physical model-driven inversion. However, due to the linearized physical model, it is difficult to obtain high-accuracy and high-resolution inversion results using model-driven inversion when the incident angles are greater than 30°. In recent years, several neural network-based prestack AVA inversion methods, known as data-driven inversion, have been developed to address this issue. However, these methods typically require a large amount of labeled data for training networks, and there is no physical mechanism in the network training process. To solve this problem, a joint data-driven and physics-driven inversion of prestack AVA elastic parameters is proposed. Under the framework of semisupervised learning, a 2D convolutional neural network and a recurrent neural network are used to establish the mapping between several adjacent prestack AVA gathers and 1D elastic parameters in the time domain. The full Zoeppritz equation suitable for large incidence angles is used as a physical constraint for training the neural network. The loss functions of the network are constructed by combing well-log data and prestack AVA seismic data. This approach enables us to use fewer labeled data for training the network and increases the physical interpretability of the training process. To further improve the inversion results, the inverse distance weighted correlation coefficient of seismic data is applied to the loss function of seismic data and well-log data. Synthetic and field data examples demonstrate that the joint data-driven and physics-driven prestack AVA elastic parameter inversion improves accuracy and resolution compared with conventional model-driven inversion.

A semi-supervised deep learning approach for cropped image detection

Deep learning algorithms have shown great performance in multimedia forensics applications using supervised learning on large-scale labeled datasets. However, constructing such extensive labeled datasets can be challenging and costly in several multimedia forensics scenarios. Additionally, heavyweight deep learning models with complex architectures and a large number of parameters require significant hardware resources for training. To address these challenges in the context of image cropping detection, a common multimedia forensics application, we propose a semi-supervised deep learning framework capable of training on a large amount of unlabeled image samples. In this framework, we leverage a teacher model, trained on a small set of labeled image samples, to rank the confidence scores of image samples in a large-scale unlabeled dataset. By utilizing the ranked image samples, we train a student network successfully. To validate the effectiveness of our collaborative training framework across various image cropping detection scenarios, we conduct extensive experiments on a large-scale dataset. The experimental results clearly demonstrate that our semi-supervised learning approach achieved a state-of-the-art performance compared to existing supervised detection frameworks, achieving an accuracy of 91.79% on the BOSSbase dataset and 89.23% on the Alaska dataset. Furthermore, we conducted in-depth research on various factors that influence detection performance in the context of semi-supervised learning. These factors include pairings of teacher–student models, the top-K selection approach, the number of unlabeled samples, the number of iterations in self-training, and the proportion of high-confidence samples using in semi-supervised learning.

A deep semi-supervised learning framework towards multi-output soft sensors development and applications in wastewater treatment processes

Soft sensors have emerged as a powerful tool for predicting quality-related but hard-to-measured variables in the wastewater treatment plants (WWTPs). However, due to high data dimensions and insufficient training data, many existing soft sensors usually suffer from low accuracy and degradation, and are unable to meet the field measurement requirements. To enhance the prediction performance and efficiency, this paper proposed a deep semi-supervised learning framework towards multi-output soft sensors in wastewater treatment processes. Firstly, a deep learning network is constructed by integrating stacked autoencoders with a multi-output neural network (SAE-MNN), which is used to eliminate redundant information from raw data and to predict multiple quality-related variables simultaneously. Secondly, the semi-supervised Co-training learning is extended to the multi-output systems and employed to address the challenge of insufficient training data. Two datasets collected from different WWTPs are provided to the effectiveness of the Co-training SAE-MNN framework, in terms of RMSSD values of 0.1073 and 0.1829, and the time consumption of 3.42 s and 12.54 s, respectively. The results demonstrate that the proposed soft sensor provides a reliable tool for multivariate prediction in WWTPs, and contributes to controlling detection costs and increasing wastewater treatment efficiency.

A Semi-Supervised Deep Learning Framework for Change Detection in Open-Pit Mines Using SAR Imagery

Detecting and monitoring changes in open-pit mines is crucial for efficient mining operations. Indeed, these changes comprise a broad spectrum of activities that can often lead to significant environmental impacts such as surface damage, air pollution, soil erosion, and ecosystem degradation. Conventional optical sensors face limitations due to cloud cover, hindering accurate observation of the mining area. To overcome this challenge, synthetic aperture radar (SAR) images have emerged as a powerful solution, due to their unique ability to penetrate clouds and provide a clear view of the ground. The open-pit mine change detection task presents significant challenges, justifying the need for a model trained for this specific task. First, different mining areas frequently include various features, resulting in a diverse range of land cover types within a single scene. This heterogeneity complicates the detection and distinction of changes within open-pit mines. Second, pseudo changes, e.g., equipment movements or humidity fluctuations, which show statistically reliable reflectivity changes, lead to false positives, as they do not directly correspond to the actual changes of interest, i.e., blasting, collapsing, or waste pile operations. In this paper, to the best of our knowledge, we present the first deep learning model in the literature that can accurately detect changes within open-pit mines using SAR images (TerraSAR-X). We showcase the fundamental role of data augmentations and a coherence layer as a critical component in enhancing the model’s performance, which initially relied solely on amplitude information. In addition, we demonstrate how, in the presence of a few labels, a pseudo-labeling pipeline can improve the model robustness, without degrading the performance by introducing misclassification points related to pseudo changes. The F1-Score results show that our deep learning approach is a reliable and effective method for SAR change detection in the open-pit mining sector.