Semi-supervised Learning Algorithm Research Articles

MF-Match: A Semi-Supervised Model for Human Action Recognition

Human action recognition (HAR) technology based on radar signals has garnered significant attention from both industry and academia due to its exceptional privacy-preserving capabilities, noncontact sensing characteristics, and insensitivity to lighting conditions. However, the scarcity of accurately labeled human radar data poses a significant challenge in meeting the demand for large-scale training datasets required by deep model-based HAR technology, thus substantially impeding technological advancements in this field. To address this issue, a semi-supervised learning algorithm, MF-Match, is proposed in this paper. This algorithm computes pseudo-labels for larger-scale unsupervised radar data, enabling the model to extract embedded human behavioral information and enhance the accuracy of HAR algorithms. Furthermore, the method incorporates contrastive learning principles to improve the quality of model-generated pseudo-labels and mitigate the impact of mislabeled pseudo-labels on recognition performance. Experimental results demonstrate that this method achieves action recognition accuracies of 86.69% and 91.48% on two widely used radar spectrum datasets, respectively, utilizing only 10% labeled data, thereby validating the effectiveness of the proposed approach.

Enhancing identification performance of cognitive impairment high-risk based on a semi-supervised learning method

BackgroundCognitive assessment plays a pivotal role in the early detection of cognitive impairment, particularly in the prevention and management of cognitive diseases such as Alzheimer’s and Lewy body dementia. Large-scale screening relies heavily on cognitive assessment scales as primary tools, with some low sensitivity and others expensive. Despite significant progress in machine learning for cognitive function assessment, its application in this particular screening domain remains underexplored, often requiring labor-intensive expert annotations. AimsThis paper introduces a semi-supervised learning algorithm based on pseudo-label with putback (SS-PP), aiming to enhance model efficiency in predicting the high risk of cognitive impairment (HR-CI) by utilizing the distribution of unlabeled samples. DataThe study involved 189 labeled samples and 215,078 unlabeled samples from real world. A semi-supervised classification algorithm was designed and evaluated by comparison with supervised methods composed by 14 traditional machine-learning methods and other advanced semi-supervised algorithms. ResultsThe optimal SS-PP model, based on GBDT, achieved an AUC of 0.947. Comparative analysis with supervised learning models and semi-supervised methods demonstrated an average AUC improvement of 8% and state-of-art performance, repectively. ConclusionThis study pioneers the exploration of utilizing limited labeled data for HR-CI predictions and evaluates the benefits of incorporating physical examination data, holding significant implications for the development of cost-effective strategies in relevant healthcare domains.

Clustering Network Traffic Using Semi-Supervised Learning

Clustering algorithms play a crucial role in early warning cybersecurity systems. They allow for the detection of new attack patterns and anomalies and enhance system performance. This paper discusses the problem of clustering data collected by a distributed system of network honeypots. In the proposed approach, when a network flow matches an attack signature, an appropriate label is assigned to it. This enables the use of semi-supervised learning algorithms and improves the quality of clustering results. The article compares the results of learning algorithms conducted with and without partial supervision, particularly non-negative matrix factorization and semi-supervised non-negative matrix factorization. Our results confirm the positive impact of labeling a portion of flows on the quality of clustering.

CPSS: Fusing consistency regularization and pseudo-labeling techniques for semi-supervised deep cardiovascular disease detection using all unlabeled electrocardiograms

Background and objective:Deep learning usually achieves good performance in the supervised way, which requires a large amount of labeled data. However, manual labeling of electrocardiograms (ECGs) is laborious that requires much medical knowledge. Semi-supervised learning (SSL) provides an effective way of leveraging unlabeled data to improve model performance, providing insight for solving this problem. The objective of this study is to improve the performance of cardiovascular disease (CVD) detection by fully utilizing unlabeled ECG. Methods:A novel SSL algorithm fusing consistency regularization and pseudo-labeling techniques (CPSS) is proposed. CPSS consists of supervised learning and unsupervised learning. For supervised learning, the labeled ECGs are mapped into prediction vectors by the classifier. The cross-entropy loss function is used to optimize the classifier. For unsupervised learning, the unlabeled ECGs are weakly and strongly augmented, and a consistency loss is used to minimize the difference between the classifier’s predictions for the two augmentations. Pseudo-labeling techniques include positive pseudo-labeling (PL) and ranking-based negative pseudo-labeling (RNL). PL introduces pseudo-labels for data with high prediction confidence. RNL assigns negative pseudo-labels to the lower-ranked categories in the prediction vectors to leverage data with low prediction confidence. In this study, VGGNet and ResNet are used as classifiers, which are jointly optimized by labeled and unlabeled ECGs. Results:CPSS has been validated on several databases. With the same number of labeled ECGs (10%), it improves the accuracies over pure supervised learning by 13.59%, 4.60%, and 5.38% in the CPSC2018, PTB-XL, and Chapman databases, respectively. CPSS achieves comparable results to the fully supervised method with only 10% of labeled ECGs, which reduces the labeling workload by 90%. In addition, to verify the practicality of CPSS, a cardiovascular disease monitoring system is designed by heterogeneously deploying the trained classifiers on an SoC (system-on-a-chip), which can detect CVD in real time. Conclusion:The results of this study indicate that the proposed CPSS can significantly improve the performance of CVD detection using unlabeled ECG, which reduces the burden of ECG labeling in deep learning. In addition, the designed monitoring system makes the proposed CPSS promising for real-world applications.

Few-Shot Steel Defect Detection Based on a Fine-Tuned Network with Serial Multi-Scale Attention

Detecting defects on a steel surface is crucial for the quality enhancement of steel, but its effectiveness is impeded by the limited number of high-quality samples, diverse defect types, and the presence of interference factors such as dirt spots. Therefore, this article proposes a fine-tuned deep learning approach to overcome these obstacles in unstructured few-shot settings. Initially, to address steel surface defect complexities, we integrated a serial multi-scale attention mechanism, concatenating attention and spatial modules, to generate feature maps that contain both channel information and spatial information. Further, a pseudo-label semi-supervised learning algorithm (SSL) based on a variant of the locally linear embedding (LLE) algorithm was proposed, enhancing the generalization capability of the model through information from unlabeled data. Afterwards, the refined model was merged into a fine-tuned few-shot object detection network, which applied extensive base class samples for initial training and sparsed new class samples for fine-tuning. Finally, specialized datasets considering defect diversity and pixel scales were constructed and tested. Compared with conventional methods, our approach improved accuracy by 5.93% in 7-shot detection tasks, markedly reducing manual workload and signifying a leap forward for practical applications in steel defect detection.

Triplet Adaptation Framework for Robust Semi-Supervised Learning.

Semi-supervised learning (SSL) suffers from severe performance degradation when labeled and unlabeled data come from inconsistent and imbalanced distribution. Nonetheless, there is a lack of theoretical guidance regarding a remedy for this issue. To bridge the gap between theoretical insights and practical solutions, we embark to an analysis of generalization bound of classic SSL algorithms. This analysis reveals that distribution inconsistency between unlabeled and labeled data can cause a significant generalization error bound. Motivated by this theoretical insight, we present a Triplet Adaptation Framework (TAF) to reduce the distribution divergence and improve the generalization of SSL models. TAF comprises three adapters: Balanced Residual Adapter, aiming to map the class distribution of labeled and unlabeled data to a uniform distribution for reducing class distribution divergence; Representation Adapter, aiming to map the representation distribution of unlabeled data to labeled one for reducing representation distribution divergence; and Pseudo-Label Adapter, aiming to align the predicted pseudo-labels with the class distribution of unlabeled data, thereby preventing erroneous pseudo-labels from exacerbating representation divergence. These three adapters collaborate synergistically to reduce the generalization bound, ultimately achieving a more robust and generalizable SSL model. Extensive experiments across various robust SSL scenarios validate the efficacy of our method.

Developing an atmospheric aging evaluation model of acrylic coatings: A semi-supervised machine learning algorithm

Developing an atmospheric aging evaluation model of acrylic coatings: A semi-supervised machine learning algorithm

Al-BERT: a semi-supervised denoising technique for disease prediction.

Medical records are a valuable source for understanding patient health conditions. Doctors often use these records to assess health without solely depending on time-consuming and complex examinations. However, these records may not always be directly relevant to a patient's current health issue. For instance, information about common colds may not be relevant to a more specific health condition. While experienced doctors can effectively navigate through unnecessary details in medical records, this excess information presents a challenge for machine learning models in predicting diseases electronically. To address this, we have developed 'al-BERT', a new disease prediction model that leverages the BERT framework. This model is designed to identify crucial information from medical records and use it to predict diseases. 'al-BERT' operates on the principle that the structure of sentences in diagnostic records is similar to regular linguistic patterns. However, just as stuttering in speech can introduce 'noise' or irrelevant information, similar issues can arise in written records, complicating model training. To overcome this, 'al-BERT' incorporates a semi-supervised layer that filters out irrelevant data from patient visitation records. This process aims to refine the data, resulting in more reliable indicators for disease correlationsand enhancing the model's predictive accuracy and utility in medical diagnostics. To discern noise diseases within patient records, especially those resembling influenza-like illnesses, our approach employs a customized semi-supervised learning algorithm equipped with a focused attention mechanism. This mechanism is specifically calibrated to enhance the model's sensitivity to chronic conditions while concurrently distilling salient features from patient records, thereby augmenting the predictive accuracy and utility of the model in clinical settings. We evaluate the performance of al-BERT using real-world health insurance data provided by Taiwan's National Health Insurance. In our study, we evaluated our model against two others: one based on BERT that uses complete disease records, and another variant that includes extra filtering techniques. Our findings show that models incorporating filtering mechanisms typically perform better than those using the entire, unfiltered dataset. Our approach resulted in improved outcomes across several key measures: AUC-ROC (an indicator of a model's ability to distinguish between classes), precision (the accuracy of positive predictions), recall (the model's ability to find all relevant cases), and overall accuracy. Most notably, our model showed a 15% improvement in recall compared to the current best-performing method in the field of disease prediction. The conducted ablation study affirms the advantages of our attention mechanism and underscores the crucial role of the selection module within al-BERT.

PiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning.

Partial label learning (PLL) is an important problem that allows each training example to be labeled with a coarse candidate set with the ground-truth label included. However, in a more practical but challenging scenario, the annotator may miss the ground-truth and provide a wrong candidate set, which is known as the noisy PLL problem. To remedy this problem, we propose the PiCO+ framework that simultaneously disambiguates the candidate sets and mitigates label noise. Core to PiCO+, we develop a novel label disambiguation algorithm PiCO that consists of a contrastive learning module along with a novel class prototype-based disambiguation method. Theoretically, we show that these two components are mutually beneficial, and can be rigorously justified from an expectation-maximization (EM) algorithm perspective. To handle label noise, we extend PiCO to PiCO+, which further performs distance-based clean sample selection, and learns robust classifiers by a semi-supervised contrastive learning algorithm. Beyond this, we further investigate the robustness of PiCO+ in the context of out-of-distribution noise and incorporate a novel energy-based rejection method for improved robustness. Extensive experiments demonstrate that our proposed methods significantly outperform the current state-of-the-art approaches in standard and noisy PLL tasks and even achieve comparable results to fully supervised learning.

Triple-task mutual consistency for semi-supervised 3D medical image segmentation

Semi-supervised deep learning algorithm is an effective means of medical image segmentation. Among these methods, multi-task learning with consistency regularization has achieved outstanding results. However, most of the existing methods usually simply embed the Signed Distance Map (SDM) task into the network, which underestimates the potential ability of SDM in edge awareness and leads to excessive dependence between tasks. In this work, we propose a novel triple-task mutual consistency (TTMC) framework to enhance shape and edge awareness capabilities, and overcome the task dependence problem underestimated in previous work. Specifically, we innovatively construct the Signed Attention Map (SAM), a novel fusion image with attention mechanism, and use it as an auxiliary task for segmentation to enhance the edge awareness ability. Then we implement a triple-task deep network, which jointly predicts the voxel-wise classification map, the Signed Distance Map and the Signed Attention Map. In our proposed framework, an optimized differentiable transformation layer associates SDM with voxel-wise classification map and SAM prediction, while task-level consistency regularization utilizes unlabeled data in an unsupervised manner. Evaluated on the public Left Atrium dataset and NIH Pancreas dataset, our proposed framework achieves significant performance gains by effectively utilizing unlabeled data, outperforming recent state-of-the-art semi-supervised segmentation methods. Code is available at https://github.com/Saocent/TTMC.

ABAE: Auxiliary Balanced AutoEncoder for class-imbalanced semi-supervised learning

Semi-supervised learning has achieved extraordinary success in prevalent image-classification benchmarks. However, a class-balanced distribution that differs notably from real-world data distribution is required. In general, models trained under class-imbalanced semi-supervised learning conditions are severely biased towards the majority classes. To address this issue, we propose a novel framework called ABAE by implanting an Auxiliary Balanced AutoEncoder branch into existing semi-supervised learning algorithms. Considering that adaptive feature augmentation for different classes can alleviate confirmation bias, we devise a class-aware reconstruction loss to train the AutoEncoder module. To further smooth the output, we adopt a graph-based label propagation scheme at the end of the AutoEncoder. Extensive experiments on CIFAR-10/100-LT, SVHN-LT and Small ImageNet-127 demonstrate the effectiveness of ABAE.

Leveraging Local Variance for Pseudo-Label Selection in Semi-supervised Learning

Semi-supervised learning algorithms that use pseudo-labeling have become increasingly popular for improving model performance by utilizing both labeled and unlabeled data. In this paper, we offer a fresh perspective on the selection of pseudo-labels, inspired by theoretical insights. We suggest that pseudo-labels with a high degree of local variance are more prone to inaccuracies. Based on this premise, we introduce the Local Variance Match (LVM) method, which aims to optimize the selection of pseudo-labels in semi-supervised learning (SSL) tasks. Our methodology is validated through a series of experiments on widely-used image classification datasets, such as CIFAR-10, CIFAR-100, and SVHN, spanning various labeled data quantity scenarios. The empirical findings show that the LVM method substantially outpaces current SSL techniques, achieving state-of-the-art results in many of these scenarios. For instance, we observed an error rate of 5.41% on CIFAR-10 with a single label for each class, 35.87% on CIFAR-100 when using four labels per class, and 1.94% on SVHN with four labels for each class. Notably, the standout error rate of 5.41% is less than 1% shy of the performance in a fully-supervised learning environment. In experiments on ImageNet with 100k labeled data, the LVM also reached state-of-the-art outcomes. Additionally, the efficacy of the LVM method is further validated by its stellar performance in speech recognition experiments.

A deep semi-supervised machine learning algorithm for detecting transportation modes based on GPS tracking data

AbstractTransportation research has benefited from GPS tracking devices since a higher volume of data can be acquired. Trip information such as travel speed, time, and most visited locations can be easily extracted from raw GPS tracking data. However, transportation modes cannot be extracted directly and require more complex analytical processes. Common approaches for detecting travel modes heavily depend on manual labelling of trajectories with accurate trip information, which is inefficient in many aspects. This paper proposes a method of semi-supervised machine learning by using minimal labelled data. The method can accept GPS trajectory with adjustable length and extract latent information with long short-term memory (LSTM) Autoencoder. The method adopts a deep neural network architecture with three hidden layers to map the latent information to detect transportation mode. The proposed method is assessed by applying it to the case study where an accuracy of 93.94% can be achieved, which significantly outperforms similar studies.

Ensemble methods and semi-supervised learning for information fusion: A review and future research directions

Advances over the past decade at the intersection of information fusion methods and Semi-Supervised Learning (SSL) are investigated in this paper that grapple with challenges related to limited labelled data. To do so, a bibliographic review of papers published since 2013 is presented, in which ensemble methods are combined with new machine learning algorithms. A total of 128 new proposals using SSL algorithms for ensemble construction are identified and classified. All the methods are categorised by approach, ensemble type, and base classifier. Experimental protocols, pre-processing, dataset usage, unlabelled ratios, and statistical tests are also assessed, underlining the major trends, and some shortcomings of particular studies. It is evident from this literature review that foundational algorithms such as self-training and co-training are influencing current developments, and that innovative ensemble techniques are continuing to emerge. Additionally, valuable guidelines are identified in the review for improving research into intrinsically semi-supervised and unsupervised pre-processing methods, especially for regression tasks.

One-Bit Supervision for Image Classification: Problem, Solution, and Beyond

This article presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. Instead of the training model using the accurate label of each sample, our setting requires the model to interact with the system by predicting the class label of each sample and learn from the answer whether the guess is correct, which provides one bit (yes or no) of information. An intriguing property of the setting is that the burden of annotation largely is alleviated in comparison to offering the accurate label. There are two keys to one-bit supervision: (i) improving the guess accuracy and (ii) making good use of the incorrect guesses. To achieve these goals, we propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. Theoretical analysis shows that one-bit annotation is more efficient than full-bit annotation in most cases and gives the conditions of combining our approach with active learning. Inspired by this, we further integrate the one-bit supervision framework into the self-supervised learning algorithm, which yields an even more efficient training schedule. Different from training from scratch, when self-supervised learning is used for initialization, both hard example mining and class balance are verified to be effective in boosting the learning performance. However, these two frameworks still need full-bit labels in the initial stage. To cast off this burden, we utilize unsupervised domain adaptation to train the initial model and conduct pure one-bit annotations on the target dataset. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.

SFWN: A Novel Semi-Supervised Feature Weighted Neural Network for Gene Data Feature Learning and Mining with Graph Modeling.

Gene expression data can serve for analyzing the genes with changed expressions, the correlation between genes and the influence of different circumstance on gene activities. However, labeling a large number of gene expression data is laborious and time-consuming. The insufficient labeled data pose a challenge to construct the deep learning model. Currently, some graph neural networks (GNN) based on semi-supervised learning mechanism only focus on the feature space and sample space of gene expression data, possibly affecting the accuracy. This paper puts forward a novel semi-supervised graph neural network model (SFWN). Firstly, we use the external knowledge of gene expression data for constructing a feature graph, a similarity kernel, and a sample graph for the first time. Later, a novel semi-supervised learning algorithm (SGA) is proposed to extract the data relationship and obtain the global sample structure better. A graph sparse module (SGCN) is also proposed to process sparse representation with gene expression data classification. To overcome the over smoothing problem, a new feature calculation method based on two spaces is proposed to feature representation analysis and calculation in this model. According to a lot of experiments and ablation studies conducted on several public datasets, SFWN exhibits a better effect and is superior to the state-of-the-art approaches (the accuracy and F1-Score are 0.9993 and 0.9899, respectively). Experimental results showed that the proposed SFWN model has strong gene expression feature learning and representation ability, and may provide a new insight and tool for relevant disease diagnosis and clinic practice.

OTAMatch: Optimal Transport Assignment with PseudoNCE for Semi-supervised Learning.

In semi-supervised learning (SSL), many approaches follow the effective self-training paradigm with consistency regularization, utilizing threshold heuristics to alleviate label noise. However, such threshold heuristics lead to the underutilization of crucial discriminative information from the excluded data. In this paper, we present OTAMatch, a novel SSL framework that reformulates pseudo-labeling as an optimal transport (OT) assignment problem and simultaneously exploits data with high confidence to mitigate the confirmation bias. Firstly, OTAMatch models the pseudo-label allocation task as a convex minimization problem, facilitating end-to-end optimization with all pseudo-labels and employing the Sinkhorn-Knopp algorithm for efficient approximation. Meanwhile, we incorporate epsilon-greedy posterior regularization and curriculum bias correction strategies to constrain the distribution of OT assignments, improving the robustness with noisy pseudo-labels. Secondly, we propose PseudoNCE, which explicitly exploits pseudo-label consistency with threshold heuristics to maximize mutual information within self-training, significantly boosting the balance of convergence speed and performance. Consequently, our proposed approach achieves competitive performance on various SSL benchmarks. Specifically, OTAMatch substantially outperforms the previous state-of-the-art SSL algorithms in realistic and challenging scenarios, exemplified by a notable 9.45% error rate reduction over SoftMatch on ImageNet with 100K-label split, underlining its robustness and effectiveness.

Research on Multi-Label Semi-Supervised Learning Algorithm Based on Dual Selection Criteria

Research on Multi-Label Semi-Supervised Learning Algorithm Based on Dual Selection Criteria

Searching to Exploit Memorization Effect in Deep Learning with Noisy Labels.

Sample selection approaches are popular in robust learning from noisy labels. However, how to control the selection process properly so that deep networks can benefit from the memorization effect is a hard problem. In this paper, motivated by the success of automated machine learning (AutoML), we propose to control the selection process by bi-level optimization. Specifically, we parameterize the selection process by exploiting the general patterns of the memorization effect in the upper-level, and then update these parameters using predicting accuracy obtained from model training in the lower-level. We further introduce semi-supervised learning algorithms to utiilize noisy-labeled data as unlabeled data. To solve the bi-level optimization problem efficiently, we consider more information from the validation curvature by the Newton method and cubic regularization method. We provide convergence analysis for both optimization methods. Results show that while both methods can converge to an (approximately) stationary point, the cubic regularization method can find better local optimal than the Newton method with less time. Experiments on both benchmark and real-world data sets demonstrate that the proposed searching method can lead to significant improvements upon existing methods. Compared with existing AutoML approaches, our method is much more efficient on finding a good selection schedule.

AI-Powered Segmentation of Invasive Carcinoma Regions in Breast Cancer Immunohistochemical Whole-Slide Images.

The automation of quantitative evaluation for breast immunohistochemistry (IHC) plays a crucial role in reducing the workload of pathologists and enhancing the objectivity of diagnoses. However, current methods face challenges in achieving fully automated immunohistochemistry quantification due to the complexity of segmenting the tumor area into distinct ductal carcinoma in situ (DCIS) and invasive carcinoma (IC) regions. Moreover, the quantitative analysis of immunohistochemistry requires a specific focus on invasive carcinoma regions. In this study, we propose an innovative approach to automatically identify invasive carcinoma regions in breast cancer immunohistochemistry whole-slide images (WSIs). Our method leverages a neural network that combines multi-scale morphological features with boundary features, enabling precise segmentation of invasive carcinoma regions without the need for additional H&E and P63 staining slides. In addition, we introduced an advanced semi-supervised learning algorithm, allowing efficient training of the model using unlabeled data. To evaluate the effectiveness of our approach, we constructed a dataset consisting of 618 IHC-stained WSIs from 170 cases, including four types of staining (ER, PR, HER2, and Ki-67). Notably, the model demonstrated an impressive intersection over union (IoU) score exceeding 80% on the test set. Furthermore, to ascertain the practical utility of our model in IHC quantitative evaluation, we constructed a fully automated Ki-67 scoring system based on the model's predictions. Comparative experiments convincingly demonstrated that our system exhibited high consistency with the scores given by experienced pathologists. Our developed model excels in accurately distinguishing between DCIS and invasive carcinoma regions in breast cancer immunohistochemistry WSIs. This method paves the way for a clinically available, fully automated immunohistochemistry quantitative scoring system.