Abundant Unlabeled Data Research Articles

DyConfidMatch: Dynamic thresholding and re-sampling for 3D semi-supervised learning

Semi-supervised learning (SSL) leverages limited labeled and abundant unlabeled data but often faces challenges with data imbalance, especially in 3D contexts. This study investigates class-level confidence as an indicator of learning status in 3D SSL, proposing a novel method that utilizes dynamic thresholding to better use unlabeled data, particularly from underrepresented classes. A re-sampling strategy is also introduced to mitigate bias towards well-represented classes, ensuring equitable class representation. Through extensive experiments in 3D SSL, our method surpasses state-of-the-art counterparts in classification and detection tasks, highlighting its effectiveness in tackling data imbalance. This approach presents a significant advancement in SSL for 3D datasets, providing a robust solution for data imbalance issues.

Diversity matters: Cross-head mutual mean-teaching for semi-supervised medical image segmentation

Semi-supervised medical image segmentation (SSMIS) has witnessed substantial advancements by leveraging limited labeled data and abundant unlabeled data. Nevertheless, existing state-of-the-art (SOTA) methods encounter challenges in accurately predicting labels for the unlabeled data, giving rise to disruptive noise during training and susceptibility to erroneous information overfitting. Moreover, applying perturbations to inaccurate predictions further impedes consistent learning. To address these concerns, we propose a novel cross-head mutual mean-teaching network (CMMT-Net) incorporated weak-strong data augmentations, thereby benefiting both co-training and consistency learning. More concretely, our CMMT-Net extends the cross-head co-training paradigm by introducing two auxiliary mean teacher models, which yield more accurate predictions and provide supplementary supervision. The predictions derived from weakly augmented samples generated by one mean teacher are leveraged to guide the training of another student with strongly augmented samples. Furthermore, two distinct yet synergistic data perturbations at the pixel and region levels are introduced. We propose mutual virtual adversarial training (MVAT) to smooth the decision boundary and enhance feature representations, and a cross-set CutMix strategy to generate more diverse training samples for capturing inherent structural data information. Notably, CMMT-Net simultaneously implements data, feature, and network perturbations, amplifying model diversity and generalization performance. Experimental results on three publicly available datasets indicate that our approach yields remarkable improvements over previous SOTA methods across various semi-supervised scenarios. The code is available at https://github.com/Leesoon1984/CMMT-Net.

Semi-Supervised One-Stage Object Detection for Maize Leaf Disease

Maize is one of the most important crops globally, and accurate diagnosis of leaf diseases is crucial for ensuring increased yields. Despite the continuous progress in computer vision technology, detecting maize leaf diseases based on deep learning still relies on a large amount of manually labeled data, and the labeling process is time-consuming and labor-intensive. Moreover, the detectors currently used for identifying maize leaf diseases have relatively low accuracy in complex experimental fields. Therefore, the proposed Agronomic Teacher, an object detection algorithm that utilizes limited labeled and abundant unlabeled data, is applied to maize leaf disease recognition. In this work, a semi-supervised object detection framework is built based on a single-stage detector, integrating the Weighted Average Pseudo-labeling Assignment (WAP) strategy and AgroYOLO detector combining Agro-Backbone network with Agro-Neck network. The WAP strategy uses weight adjustments to set objectness and classification scores as evaluation criteria for pseudo-labels reliability assignment. Agro-Backbone network accurately extracts features of maize leaf diseases and obtains richer semantic information. Agro-Neck network enhances feature fusion by utilizing multi-layer features for collaborative combinations. The effectiveness of the proposed method is validated on the MaizeData and PascalVOC datasets at different annotation ratios. Compared to the baseline model, Agronomic Teacher leverages abundant unlabeled data to achieve a 6.5% increase in mAP (0.5) on the 30% labeled MaizeData. On the 30% labeled PascalVOC dataset, the mAP (0.5) improved by 8.2%, demonstrating the method’s potential for generalization.

Semi-supervised learning with flexible threshold for non-intrusive load monitoring

Non-intrusive load monitoring (NILM) can obtain fine-grained power consumption information for individual appliances within the user without installing additional hardware sensors. With the rapid development of the deep learning model, many methods have been utilized to address NILM problems and have achieved enhanced appliance identification performance. However, supervised learning models require a substantial volume of annotated data to function effectively, which is time-consuming, laborious, and difficult to implement in real scenarios. In this paper, we propose a novel semi-supervised learning method that combines consistency regularization and pseudo-labels to help identification of appliances with limited labeled data and an abundance of unlabeled data. In addition, given the different learning difficulties of various appliance categories, for example, feature learning is more difficult for multi-state appliances than two-state appliances, the thresholds employed for different appliances are adjusted in a flexible way at each time step so that the informative unlabeled data and their pseudo-labels can be delivered. Experiments have been conducted on publicly available datasets, and the results indicate that the proposed method attains superior appliance identification performance compared to cutting-edge methods.

A Survey on the Applications of Semi-supervised Learning to Cyber-security

Machine Learning’s widespread application owes to its ability to develop accurate and scalable models. In cyber-security, where labeled data is scarce, Semi-Supervised Learning (SSL) emerges as a potential solution. SSL excels at tasks challenging traditional supervised and unsupervised algorithms by leveraging limited labeled data alongside abundant unlabeled data. This article presents a comprehensive survey of SSL in cyber-security, focusing on countering diverse cybercrimes, particularly intrusion detection. Despite its potential, a notable research gap persists, with few recent studies comprehensively reviewing SSL’s application in cyber-security. This study examines state-of-the-art SSL techniques tailored for cyber-security to address this gap. Relevant methods are identified, and their effectiveness is evaluated to empower researchers and practitioners with insights to enhance cyber-security measures. This work sheds light on SSL’s potential in addressing data scarcity in cyber-security domains in addition to outlining new research directions to advance this crucial field. By bridging this research gap, this manuscript paves the way for enhanced cyber-threat detection and mitigation in an increasingly interconnected world.

Non-intrusive load monitoring based on MoCo_v2, time series self-supervised learning

Traditional non-intrusive load monitoring (NILM) methods rely on massive historical labeled data. However, due to the privacy and high labeling cost of datasets, their generality and feasibility are limited, with poor performance in identifying devices with multiple states or similar features. To address this issue, this paper proposes an SSCL-LM framework based on temporal convolutional network (TCN), integrating contrastive self-supervised learning into NILM. Initially, through contrastive self-supervised pre-training, the framework learns well-represented load temporal characteristics from abundant unlabeled 1D power data. Then a small amount of labeled data is used to fine-tune the classifier to learn load categories represented by different temporal characteristics. Finally, test data is inputted into the model for load identification. Validating on the REDD dataset, the results demonstrate that with only 30% labeled data for fine-tuning, the proposed method achieves a 4.04% higher F1 score compared to traditional supervised methods. Moreover, utilizing only 1D power features, this method exhibits superior identification performance for devices with multiple states or similar features. Furthermore, this method enables performance transfer among loads of the same device with different parameters across different households, verifying its strong generalization and practicality.

Synergizing Unsupervised and Supervised Learning: A Hybrid Approach for Accurate Natural Language Task Modeling

While supervised learning models have shown remarkable performance in various natural language processing (NLP) tasks, their success heavily relies on the availability of large-scale labeled datasets, which can be costly and time-consuming to obtain. Conversely, unsupervised learning techniques can leverage abundant unlabeled text data to learn rich representations, but they do not directly optimize for specific NLP tasks. This paper presents a novel hybrid approach that synergizes unsupervised and supervised learning to improve the accuracy of NLP task modeling. While supervised models excel at specific tasks, they rely on large labeled datasets. Unsupervised techniques can learn rich representations from abundant unlabeled text but don't directly optimize for tasks. Our methodology integrates an unsupervised module that learns representations from unlabeled corpora (e.g., language models, word embeddings) and a supervised module that leverages these representations to enhance task-specific models [4]. We evaluate our approach on text classification and named entity recognition (NER), demonstrating consistent performance gains over supervised baselines. For text classification, contextual word embeddings from a language model pretrain a recurrent or transformer-based classifier. For NER, word embeddings initialize a BiLSTM sequence labeler. By synergizing techniques, our hybrid approach achieves SOTA results on benchmark datasets, paving the way for more data-efficient and robust NLP systems.

SCAC: A Semi-Supervised Learning Approach for Cervical Abnormal Cell Detection.

Cervical abnormal cell detection plays a crucial role in the early screening of cervical cancer. In recent years, some deep learning-based methods have been proposed. However, these methods rely heavily on large amounts of annotated images, which are time-consuming and labor-intensive to acquire, thus limiting the detection performance. In this paper, we present a novel Semi-supervised Cervical Abnormal Cell detector (SCAC), which effectively utilizes the abundant unlabeled data. We utilize Transformer as the backbone of SCAC to capture long-range dependencies to mimic the diagnostic process of pathologists. In addition, in SCAC, we design a Unified Strong and Weak Augment strategy (USWA) that unifies two data augmentation pipelines, implementing consistent regularization in semi-supervised learning and enhancing the diversity of the training data. We also develop a Global Attention Feature Pyramid Network (GAFPN), which utilizes the attention mechanism to better extract multi-scale features from cervical cytology images. Notably, we have created an unlabeled cervical cytology image dataset, which can be leveraged by semi-supervised learning to enhance detection accuracy. To the best of our knowledge, this is the first publicly available large unlabeled cervical cytology image dataset. By combining this dataset with two publicly available annotated datasets, we demonstrate that SCAC outperforms other existing methods, achieving state-of-the-art performance. Additionally, comprehensive ablation studies are conducted to validate the effectiveness of USWA and GAFPN. These promising results highlight the capability of SCAC to achieve high diagnostic accuracy and extensive clinical applications.

Anticipating Trends and Innovations for A Sustainable Automotive Industrial Revolution

Graph classification aims to predict the label associated with a graph and is an important graph analytic task with widespread applications. Recently, graph neural networks (GNNs) have achieved state-of-the-art results on purely supervised graph classification by virtue of the powerful representation ability of neural networks. However, almost all of them ignore the fact that graph classification usually lacks reasonably sufficient labelled data in practical scenarios due to the inherent labelling difficulty caused by the high complexity of graph data. The existing semi-supervised GNNs typically focus on the task of node classification and are incapable of dealing with graph classification. To tackle the challenging but practically useful scenario, we propose a novel and general semi-supervised GNN framework for graph classification, which takes full advantage of a slight amount of labelled graphs and abundant unlabelled graph data. In our framework, we train two GNNs as complementary views for collaboratively learning high-quality classifiers using both labelled and unlabelled graphs. To further exploit the view itself, we constantly select pseudo-labelled graph examples with high confidence from its own view for enlarging the labelled graph dataset and enhancing predictions on graphs. Furthermore, the proposed framework is investigated on two specific implementation regimes with a few labelled graphs and the extremely few labelled graphs, respectively. Extensive experimental results demonstrate the effectiveness of our proposed semi-supervised GNN framework for graph classification on several benchmark datasets

Boundary sample-based class-weighted semi-supervised learning for malignant tumor classification of medical imaging.

Medical image classification plays a pivotal role within the field of medicine. Existing models predominantly rely on supervised learning methods, which necessitate large volumes of labeled data for effective training. However, acquiring and annotating medical image data is both an expensive and time-consuming endeavor. In contrast, semi-supervised learning methods offer a promising approach by harnessing limited labeled data alongside abundant unlabeled data to enhance the performance of medical image classification. Nonetheless, current methods often encounter confirmation bias due to noise inherent in self-generated pseudo-labels and the presence of boundary samples from different classes. To overcome these challenges, this study introduces a novel framework known as boundary sample-based class-weighted semi-supervised learning (BSCSSL) for medical image classification. Our method aims to alleviate the impact of intra- and inter-class boundary samples derived from unlabeled data. Specifically, we address reliable confidential data and inter-class boundary samples separately through the utilization of an inter-class boundary sample mining module. Additionally, we implement an intra-class boundary sample weighting mechanism to extract class-aware features specific to intra-class boundary samples. Rather than discarding such intra-class boundary samples outright, our approach acknowledges their intrinsic value despite the difficulty associated with accurate classification, as they contribute significantly to model prediction. Experimental results on widely recognized medical image datasets demonstrate the superiority of our proposed BSCSSL method over existing semi-supervised learning approaches. By enhancing the accuracy and robustness of medical image classification, our BSCSSL approach yields considerable implications for advancing medical diagnosis and future research endeavors.

Auto-Labeling for Pattern Recognition of Wafer Defect Maps in Semiconductor Manufacturing

Abstract In practical semiconductor processes, the defect analysis for wafer map is a critical step for improving product quality and yield. These defect patterns can provide important process information so that the process engineers can identify the key cause of process anomalies. However, in supervised learning, the manual annotation for wafer maps is an extremely exhausting task, and it can also induce misjudgment when a long-term operation is implemented. Toward this end, this paper proposes a new auto-labeling system based on ensemble classification. The noted VGG16 model is used in ensemble learning as the building block to train the classifier via a limited number of labeled data. Through the model being trained, the auto-labeling procedure is executed to annotate abundant unlabeled data. Therefore, the classification performances between the models trained by supervised and semi-supervised learning can be compared. In addition, the gradient-weighted class activation mapping (Grad-CAM) is also adopted to analyze and verify the quality of auto-labeling by visual inspection. Based on the experimental results, the proposed auto-label system can return a satisfactory classification performance, and then, the manual labeling operation can be drastically reduced. The classification performance for wafer defect patterns can be further assured as the auto-labeled data are given with corresponding confidence scores of specific defect patterns being identified in this study.

LensePro: label noise-tolerant prototype-based network for improving cancer detection in prostate ultrasound with limited annotations.

The standard of care for prostate cancer (PCa) diagnosis is the histopathological analysis of tissue samples obtained via transrectal ultrasound (TRUS) guided biopsy. Models built with deep neural networks (DNNs) hold the potential for direct PCa detection from TRUS, which allows targeted biopsy and subsequently enhances outcomes. Yet, there are ongoing challenges with training robust models, stemming from issues such as noisy labels, out-of-distribution (OOD) data, and limited labeled data. This study presents LensePro, a unified method that not only excels in label efficiency but also demonstrates robustness against label noise and OOD data. LensePro comprises two key stages: first, self-supervised learning to extract high-quality feature representations from abundant unlabeled TRUS data and, second, label noise-tolerant prototype-based learning to classify the extracted features. Using data from 124 patients who underwent systematic prostate biopsy, LensePro achieves an AUROC, sensitivity, and specificity of 77.9%, 85.9%, and 57.5%, respectively, for detecting PCa in ultrasound. Our model shows it is effective for detecting OOD data in test time, critical for clinical deployment. Ablation studies demonstrate that each component of our method improves PCa detection by addressing one of the three challenges, reinforcing the benefits of a unified approach. Through comprehensive experiments, LensePro demonstrates its state-of-the-art performance for TRUS-based PCa detection. Although further research is necessary to confirm its clinical applicability, LensePro marks a notable advancement in enhancing automated computer-aided systems for detecting prostate cancer in ultrasound.

Semi-supervised Learning of Dynamical Systems with Neural Ordinary Differential Equations: A Teacher-Student Model Approach

Modeling dynamical systems is crucial for a wide range of tasks, but it remains challenging due to complex nonlinear dynamics, limited observations, or lack of prior knowledge. Recently, data-driven approaches such as Neural Ordinary Differential Equations (NODE) have shown promising results by leveraging the expressive power of neural networks to model unknown dynamics. However, these approaches often suffer from limited labeled training data, leading to poor generalization and suboptimal predictions. On the other hand, semi-supervised algorithms can utilize abundant unlabeled data and have demonstrated good performance in classification and regression tasks. We propose TS-NODE, the first semi-supervised approach to modeling dynamical systems with NODE. TS-NODE explores cheaply generated synthetic pseudo rollouts to broaden exploration in the state space and to tackle the challenges brought by lack of ground-truth system data under a teacher-student model. TS-NODE employs an unified optimization framework that corrects the teacher model based on the student's feedback while mitigating the potential false system dynamics present in pseudo rollouts. TS-NODE demonstrates significant performance improvements over a baseline Neural ODE model on multiple dynamical system modeling tasks.

Semi-Supervised Multiview Feature Selection With Adaptive Graph Learning.

As data sources become ever more numerous with increased feature dimensionality, feature selection for multiview data has become an important technique in machine learning. Semi-supervised multiview feature selection (SMFS) focuses on the problem of how to obtain a discriminative feature subset from heterogeneous feature spaces in the case of abundant unlabeled data with little labeled data. Most existing methods suffer from unreliable similarity graph structure across different views since they separate the graph construction from feature selection and use the fixed graphs that are susceptible to noisy features. Furthermore, they directly concatenate multiple feature projections for feature selection, neglecting the contribution diversity among projections. To alleviate these problems, we present an SMFS to simultaneously select informative features and learn a unified graph through the data fusion from aspects of feature projection and similarity graph. Specifically, SMFS adaptively weights different feature projections and flexibly fuses them to form a joint weighted projection, preserving the complementarity and consensus of the original views. Moreover, an implicit graph fusion is devised to dynamically learn a compatible graph across views according to the similarity structure in the learned projection subspace, where the undesirable effects of noisy features are largely alleviated. A convergent method is derived to iteratively optimize SMFS. Experiments on various datasets validate the effectiveness and superiority of SMFS over state-of-the-art methods.

Enhancing Human Activity Recognition in Smart Homes with Self-Supervised Learning and Self-Attention.

Deep learning models have gained prominence in human activity recognition using ambient sensors, particularly for telemonitoring older adults' daily activities in real-world scenarios. However, collecting large volumes of annotated sensor data presents a formidable challenge, given the time-consuming and costly nature of traditional manual annotation methods, especially for extensive projects. In response to this challenge, we propose a novel AttCLHAR model rooted in the self-supervised learning framework SimCLR and augmented with a self-attention mechanism. This model is designed for human activity recognition utilizing ambient sensor data, tailored explicitly for scenarios with limited or no annotations. AttCLHAR encompasses unsupervised pre-training and fine-tuning phases, sharing a common encoder module with two convolutional layers and a long short-term memory (LSTM) layer. The output is further connected to a self-attention layer, allowing the model to selectively focus on different input sequence segments. The incorporation of sharpness-aware minimization (SAM) aims to enhance model generalization by penalizing loss sharpness. The pre-training phase focuses on learning representative features from abundant unlabeled data, capturing both spatial and temporal dependencies in the sensor data. It facilitates the extraction of informative features for subsequent fine-tuning tasks. We extensively evaluated the AttCLHAR model using three CASAS smart home datasets (Aruba-1, Aruba-2, and Milan). We compared its performance against the SimCLR framework, SimCLR with SAM, and SimCLR with the self-attention layer. The experimental results demonstrate the superior performance of our approach, especially in semi-supervised and transfer learning scenarios. It outperforms existing models, marking a significant advancement in using self-supervised learning to extract valuable insights from unlabeled ambient sensor data in real-world environments.

A semi-supervised concept-cognitive computing system for dynamic classification decision making with limited feedback information

In dynamic environments, making classification decisions based on classical intelligent decision support systems is a challenge, as the classification performance of decision-making and the time-cost of learning need to be considered simultaneously. Moreover, many tasks of classification decisions lack label information because annotating data is time-consuming, labor-intensive and expensive process. This means that some standard intelligent decision support systems will perform inferior performance if they cannot dynamically make full use of the information behind abundant unlabeled data. Therefore, by incorporating knowledge representation and dynamic updating mechanisms into concept learning processes, we introduce a novel dynamic concept learning approach, namely semi-supervised concept-cognitive computing system (s2C3S), for making classification decisions by jointly utilizing some labeled data and abundant unlabeled data under dynamic environments. A theoretical analysis has shown that the proposed s2C3S can achieve significantly lower computational costs and higher classification accuracies than the existing incremental K Nearest Neighbor method (IKNN) and concept-cognitive computing system (C3S). The experimental results on various datasets further demonstrated that our system is effective for dynamic classification decision-making with limited labeled data under dynamic learning processes. Additionally, s2C3S can also be applied to computer-assisted intelligent diagnosis from the given medical images (such as chest X-ray images) dynamically and accurately.

CrossMatch: Enhance Semi-Supervised Medical Image Segmentation with Perturbation Strategies and Knowledge Distillation.

Semi-supervised learning for medical image segmentation presents a unique challenge of efficiently using limited labeled data while leveraging abundant unlabeled data. Despite advancements, existing methods often do not fully exploit the potential of the unlabeled data for enhancing model robustness and accuracy. In this paper, we introduce CrossMatch, a novel framework that integrates knowledge distillation with dual perturbation strategies, image-level and feature-level, to improve the model's learning from both labeled and unlabeled data. CrossMatch employs multiple encoders and decoders to generate diverse data streams, which undergo self-knowledge distillation to enhance the consistency and reliability of predictions across varied perturbations. Our method significantly surpasses other state-of-the-art techniques in standard benchmarks by effectively minimizing the gap between training on labeled and unlabeled data and improving edge accuracy and generalization in medical image segmentation. The efficacy of CrossMatch is demonstrated through extensive experimental validations, showing remarkable performance improvements without increasing computational costs. Code for this implementation is made available at https://github.com/AiEson/CrossMatch.git.

XRDMatch: a semi-supervised learning framework to efficiently discover room temperature lithium superionic conductors

We propose XRDMatch, a semi-supervised learning framework that integrates consistency regularization and pseudo-labeling. Using X-ray diffraction patterns as descriptors, it effectively addresses data scarcity by leveraging abundant unlabeled data.

Tower Masking MIM: A Self-Supervised Pretraining Method for Power Line Inspection

For intelligent inspection of power lines, a core task is to detect components in aerial images. Currently, deep supervised learning, a data-hungry paradigm, has attracted great attention. However, considering real-world scenarios, labeled data are usually limited, and the utilization of abundant unlabeled data is rarely investigated in this field. This study deploys a pretrained model for power line component detection based on a self-supervised pretraining approach, which exploits useful information from unannotated data. Concretely, we design a new masking strategy based on the structural characteristic of power lines to guide the pretraining process with meaningful semantic content. Meanwhile, a Siamese architecture is proposed to extract complete global features by using dual reconstruction with semantic targets provided by the proposed masking strategy. Then, the knowledge distillation is utilized to enable the pretrained model to learn both domain-specific and general representations. Moreover, a feature pyramid mechanism is adopted to capture multi-scale features, which can benefit the detection task. Experimental results show that the proposed approach can successfully improve the performance of a variety detection frameworks for power line components, and outperforms other self-supervised pretraining methods.

Lithofacies Identification from Wire-Line Logs Using an Unsupervised Data Clustering Algorithm

Stratigraphic identification from wire-line logs and core samples is a common method for lithology classification. This traditional approach is considered superior, despite its significant financial cost. Artificial neural networks and machine learning offer alternative, cost-effective means for automated data interpretation, allowing geoscientists to extract insights from data. At the same time, supervised and semi-supervised learning techniques are commonly employed, requiring a sufficient amount of labeled data to be generated through manual interpretation. Typically, there are abundant unlabeled geophysical data while labeled data are scarcer. Supervised and semi-supervised techniques partially address the cost issue. An underutilized class of machine-learning-based methods, unsupervised data clustering, can perform consonant classification by grouping similar data without requiring known results, presenting an even more cost-effective solution. In this study, we examine a state-of-the-art unsupervised data clustering algorithm called piecemeal clustering to identify lithofacies from wire-line logs, effectively addressing these challenges. The piecemeal clustering algorithm groups similar wire-log signatures into clusters, determines the number of clusters present in the data, and assigns each signature to one of the clusters, each of which represents a lithofacies. To evaluate the performance, we tested the algorithm on publicly released data from ten wells drilled in the Hugoton and Panoma fields of southwest Kansas and northwest Oklahoma, respectively. The data consist of two major groups: marine and non-marine facies. The study herein is centered around addressing two fundamental research questions regarding the accuracy and practicality of the piecemeal clustering algorithm. The algorithm successfully identified nine distinct clusters in our dataset, aligning with the cluster count observed in previously published works employing the same data. Regarding mapping accuracy, the results were notable, with success rates of 81.90% and 45.20% with and without considering adjacent facies, respectively. Further detailed analysis of the results was conducted for individual types of facies and independently for each well. These findings suggest the algorithm’s precision in characterizing the geological formations. To assess its performance, a comprehensive comparative analysis was conducted, encompassing other data clustering algorithms, as well as supervised and semi-supervised machine learning techniques. Notably, the piecemeal clustering algorithm outperformed alternative data clustering methods. Furthermore, despite its unsupervised nature, the algorithm demonstrated competitiveness by yielding results comparable to, or even surpassing, those obtained through supervised and semi-supervised techniques.