Large Amount Of Unlabeled Data Research Articles

Cytometry masked autoencoder: An accurate and interpretable automated immunophenotyper.

Single-cell cytometry data are crucial for understanding the role of the immune system in diseases and responses to treatment. However, traditional methods for annotating cytometry data face challenges in scalability, robustness, and accuracy. We propose a cytometry masked autoencoder (cyMAE), which automates immunophenotyping tasks including cell type annotation. The model upholds user-defined cell type definitions, facilitating interpretability and cross-study comparisons. The training of cyMAE has a self-supervised phase, which leverages large amounts of unlabeled data, followed by fine-tuning on specialized tasks using smaller amounts of annotated data. The cost of training a new model is amortized over repeated inferences on new datasets using the same panel. Through validation across multiple studies using the same panel, we demonstrate that cyMAE delivers accurate and interpretable cellular immunophenotyping and improves the prediction of subject-level metadata. This proof of concept marks a significant step forward for large-scale immunology studies.

Consistency Self-Training Semi-Supervised Method for Road Extraction from Remote Sensing Images

Road extraction techniques based on remote sensing image have significantly advanced. Currently, fully supervised road segmentation neural networks based on remote sensing images require a significant number of densely labeled road samples, limiting their applicability in large-scale scenarios. Consequently, semi-supervised methods that utilize fewer labeled data have gained increasing attention. However, the imbalance between a small quantity of labeled data and a large volume of unlabeled data leads to local detail errors and overall cognitive mistakes in semi-supervised road extraction. To address this challenge, this paper proposes a novel consistency self-training semi-supervised method (CSSnet), which effectively learns from a limited number of labeled data samples and a large amount of unlabeled data. This method integrates self-training semi-supervised segmentation with semi-supervised classification. The semi-supervised segmentation component relies on an enhanced generative adversarial network for semantic segmentation, which significantly reduces local detail errors. The semi-supervised classification component relies on an upgraded mean-teacher network to handle overall cognitive errors. Our method exhibits excellent performance with a modest amount of labeled data. This study was validated on three separate road datasets comprising high-resolution remote sensing satellite images and UAV photographs. Experimental findings showed that our method consistently outperformed state-of-the-art semi-supervised methods and several classic fully supervised methods.

A semi‐supervised segmentation network fusing pseudo‐label with multi‐level feature consistency correction for hard exudates

AbstractTimely detection of hard exudates in fundus images can effectively avoid the severity of the disease, but the labelling of small and numerous lesion areas requires a lot of labour costs. This paper proposes a semi‐supervised segmentation network, which integrates pseudo‐labels and multi‐level features consistency correction. It achieves accurate segmentation of hard exudates by making full use of a small amount of labelled data and a large amount of unlabelled data. The network effectively extracts features from the unlabelled data through knowledge transfer of the teacher‐student model, and incorporates a Transformer network for auxiliary training to promote the quality of transfer. In addition, three unsupervised losses are introduced to improve the performance: the perturbation loss improves the robustness of the model to noise by adding different noises to the same input; the multi‐level feature consistency correction loss ensures the consistency of features of the student model at different scales; and the pseudo‐labelling cross‐supervision loss utilizes the generated pseudo‐labels for supervision between CNN and Transformer. By comparing the segmentation results with different proportion of the labelled data, it has better segmentation performance compared to other methods. The proposed methods can totally increase dice by 16.56% and mean intersection over union (MIoU) by 25.11%.

Semi-Supervised Encrypted Malicious Traffic Detection Based on Multimodal Traffic Characteristics.

The exponential growth of encrypted network traffic poses significant challenges for detecting malicious activities online. The scale of emerging malicious traffic is significantly smaller than that of normal traffic, and the imbalanced data distribution poses challenges for detection. However, most existing methods rely on single-category features for classification, which struggle to detect covert malicious traffic behaviors. In this paper, we introduce a novel semi-supervised approach to identify malicious traffic by leveraging multimodal traffic characteristics. By integrating the sequence and topological information inherent in the traffic, we achieve a multifaceted representation of encrypted traffic. We design two independent neural networks to learn the corresponding sequence and topological features from the traffic. This dual-feature extraction enhances the model's robustness in detecting anomalies within encrypted traffic. The model is trained using a joint strategy that minimizes both the reconstruction error from the autoencoder and the classification loss, allowing it to effectively utilize limited labeled data alongside a large amount of unlabeled data. A confidence-estimation module enhances the classifier's ability to detect unknown attacks. Finally, our method is evaluated on two benchmark datasets, UNSW-NB15 and CICIDS2017, under various scenarios, including different training set label ratios and the presence of unknown attacks. Our model outperforms other models by 3.49% and 5.69% in F1 score at labeling rates of 1% and 0.1%, respectively.

Semi-Supervised Triply Robust Inductive Transfer Learning

In this work, we propose a Semi-supervised Triply Robust Inductive transFer LEarning (STRIFLE) approach, which integrates heterogeneous data from a label-rich source population and a label-scarce target population and uses a large amount of unlabeled data simultaneously to improve the learning accuracy in the target population. Specifically, we consider a high dimensional covariate shift setting and employ two nuisance models, a density ratio model and an imputation model, to combine transfer learning and surrogate-assisted semi-supervised learning strategies effectively and achieve triple robustness. While the STRIFLE approach assumes the target and source populations to share the same conditional distribution of outcome Y given both the surrogate features S and predictors X , it allows the true underlying model of Y⏧ X to differ between the two populations due to the potential covariate shift in S and X . Different from double robustness, even if both nuisance models are misspecified or the distribution of Y⏧ S , X is not the same between the two populations when the shifted source population and the target population share enough similarities, the triply robust STRIFLE estimator can still partially use the source population when the shifted source population and the target population share enough similarities. Moreover, it is guaranteed to be no worse than the target-only surrogate-assisted semi-supervised estimator with an additional error term from transferability detection. These desirable properties of our estimator are established theoretically and verified in finite samples via extensive simulation studies. We use the STRIFLE estimator to train a Type II diabetes polygenic risk prediction model for the African American target population by transferring knowledge from electronic health records linked genomic data observed in a larger European source population. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Self-supervised learning of Vision Transformers for digital soil mapping using visual data

In arid environments, prospecting cultivable land is challenging due to harsh climatic conditions and vast, hard-to-access areas. However, the soil is often bare, with little vegetation cover, making it easy to observe from above. Hence, remote sensing can drastically reduce costs to explore these areas. For the past few years, deep learning has extended remote sensing analysis, first with Convolutional Neural Networks (CNNs), then with Vision Transformers (ViTs). The main drawback of deep learning methods is their reliance on large calibration datasets, as data collection is a cumbersome and costly task, particularly in drylands. However, recent studies demonstrate that ViTs can be trained in a self-supervised manner to take advantage of large amounts of unlabelled data to pre-train models. These backbone models can then be finetuned to learn a supervised regression model with few labelled data.In our study, we trained ViTs in a self-supervised way with a 9500 km2 satellite image of dry-lands in Saudi Arabia with a spatial resolution of 1.5 m per pixel. The resulting models were used to extract features describing the bare soil and predict soil attributes (pH H2O, pH KCl, Si composition). Using only RGB data, we can accurately predict these soil properties and achieve, for instance, an RMSE of 0.40 ± 0.03 when predicting alkaline soil pH. We also assess the effectiveness of adding additional covariates, such as elevation. The pretrained models can as well be used as visual features extractors. These features can be used to automatically generate a clustered map of an area or as input of random forests models, providing a versatile way to generate maps with limited labelled data and input variables.

A representation learning-based approach to enhancing manufacturing quality for low-voltage electrical products

In low-voltage electrical product manufacturing, resolving quality issues is heavily reliant on engineering experience, and can be time-consuming and error-prone. Through quality management systems, a large number of historical defect cases can be consolidated for analysis along with relevant causes. However, these defect descriptions are often casually described with a mix of Chinese and English language, containing domain-specific terms. Additionally, defect product features have varieties and complex relationships. Therefore, historical defect cases have not been effectively utilized to support manufacturing quality issues. To address this challenge, this study proposes a representation learning-based approach to enhance manufacturing quality. Key research contributions include: (1) A two-stage word embedding technique based on the pre-trained language model. First, TSDAE is utilized for unsupervised pre-training on a large amount of unlabeled data. Then, Sentence-BERT is utilized for fine-tuning on a small set of labeled similar sentence pairs. This process yields a pre-trained language model specific to low-voltage electrical product defects. (2) NSHPSAGE graph embedding model based on the constructed product feature knowledge graph. We select more valuable neighboring nodes during sampling and explore different aggregation functions to enhance graph embedding performance. This model effectively aggregates product feature information into “Defect_Case” nodes, yielding graph embedding vectors. The model exhibits good Weighted-Precision and Weighted-Recall with a short training duration, and it can handle new nodes, addressing the issue of heterogeneous graph embedding. (3) A defect case recommendation technique that fuses word embedding and graph embedding. We use Multi-Head Attention Fusion in the late-fusion to obtain defect case vectors. This approach comprehensively considers defect description semantic knowledge and complex product feature relationships, enabling accurate defect case recommendation with the prototype system.

Improved Road Extraction Models through Semi-Supervised Learning with ACCT

Improving the performance and reducing the training cost of road extraction models in the absence of samples is important for updating road maps. Despite the success of recent road extraction models on standard datasets, they often fail to perform when applied to new datasets or real-world scenarios where labeled samples are not available. In this paper, our focus diverges from the typical quest to pinpoint the optimal road extraction model or evaluate generalization prowess across models. Instead, we propose a method called Asymmetric Consistent Co-Training (ACCT) to train existing road extraction models faster and make them perform better in new scenarios lacking samples. ACCT uses two models with different structures and a supervision module to enhance accuracy through mutual learning. Labeled and unlabeled images are processed by both models to generate road maps from different perspectives. The supervision module ensures consistency between predictions by computing losses based on labeling status. ACCT iteratively adjusts parameters using unlabeled data, improving generalization. Empirical evaluations show that ACCT improves IoU by 2.79% to 10.26% using only 1/8 of the labeled data compared to fully supervised methods. It also reduces parameters by over 49% compared to state-of-the-art semi-supervised methods while maintaining similar accuracy. These results highlight the potential of leveraging large amounts of unlabeled data to enhance road extraction models as data acquisition technology advances.

Semi-Supervised Interior Decoration Style Classification with Contrastive Mutual Learning

Precisely identifying interior decoration styles holds substantial significance in directing interior decoration practices. Nevertheless, constructing accurate models for the automatic classification of interior decoration styles remains challenging due to the scarcity of expert annotations. To address this problem, we propose a novel pseudo-label-guided contrastive mutual learning framework (PCML) for semi-supervised interior decoration style classification by harnessing large amounts of unlabeled data. Specifically, PCML introduces two distinct subnetworks and selectively utilizes the diversified pseudo-labels generated by each for mutual supervision, thereby mitigating the issue of confirmation bias. For labeled images, the inconsistent pseudo-labels generated by the two subnetworks are employed to identify images that are prone to misclassification. We then devise an inconsistency-aware relearning (ICR) regularization model to perform a review training process. For unlabeled images, we introduce a class-aware contrastive learning (CCL) regularization to learn their discriminative feature representations using the corresponding pseudo-labels. Since the use of distinct subnetworks reduces the risk of both models producing identical erroneous pseudo-labels, CCL can reduce the possibility of noise data sampling to enhance the effectiveness of contrastive learning. The performance of PCML is evaluated on five interior decoration style image datasets. For the average AUC, accuracy, sensitivity, specificity, precision, and F1 scores, PCML obtains improvements of 1.67%, 1.72%, 3.65%, 1.0%, 4.61%, and 4.66% in comparison with the state-of-the-art method, demonstrating the superiority of our method.

Meta-Adaptable-Adapter: Efficient adaptation of self-supervised models for low-resource speech recognition

Self-supervised models have demonstrated remarkable performance in speech processing by learning latent representations from large amounts of unlabeled data. Adapting these models to low-resource languages yields promising results, but the computational cost of fine-tuning all model parameters is prohibitively high. Adapters offer a solution by introducing lightweight bottleneck structures into pre-trained models for downstream tasks, enabling efficient parameter adaptation. However, randomly initialized adapters often underperform in extremely low-resource scenarios. To address this issue, we explore the Meta-Adapter for self-supervised models and analyzed some limitations of Meta-Adapter including poor learning in language-specific knowledge and meta-overfitting problems. To relieve these problems, we propose the Meta-Adaptable-Adapter (MAA), a new meta leaning algorithm that adapts to low-resource languages quickly and effectively. MAA learns task-specific adapters for feature extraction, and task-independent adapters for feature combination. The experiments on three datasets show superior performance on 31 low-resource languages across seven different language families compared to other adapters, showing better generalization and extensibility.

Artificial neural networks with better analysis reliability in data mining

If there are relatively few cases, semi-supervised learning approaches make advantage of a large amount of unlabeled data to assist develop a better classifier. To expand the labeled training set and update the classifier, a fundamental method is to select and label the unlabeled instances for which the current classifier has higher classification confidence. This approach is primarily used in two distinct semi-supervised learning paradigms: co-training and self-training. However, compared to self-labeled examples that would be tagged by a classifier, the real labeled instances will be more trustworthy. Incorrect label assignment to unlabeled occurrences might potentially compromise the classifier's accuracy in classification. This research presents a novel instance selection method based on actual labeled data. This will take into account the classifier's current performance on unlabeled data in addition to its performance on actual labeled data alone. This uses the accuracy changes in the newly trained classifier over the original labeled data as a criterion in each iteration to determine whether or not the selected most confident unlabeled examples would be accepted by a subsequent iteration. Naïve Bayes (NB) will be used as the basic classifier in the co-training and self-training studies. The findings indicate that the accuracy and categorization of self-training and co-training will be greatly enhanced by SIS. As compared to semi-supervised classification methods, it will enhance accuracy, precision, recall, and F1 score, according to the findings.

Discriminative atoms embedding relation dual network for classification of choroidal neovascularization in OCT images

Choroidal neovascularization (CNV) is an eye disease that can cause vision loss. Automatic CNV classification in OCT images is crucial in the treatment of CNV. However, two problems arise for CNV classification in OCT images. The subtle visual differences between different CNV types render classification difficult. Additionally, it is difficult to obtain sufficient labeled data, which results in performance degradation. In order to solve these two problems, a discriminative atom-embedding relation dual network is proposed in this paper. Considering that semi-supervised learning (SSL) is an effective machine learning framework to make full use of limited labeled data and a large amount of unlabeled data, the proposed network is developed within an SSL framework. To capture the visual differences, novel discriminative atoms are first introduced to mine discriminative information between different CNV types. Subsequently, a relation module is incorporated to embed the learned discriminative atom information into the features. This makes the learned features capable of distinguishing between different CNV types. Moreover, a novel relation consistency loss is proposed to further improve the robustness of the learned features. Experimental results on private and public datasets demonstrate the effectiveness of the proposed method.

Novel mining conveyor monitoring system based on quasi-distributed optical fiber accelerometer array and self-supervised learning

Belt conveyors in mining are crucial, with downtime leading to significant losses and safety hazards. Unplanned shutdowns often result from idler failures. To address this, an online monitoring system for continuous idler health assessment is proposed. Considering the large number and dense spatial distribution of idlers over long distances, this work presents a system that utilizes a quasi-distributed optical fiber accelerometer array. This array incorporates phase-sensitive optical time domain reflectometry (Phase-OTDR) interrogation technology and ultra-weak fiber Bragg gratings (UWFBGs) to effectively capture idler vibrations. The designed array achieves high-sensitivity vibration sensing with a sensitivity of 2.4 rad/g and a resolution of 1.7 mg/Hz. After collecting the vibrations of idlers by the designed accelerometer array, an automatic fault classification algorithm based on self-supervised learning (SSL) is introduced, which requires only a small number of labeled samples. By leveraging large amount of unlabeled data in the pretext task, the algorithm efficiently extracts latent features from the quasi-distributed accelerometer array. A diagnosis accuracy of 95.37 % can be achieved on a seven-class classification task with only 3.6 % labeled data (16 samples/class). This system offers a promising solution for idler monitoring, combining high sensitivity, distributed measurement capabilities, enhanced security, and superior fault detection accuracy.

Knowing the unknowns: Network traffic detection with open-set semi-supervised learning

Network traffic classification plays a crucial role in network security and network quality of service. The new emerging traffic data in the real world is unlabeled and may contain unknown classes, which introduces the problem of identifying unknown-class traffic in unlabeled data. However, existing deep learning-based methods either rely on large amounts of labeled data or assume the labeled and unlabeled data share the common label space. Therefore, they are unable to identify unknown traffic, resulting in decreased detection accuracy. This paper introduces DivinEye, a novel method for unknown network traffic detection based on open-set semi-supervised learning. DivinEye utilizes a small quantity of labeled data and a large amount of unlabeled data to train the model, achieving the ability to identify known classes accurately while also detecting unknown ones. To leverage incoming unlabeled traffic, DivinEye employs open-set semi-supervised techniques to select known-class data from unlabeled data to optimize the known traffic detection model. To detect unknown traffic, DivinEye combines the known traffic classifier and multi-binary classifier to construct an unknown traffic detection model, generating auxiliary targets for all unlabeled data as supervision to train the model. Trace-driven experiments demonstrate that DivinEye outperforms state-of-the-art methods by a large margin.

OSSEFS: An online semi-supervised ensemble fuzzy system for data streams learning with missing values

The incompleteness of data samples in data streams always affects the performance of learning model. In order to learn data streams with missing values from features and a large amount of unlabeled data, this paper proposes a semi-supervised ensemble algorithm based on evolving fuzzy systems (EFS) named OSSEFS. Aiming at missing values, a fuzzy rule based online imputation strategy is designed to improve the data feature integrity. Aiming at unlabeled data, a new ensemble fuzzy system architecture is proposed to fully utilize both labeled and unlabeled data implicit information to predict data labels accurately. In addition, an online rule parameters optimization strategy is proposed to achieve adaptive adjustment of rule distribution. Thanks to the above strategies, OSSEFS can perform well in data environments containing missing values and unlabeled data. Comparative experimental results on multiple datasets show that OSSEFS outperforms the classical semi-supervised regression algorithms.

Toward Few-Label Vertical Federated Learning

Federated Learning (FL) provides a novel paradigm for privacy-preserving machine learning, enabling multiple clients to collaborate on model training without sharing private data. To handle multi-source heterogeneous data, Vertical Federated Learning (VFL) has been extensively investigated. However, in the context of VFL, the label information tends to be kept in one authoritative client and is very limited. This poses two challenges for model training in the VFL scenario. On the one hand, a small number of labels cannot guarantee to train a well VFL model with informative network parameters, resulting in unclear boundaries for classification decisions. On the other hand, the large amount of unlabeled data is dominant and should not be discounted, and it is worthwhile to focus on how to leverage them to improve representation modeling capabilities. To address the preceding two challenges, we first introduce supervised contrastive loss to enhance the intra-class aggregation and inter-class estrangement, which is to deeply explore label information and improve the effectiveness of downstream classification tasks. Then, for unlabeled data, we introduce a pseudo-label-guided consistency mechanism to induce the classification results coherent across clients, which allows the representations learned by local networks to absorb the knowledge from other clients, and alleviates the disagreement between different clients for classification tasks. We conduct sufficient experiments on four commonly used datasets, and the experimental results demonstrate that our method is superior to the state-of-the-art methods, especially in the low-label rate scenario, and the improvement becomes more significant.

Stacked co-training for semi-supervised multi-label learning

Due to the difficulty of annotation, multi-label learning sometimes obtains a small amount of labeled data and a large amount of unlabeled data as supplements. To make up this issue, many algorithms extended the existing semi-supervised strategies in single-label patterns to multi-label applications, but failed to effectively consider the characteristics of semi-supervised multi-label learning. In this paper, a novel method named SCTML (Stacked Co-Training for Multi-Label learning) is proposed for semi-supervised multi-label learning. Through a two-layer stacking framework, SCTML learns label correlation in both base learners and meta learner, and effectively incorporates the semi-supervised assumptions of co-training, clustering and manifold. Extensive experiments demonstrate that the combination of multiple semi-supervised learning strategies effectively solves the semi-supervised multi-label learning problem.

Uncertainty estimation- and attention-based semi-supervised models for automatically delineate clinical target volume in CBCT images of breast cancer

ObjectivesAccurate segmentation of the clinical target volume (CTV) of CBCT images can observe the changes of CTV during patients' radiotherapy, and lay a foundation for the subsequent implementation of adaptive radiotherapy (ART). However, segmentation is challenging due to the poor quality of CBCT images and difficulty in obtaining target volumes. An uncertainty estimation- and attention-based semi-supervised model called residual convolutional block attention-uncertainty aware mean teacher (RCBA-UAMT) was proposed to delineate the CTV in cone-beam computed tomography (CBCT) images of breast cancer automatically.MethodsA total of 60 patients who undergone radiotherapy after breast-conserving surgery were enrolled in this study, which involved 60 planning CTs and 380 CBCTs. RCBA-UAMT was proposed by integrating residual and attention modules in the backbone network 3D UNet. The attention module can adjust channel and spatial weights of the extracted image features. The proposed design can train the model and segment CBCT images with a small amount of labeled data (5%, 10%, and 20%) and a large amount of unlabeled data. Four types of evaluation metrics, namely, dice similarity coefficient (DSC), Jaccard, average surface distance (ASD), and 95% Hausdorff distance (95HD), are used to assess the model segmentation performance quantitatively.ResultsThe proposed method achieved average DSC, Jaccard, 95HD, and ASD of 82%, 70%, 8.93, and 1.49 mm for CTV delineation on CBCT images of breast cancer, respectively. Compared with the three classical methods of mean teacher, uncertainty-aware mean-teacher and uncertainty rectified pyramid consistency, DSC and Jaccard increased by 7.89–9.33% and 14.75–16.67%, respectively, while 95HD and ASD decreased by 33.16–67.81% and 36.05–75.57%, respectively. The comparative experiment results of the labeled data with different proportions (5%, 10% and 20%) showed significant differences in the DSC, Jaccard, and 95HD evaluation indexes in the labeled data with 5% versus 10% and 5% versus 20%. Moreover, no significant differences were observed in the labeled data with 10% versus 20% among all evaluation indexes. Therefore, we can use only 10% labeled data to achieve the experimental objective.ConclusionsUsing the proposed RCBA-UAMT, the CTV of breast cancer CBCT images can be delineated reliably with a small amount of labeled data. These delineated images can be used to observe the changes in CTV and lay the foundation for the follow-up implementation of ART.

Employing feature mixture for active learning of object detection

Active learning aims to select the most informative samples for annotation from a large amount of unlabeled data, in order to reduce time-consuming and labor-intensive manual labeling efforts. Although active learning for object detection has made substantial progress in recent years, developing an accurate and efficient active learning algorithm for object detection remains a challenge. In this paper, we propose a novel unsupervised active learning method for deep object detection. This is based on our hypotheses that an object is more likely to be wrongly predicted by the model, if the prediction changes when its feature representations are slightly mixed by another feature representations at a very small ratio. Such unlabeled samples can be regarded as informative samples that can be selected by active learning. Our method employs base representations of all categories generated from the object detection network to examine the robustness of every detected object. We design a scoring function to calculate the informative score of each unlabeled image. We conduct extensive experiments on two public datasets, i.e., PASCAL VOC and MS-COCO. Experiment results show that our approach consistently outperforms state-of-the-art single-model based methods with significant margins. Our approach also performs on par with multi-model based methods, at much lesser computational cost.

Semi-Supervised Training for (Pre-Stack) Seismic Data Analysis

A lack of labeled examples is a problem in different domains, such as text and image processing, medicine, and static reservoir characterization, because supervised learning relies on vast volumes of these data to perform successfully, but this is quite expensive. However, large amounts of unlabeled data exist in these domains. The deep semi-supervised learning (DSSL) approach leverages unlabeled data to improve supervised learning performance using deep neural networks. This approach has succeeded in image recognition, text classification, and speech recognition. Nevertheless, there have been few works on pre-stack seismic reservoir characterization, in which knowledge of rock and fluid properties is fundamental for oil exploration. This paper proposes a methodology to estimate acoustic impedance using pre-stack seismic data and DSSL with a recurrent neural network. The few labeled datasets for training were pre-processed from raw seismic and acoustic impedance data from five borehole logs. The results showed that the acoustic impedance estimation at the well location and outside it was better predicted by the DSSL compared to the supervised version of the same neural network. Therefore, employing a large amount of unlabeled data can be helpful in the development of seismic data interpretation systems.