Large Amount Of Unlabeled Data Research Articles

A representation learning-based approach to enhancing manufacturing quality for low-voltage electrical products

In low-voltage electrical product manufacturing, resolving quality issues is heavily reliant on engineering experience, and can be time-consuming and error-prone. Through quality management systems, a large number of historical defect cases can be consolidated for analysis along with relevant causes. However, these defect descriptions are often casually described with a mix of Chinese and English language, containing domain-specific terms. Additionally, defect product features have varieties and complex relationships. Therefore, historical defect cases have not been effectively utilized to support manufacturing quality issues. To address this challenge, this study proposes a representation learning-based approach to enhance manufacturing quality. Key research contributions include: (1) A two-stage word embedding technique based on the pre-trained language model. First, TSDAE is utilized for unsupervised pre-training on a large amount of unlabeled data. Then, Sentence-BERT is utilized for fine-tuning on a small set of labeled similar sentence pairs. This process yields a pre-trained language model specific to low-voltage electrical product defects. (2) NSHPSAGE graph embedding model based on the constructed product feature knowledge graph. We select more valuable neighboring nodes during sampling and explore different aggregation functions to enhance graph embedding performance. This model effectively aggregates product feature information into “Defect_Case” nodes, yielding graph embedding vectors. The model exhibits good Weighted-Precision and Weighted-Recall with a short training duration, and it can handle new nodes, addressing the issue of heterogeneous graph embedding. (3) A defect case recommendation technique that fuses word embedding and graph embedding. We use Multi-Head Attention Fusion in the late-fusion to obtain defect case vectors. This approach comprehensively considers defect description semantic knowledge and complex product feature relationships, enabling accurate defect case recommendation with the prototype system.

Meta-Adaptable-Adapter: Efficient adaptation of self-supervised models for low-resource speech recognition

Self-supervised models have demonstrated remarkable performance in speech processing by learning latent representations from large amounts of unlabeled data. Adapting these models to low-resource languages yields promising results, but the computational cost of fine-tuning all model parameters is prohibitively high. Adapters offer a solution by introducing lightweight bottleneck structures into pre-trained models for downstream tasks, enabling efficient parameter adaptation. However, randomly initialized adapters often underperform in extremely low-resource scenarios. To address this issue, we explore the Meta-Adapter for self-supervised models and analyzed some limitations of Meta-Adapter including poor learning in language-specific knowledge and meta-overfitting problems. To relieve these problems, we propose the Meta-Adaptable-Adapter (MAA), a new meta leaning algorithm that adapts to low-resource languages quickly and effectively. MAA learns task-specific adapters for feature extraction, and task-independent adapters for feature combination. The experiments on three datasets show superior performance on 31 low-resource languages across seven different language families compared to other adapters, showing better generalization and extensibility.

Artificial neural networks with better analysis reliability in data mining

If there are relatively few cases, semi-supervised learning approaches make advantage of a large amount of unlabeled data to assist develop a better classifier. To expand the labeled training set and update the classifier, a fundamental method is to select and label the unlabeled instances for which the current classifier has higher classification confidence. This approach is primarily used in two distinct semi-supervised learning paradigms: co-training and self-training. However, compared to self-labeled examples that would be tagged by a classifier, the real labeled instances will be more trustworthy. Incorrect label assignment to unlabeled occurrences might potentially compromise the classifier's accuracy in classification. This research presents a novel instance selection method based on actual labeled data. This will take into account the classifier's current performance on unlabeled data in addition to its performance on actual labeled data alone. This uses the accuracy changes in the newly trained classifier over the original labeled data as a criterion in each iteration to determine whether or not the selected most confident unlabeled examples would be accepted by a subsequent iteration. Naïve Bayes (NB) will be used as the basic classifier in the co-training and self-training studies. The findings indicate that the accuracy and categorization of self-training and co-training will be greatly enhanced by SIS. As compared to semi-supervised classification methods, it will enhance accuracy, precision, recall, and F1 score, according to the findings.

Semi-supervised Triply Robust Inductive Transfer Learning

In this work, we propose a Semi-supervised Triply Robust Inductive transFer LEarning (STRIFLE) approach, which integrates heterogeneous data from a label-rich source population and a label-scarce target population and utilizes a large amount of unlabeled data simultaneously to improve the learning accuracy in the target population. Specifically, we consider a high dimensional covariate shift setting and employ two nuisance models, a density ratio model and an imputation model, to combine transfer learning and surrogate-assisted semi-supervised learning strategies organically and achieve triple robustness. While the STRIFLE approach assumes the target and source populations to share the same conditional distribution of outcome Y given both the surrogate features S and predictors X , it allows the true underlying model of Y � X to differ between the two populations due to the potential covariate shift in S and X . Different from double robustness, even if both nuisance models are misspecified or the distribution of Y � S , X is not the same between the two populations when the shifted source population and the target population share enough similarities, the triply robust STRIFLE estimator can still partially utilize the source population, and it is guaranteed to be no worse than the target-only surrogate-assisted semi-supervised estimator with an additional error term from transferability detection. These desirable properties of our estimator are established theoretically and verified in finite samples via extensive simulation studies. We utilize the STRIFLE estimator to train a Type II diabetes polygenic risk prediction model for the African American target population by transferring knowledge from electronic health records linked genomic data observed in a larger European source population.

Discriminative atoms embedding relation dual network for classification of choroidal neovascularization in OCT images

Choroidal neovascularization (CNV) is an eye disease that can cause vision loss. Automatic CNV classification in OCT images is crucial in the treatment of CNV. However, two problems arise for CNV classification in OCT images. The subtle visual differences between different CNV types render classification difficult. Additionally, it is difficult to obtain sufficient labeled data, which results in performance degradation. In order to solve these two problems, a discriminative atom-embedding relation dual network is proposed in this paper. Considering that semi-supervised learning (SSL) is an effective machine learning framework to make full use of limited labeled data and a large amount of unlabeled data, the proposed network is developed within an SSL framework. To capture the visual differences, novel discriminative atoms are first introduced to mine discriminative information between different CNV types. Subsequently, a relation module is incorporated to embed the learned discriminative atom information into the features. This makes the learned features capable of distinguishing between different CNV types. Moreover, a novel relation consistency loss is proposed to further improve the robustness of the learned features. Experimental results on private and public datasets demonstrate the effectiveness of the proposed method.

Novel mining conveyor monitoring system based on quasi-distributed optical fiber accelerometer array and self-supervised learning

Belt conveyors in mining are crucial, with downtime leading to significant losses and safety hazards. Unplanned shutdowns often result from idler failures. To address this, an online monitoring system for continuous idler health assessment is proposed. Considering the large number and dense spatial distribution of idlers over long distances, this work presents a system that utilizes a quasi-distributed optical fiber accelerometer array. This array incorporates phase-sensitive optical time domain reflectometry (Phase-OTDR) interrogation technology and ultra-weak fiber Bragg gratings (UWFBGs) to effectively capture idler vibrations. The designed array achieves high-sensitivity vibration sensing with a sensitivity of 2.4 rad/g and a resolution of 1.7 mg/Hz. After collecting the vibrations of idlers by the designed accelerometer array, an automatic fault classification algorithm based on self-supervised learning (SSL) is introduced, which requires only a small number of labeled samples. By leveraging large amount of unlabeled data in the pretext task, the algorithm efficiently extracts latent features from the quasi-distributed accelerometer array. A diagnosis accuracy of 95.37 % can be achieved on a seven-class classification task with only 3.6 % labeled data (16 samples/class). This system offers a promising solution for idler monitoring, combining high sensitivity, distributed measurement capabilities, enhanced security, and superior fault detection accuracy.

Knowing the unknowns: Network traffic detection with open-set semi-supervised learning

Network traffic classification plays a crucial role in network security and network quality of service. The new emerging traffic data in the real world is unlabeled and may contain unknown classes, which introduces the problem of identifying unknown-class traffic in unlabeled data. However, existing deep learning-based methods either rely on large amounts of labeled data or assume the labeled and unlabeled data share the common label space. Therefore, they are unable to identify unknown traffic, resulting in decreased detection accuracy. This paper introduces DivinEye, a novel method for unknown network traffic detection based on open-set semi-supervised learning. DivinEye utilizes a small quantity of labeled data and a large amount of unlabeled data to train the model, achieving the ability to identify known classes accurately while also detecting unknown ones. To leverage incoming unlabeled traffic, DivinEye employs open-set semi-supervised techniques to select known-class data from unlabeled data to optimize the known traffic detection model. To detect unknown traffic, DivinEye combines the known traffic classifier and multi-binary classifier to construct an unknown traffic detection model, generating auxiliary targets for all unlabeled data as supervision to train the model. Trace-driven experiments demonstrate that DivinEye outperforms state-of-the-art methods by a large margin.

OSSEFS: An online semi-supervised ensemble fuzzy system for data streams learning with missing values

The incompleteness of data samples in data streams always affects the performance of learning model. In order to learn data streams with missing values from features and a large amount of unlabeled data, this paper proposes a semi-supervised ensemble algorithm based on evolving fuzzy systems (EFS) named OSSEFS. Aiming at missing values, a fuzzy rule based online imputation strategy is designed to improve the data feature integrity. Aiming at unlabeled data, a new ensemble fuzzy system architecture is proposed to fully utilize both labeled and unlabeled data implicit information to predict data labels accurately. In addition, an online rule parameters optimization strategy is proposed to achieve adaptive adjustment of rule distribution. Thanks to the above strategies, OSSEFS can perform well in data environments containing missing values and unlabeled data. Comparative experimental results on multiple datasets show that OSSEFS outperforms the classical semi-supervised regression algorithms.

Toward Few-Label Vertical Federated Learning

Federated Learning (FL) provides a novel paradigm for privacy-preserving machine learning, enabling multiple clients to collaborate on model training without sharing private data. To handle multi-source heterogeneous data, Vertical Federated Learning (VFL) has been extensively investigated. However, in the context of VFL, the label information tends to be kept in one authoritative client and is very limited. This poses two challenges for model training in the VFL scenario. On the one hand, a small number of labels cannot guarantee to train a well VFL model with informative network parameters, resulting in unclear boundaries for classification decisions. On the other hand, the large amount of unlabeled data is dominant and should not be discounted, and it is worthwhile to focus on how to leverage them to improve representation modeling capabilities. To address the preceding two challenges, we first introduce supervised contrastive loss to enhance the intra-class aggregation and inter-class estrangement, which is to deeply explore label information and improve the effectiveness of downstream classification tasks. Then, for unlabeled data, we introduce a pseudo-label-guided consistency mechanism to induce the classification results coherent across clients, which allows the representations learned by local networks to absorb the knowledge from other clients, and alleviates the disagreement between different clients for classification tasks. We conduct sufficient experiments on four commonly used datasets, and the experimental results demonstrate that our method is superior to the state-of-the-art methods, especially in the low-label rate scenario, and the improvement becomes more significant.

Stacked co-training for semi-supervised multi-label learning

Due to the difficulty of annotation, multi-label learning sometimes obtains a small amount of labeled data and a large amount of unlabeled data as supplements. To make up this issue, many algorithms extended the existing semi-supervised strategies in single-label patterns to multi-label applications, but failed to effectively consider the characteristics of semi-supervised multi-label learning. In this paper, a novel method named SCTML (Stacked Co-Training for Multi-Label learning) is proposed for semi-supervised multi-label learning. Through a two-layer stacking framework, SCTML learns label correlation in both base learners and meta learner, and effectively incorporates the semi-supervised assumptions of co-training, clustering and manifold. Extensive experiments demonstrate that the combination of multiple semi-supervised learning strategies effectively solves the semi-supervised multi-label learning problem.

Uncertainty estimation- and attention-based semi-supervised models for automatically delineate clinical target volume in CBCT images of breast cancer

ObjectivesAccurate segmentation of the clinical target volume (CTV) of CBCT images can observe the changes of CTV during patients' radiotherapy, and lay a foundation for the subsequent implementation of adaptive radiotherapy (ART). However, segmentation is challenging due to the poor quality of CBCT images and difficulty in obtaining target volumes. An uncertainty estimation- and attention-based semi-supervised model called residual convolutional block attention-uncertainty aware mean teacher (RCBA-UAMT) was proposed to delineate the CTV in cone-beam computed tomography (CBCT) images of breast cancer automatically.MethodsA total of 60 patients who undergone radiotherapy after breast-conserving surgery were enrolled in this study, which involved 60 planning CTs and 380 CBCTs. RCBA-UAMT was proposed by integrating residual and attention modules in the backbone network 3D UNet. The attention module can adjust channel and spatial weights of the extracted image features. The proposed design can train the model and segment CBCT images with a small amount of labeled data (5%, 10%, and 20%) and a large amount of unlabeled data. Four types of evaluation metrics, namely, dice similarity coefficient (DSC), Jaccard, average surface distance (ASD), and 95% Hausdorff distance (95HD), are used to assess the model segmentation performance quantitatively.ResultsThe proposed method achieved average DSC, Jaccard, 95HD, and ASD of 82%, 70%, 8.93, and 1.49 mm for CTV delineation on CBCT images of breast cancer, respectively. Compared with the three classical methods of mean teacher, uncertainty-aware mean-teacher and uncertainty rectified pyramid consistency, DSC and Jaccard increased by 7.89–9.33% and 14.75–16.67%, respectively, while 95HD and ASD decreased by 33.16–67.81% and 36.05–75.57%, respectively. The comparative experiment results of the labeled data with different proportions (5%, 10% and 20%) showed significant differences in the DSC, Jaccard, and 95HD evaluation indexes in the labeled data with 5% versus 10% and 5% versus 20%. Moreover, no significant differences were observed in the labeled data with 10% versus 20% among all evaluation indexes. Therefore, we can use only 10% labeled data to achieve the experimental objective.ConclusionsUsing the proposed RCBA-UAMT, the CTV of breast cancer CBCT images can be delineated reliably with a small amount of labeled data. These delineated images can be used to observe the changes in CTV and lay the foundation for the follow-up implementation of ART.

Employing feature mixture for active learning of object detection

Active learning aims to select the most informative samples for annotation from a large amount of unlabeled data, in order to reduce time-consuming and labor-intensive manual labeling efforts. Although active learning for object detection has made substantial progress in recent years, developing an accurate and efficient active learning algorithm for object detection remains a challenge. In this paper, we propose a novel unsupervised active learning method for deep object detection. This is based on our hypotheses that an object is more likely to be wrongly predicted by the model, if the prediction changes when its feature representations are slightly mixed by another feature representations at a very small ratio. Such unlabeled samples can be regarded as informative samples that can be selected by active learning. Our method employs base representations of all categories generated from the object detection network to examine the robustness of every detected object. We design a scoring function to calculate the informative score of each unlabeled image. We conduct extensive experiments on two public datasets, i.e., PASCAL VOC and MS-COCO. Experiment results show that our approach consistently outperforms state-of-the-art single-model based methods with significant margins. Our approach also performs on par with multi-model based methods, at much lesser computational cost.

Semi-Supervised Training for (Pre-Stack) Seismic Data Analysis

A lack of labeled examples is a problem in different domains, such as text and image processing, medicine, and static reservoir characterization, because supervised learning relies on vast volumes of these data to perform successfully, but this is quite expensive. However, large amounts of unlabeled data exist in these domains. The deep semi-supervised learning (DSSL) approach leverages unlabeled data to improve supervised learning performance using deep neural networks. This approach has succeeded in image recognition, text classification, and speech recognition. Nevertheless, there have been few works on pre-stack seismic reservoir characterization, in which knowledge of rock and fluid properties is fundamental for oil exploration. This paper proposes a methodology to estimate acoustic impedance using pre-stack seismic data and DSSL with a recurrent neural network. The few labeled datasets for training were pre-processed from raw seismic and acoustic impedance data from five borehole logs. The results showed that the acoustic impedance estimation at the well location and outside it was better predicted by the DSSL compared to the supervised version of the same neural network. Therefore, employing a large amount of unlabeled data can be helpful in the development of seismic data interpretation systems.

Semi-supervised learning in diagnosis of infant hip dysplasia towards multisource ultrasound images.

Automated diagnosis of infant hip dysplasia is heavily affected by the individual differences among infants and ultrasound machines. Hip sonographic images of 493 infants from various ultrasound machines were collected in the Department of Orthopedics in Yangzhou Maternal and Child Health Care Service Centre. Herein, we propose a semi-supervised learning method based on a feature pyramid network (FPN) and a contrastive learning scheme based on a Siamese architecture. A large amount of unlabeled data of ultrasound images was used via the Siamese network in the pre-training step, and then a small amount of annotated data for anatomical structures was adopted to train the model for landmark identification and standard plane recognition. The method was evaluated on our collected dataset. The method achieved a mean Dice similarity coefficient (DSC) of 0.7873 and a mean Hausdorff distance (HD) of 5.0102 in landmark identification, compared to the model without contrastive learning, which had a mean DSC of 0.7734 and a mean HD of 6.1586. The accuracy, precision, and recall of standard plane recognition were 95.4%, 91.64%, and 94.86%, respectively. The corresponding area under the curve (AUC) was 0.982. This study proposes a semi-supervised deep learning method following Graf's principle, which can better utilize a large volume of ultrasound images from various devices and infants. This method can identify the landmarks of infant hips more accurately than manual operators, thereby improving the efficiency of diagnosis of infant hip dysplasia.

Self-training with dual uncertainty for semi-supervised MRI image segmentation

The goal of semi-supervised MRI image segmentation is to train high-accuracy segmentation models using a small amount of labeled data and a large amount of unlabeled data, thereby alleviating the burden of manual annotation by physicians. Self-training is a traditional semi-supervised learning method aimed at acquiring additional image features by generating pseudo-labels for unlabeled data. However, this method may lead to an unstable training process due to differences between samples. Additionally, the predictions of model may contain erroneous noise, leading to low-confidence pseudo-labels. To address these issues, we proposed a dual uncertainty-guided self-training framework (DUST) for semi-supervised MRI image segmentation. Specifically, the two uncertainties consist of sample-level uncertainty and pixel-level uncertainty. The sample-level uncertainty is intended to achieve a stable and smooth training process. The pixel-level uncertainty is intended to rectify pseudo-labels. We conducted a series of experiments on two public MRI image datasets, ACDC2017 and Prostate2018. Compared to the baseline, we improved the Dice scores by 5.0% and 4.0% over the two datasets, respectively. Furthermore, the experimental results indicate that the proposed method has certain advantages compared to the comparative methods. This validates the feasibility and applicability of our method.

CDEST: Class Distinguishability-Enhanced Self-Training Method for Adopting Pre-Trained Models to Downstream Remote Sensing Image Semantic Segmentation

The self-supervised learning (SSL) technique, driven by massive unlabeled data, is expected to be a promising solution for semantic segmentation of remote sensing images (RSIs) with limited labeled data, revolutionizing transfer learning. Traditional ‘local-to-local’ transfer from small, local datasets to another target dataset plays an ever-shrinking role due to RSIs’ diverse distribution shifts. Instead, SSL promotes a ‘global-to-local’ transfer paradigm, in which generalized models pre-trained on arbitrarily large unlabeled datasets are fine-tuned to the target dataset to overcome data distribution shifts. However, the SSL pre-trained models may contain both useful and useless features for the downstream semantic segmentation task, due to the gap between the SSL tasks and the downstream task. To adapt such pre-trained models to semantic segmentation tasks, traditional supervised fine-tuning methods that use only a small number of labeled samples may drop out useful features due to overfitting. The main reason behind this is that supervised fine-tuning aims to map a few training samples from the high-dimensional, sparse image space to the low-dimensional, compact semantic space defined by the downstream labels, resulting in a degradation of the distinguishability. To address the above issues, we propose a class distinguishability-enhanced self-training (CDEST) method to support global-to-local transfer. First, the self-training module in CDEST introduces a semi-supervised learning mechanism to fully utilize the large amount of unlabeled data in the downstream task to increase the size and diversity of the training data, thus alleviating the problem of biased overfitting of the model. Second, the supervised and semi-supervised contrastive learning modules of CDEST can explicitly enhance the class distinguishability of features, helping to preserve the useful features learned from pre-training while adapting to downstream tasks. We evaluate the proposed CDEST method on four RSI semantic segmentation datasets, and our method achieves optimal experimental results on all four datasets compared to supervised fine-tuning as well as three semi-supervised fine-tuning methods.

SDMI-Net: Spatially Dependent Mutual Information Network for semi-supervised medical image segmentation

Semi-supervised medical image segmentation strives to polish deep models with a small amount of labeled data and a large amount of unlabeled data. The efficiency of most semi-supervised medical image segmentation methods based on voxel-level consistency learning is affected by low-confidence voxels. In addition, voxel-level consistency learning fails to consider the spatial correlation between neighboring voxels. To encourage reliable voxel-level consistent learning, we propose a dual-teacher affine consistent uncertainty estimation method to filter out some voxels with high uncertainty. Moreover, we design the spatially dependent mutual information module, which enhances the spatial dependence between neighboring voxels by maximizing the mutual information between the local voxel blocks predicted from the dual-teacher models and the student model, enabling consistent learning at the block level. On two benchmark medical image segmentation datasets, including the Left Atrial Segmentation Challenge dataset and the BraTS-2019 dataset, our method achieves state-of-the-art performance in both quantitative and qualitative aspects.

Diffusion Language-Shapelets for Semi-supervised Time-Series Classification

Semi-supervised time-series classification could effectively alleviate the issue of lacking labeled data. However, existing approaches usually ignore model interpretability, making it difficult for humans to understand the principles behind the predictions of a model. Shapelets are a set of discriminative subsequences that show high interpretability in time series classification tasks. Shapelet learning-based methods have demonstrated promising classification performance. Unfortunately, without enough labeled data, the shapelets learned by existing methods are often poorly discriminative, and even dissimilar to any subsequence of the original time series. To address this issue, we propose the Diffusion Language-Shapelets model (DiffShape) for semi-supervised time series classification. In DiffShape, a self-supervised diffusion learning mechanism is designed, which uses real subsequences as a condition. This helps to increase the similarity between the learned shapelets and real subsequences by using a large amount of unlabeled data. Furthermore, we introduce a contrastive language-shapelets learning strategy that improves the discriminability of the learned shapelets by incorporating the natural language descriptions of the time series. Experiments have been conducted on the UCR time series archive, and the results reveal that the proposed DiffShape method achieves state-of-the-art performance and exhibits superior interpretability over baselines.

Robust transfer subspace learning based on low-rank and sparse representation for bearing fault diagnosis

With the development of industrial intelligence, data-driven fault diagnosis plays an important role in prognostics and health management. However, there is usually a large amount of unlabeled data from different working conditions, making cross-domain fault diagnosis unstable and inflexible. To deal with this issue, we propose two novel transfer subspace learning methods based on the low-rank sparse representation (LRSR), called LRSR-G and LRSR-R. Specifically, LRSR-G integrates an additional matrix with LRSR to characterize the Gaussian noise for robustness, as well as capture global and local structures. Furthermore, LRSR-R adaptively learns the label matrix from samples instead of using the binary labeling matrix in LRSR-G, thus providing the possibility to improve the flexibility. In addition, we develop two efficient algorithms using the alternating direction method of multipliers to solve the proposed LRSR-G and LRSR-R. Extensive experiments are conducted on the Case Western Reserve University dataset and Jiangnan University (JNU) dataset. The results show that the proposed LRSR-G and LRSR-R perform better than the existing methods, while LRSR-R has more potential in cross-domain fault diagnosis tasks.

Kernel Path for Semisupervised Support Vector Machine.

Semisupervised support vector machine (S3VM) is a powerful semisupervised learning model that can use large amounts of unlabeled data to train high-quality classification models. The choice of kernel parameters in the kernel function determines the mapping between the input space and the feature space and is crucial to the performance of the S3VM. Kernel path algorithms have been widely recognized as one of the most efficient tools to trace the solutions with respect to a kernel parameter. However, existing kernel path algorithms are limited to convex problems, while S3VM is nonconvex problem. To address this challenging problem, in this article, we first propose a kernel path algorithm of S3VM (KPS3VM), which can track the solutions of the nonconvex S3VM with respect to a kernel parameter. Specifically, we estimate the position of the breakpoint by monitoring the change of the sample sets. In addition, we also use an incremental and decremental learning algorithm to deal with the Karush-Khun-Tucker violating samples in the process of tracking the solutions. More importantly, we prove the finite convergence of our KPS3VM algorithm. Experimental results on various benchmark datasets not only validate the effectiveness of our KPS3VM algorithm but also show the advantage of choosing the optimal kernel parameters.