Semi-supervised Learning Framework Research Articles

Slr-kit: A semi-supervised machine learning framework for systematic literature reviews

Systematic Literature Review (SLR) is nowadays a challenging task due to the large number of papers that typically compose the scientific material of the topic to review. Recently, a lot of research effort has been devoted to automate, even partially, the stages of an SLR. This paper proposes the design and implementation of a workflow and a set of tools – called slr-kit – to support key tasks in an SLR. The proposed approach leverages a semi-supervised strategy, in which time-consuming processes are carried out using automatic tools, whereas manual tasks have been optimized by carefully designed support tools to reduce the overall required effort. Important parts of the workflow include the extraction of key terms directly from the abstracts of the papers to survey, and the subsequent topic modeling that allows for a thematic clustering of the corpus of papers. In the proposed workflow, the former task is carried out by exploiting a novel tool, called FAst WOrd Classifier (FAWOC). The latter, instead, is designed to be automatically carried out by leveraging an ad-hoc solution based on the application of the Latent Dirichlet Allocation (LDA) algorithm. The result of the process consists in a set of statistics regarding the relationship among papers, topics, and their trend of publication on journals and conference proceedings. The validity of the method is demonstrated with an application to a dataset related to the scientific field of NLP, while its accuracy is assessed by the manual examination of the results by domain experts.

A Well-Overflow Prediction Algorithm Based on Semi-Supervised Learning

Oil drilling is the core process of oil and natural gas resources exploitation. Well overflow is one of the biggest threats to safety drilling. Prediction of the overflow in advance can effectively avoid the occurrence of this kind of accident. However, the drilling history has unbalanced distribution, and labeling data is a time-consuming and laborious job. To address this issue, an overflow-prediction algorithm based on semi-supervised learning is designed in this paper, which can accurately predict overflow 10 min in advance when the labeled data are limited. Firstly, a three-step feature-selection algorithm is conducted to extract 22 features, and the time series samples are constructed through a 500-width sliding window with step size 1. Then, the Mean Teacher model with Jitter noise is employed to train the labeled and unlabeled data at the same time, in which a fused CNN-LSTM network is built for time-series prediction. Compared with supervised learning and other semi-supervised learning frameworks, the results show that the proposed model based on only 200 labeled samples is able to achieve the same effect as supervised learning method using 1000 labeled samples, and the prediction accuracy can reach 87.43% 10 min in advance. With the increase in the proportion of unlabeled samples, the performance of the model can sustain a rise within a certain range.

Fault Diagnosis for Power Transformers through Semi-Supervised Transfer Learning.

The fault diagnosis of power transformers is a challenging problem. The massive multisource fault is heterogeneous, the type of fault is undetermined sometimes, and one device has only met a few kinds of faults in the past. We propose a fault diagnosis method based on deep neural networks and a semi-supervised transfer learning framework called adaptive reinforcement (AR) to solve the above limitations. The innovation of this framework consists of its enhancement of the consistency regularization algorithm. The experiments were conducted on real-world 110 kV power transformers’ three-phase fault grounding currents of the iron cores from various devices with four types of faults: Phases A, B, C and ABC to ground. We trained the model on the source domain and then transferred the model to the target domain, which included the unbalanced and undefined fault datasets. The results show that our proposed model reaches over accuracy in classifying the type of fault and outperforms other popular networks. Our AR framework fits target devices’ fault data with fewer dozen epochs than other novel semi-supervised techniques. Combining the deep neural network and the AR framework helps diagnose the power transformers, which lack diagnosis knowledge, with much less training time and reliable accuracy.

Geometric generative adversarial net based multiple methods for spectrum sensing in cognitive radio networks

The majority of recently developed approaches require a significant number of labelled samples. The proposed system are dedicated to using less marked samples for automatic modulation detection in the cognitive radio domain. The proposed signal classifier generative adversarial nets (GANs) methodology is a semi-supervised learning framework that focuses on adversarial analysis GANs are a major step forward in the development of competitive generative networks, and they've spawned a slew of apparently unrelated versions. The discovery of a single geometric form in GAN and its derivatives is one of the paper's key contributions. In three geometric stages, by demonstrate how to train an adversarial generative model: updating the discriminator parameter away from the separating hyperplane, looking for the separating hyperplane, and updating the generator along the usual vector route of the separating hyperplane. The shortcomings in current approaches are shown by this geometric intuition, leading us to suggest a new geometric GAN formulation that maximizes the margin using SVM separating hyperplane. An equilibrium is reached between the discriminator and generator in the geometric GAN, according to our theoretical research. Furthermore, detailed computational results showing the superior efficiency of the GAN engineering network were obtained.

Augmented Semi-Supervised Learning for Salient Object Detection with Edge Computing

Salient object detection (SOD) from raw sensor images in the edge networks can effectively speed up the decision-making process in the complex environments, because it simulates the mechanism of human attention to identify salient objects from images. The success of supervised deep learning approaches have been widely proved SOD field. However, the imbalanced and limited training data at each edge device pose a huge challenge for us to deploy deep learning methods in the edge computing environments. In this article, we propose a cloud-edge distributed augmented semi-supervised learning architecture for SOD over the edge networks. The framework consists of two components: the base classification networks are employed in different edge nodes, and the reverse augmented network is employed in cloud. First, the base classification networks are trained with data from edge nodes while the reverse augmented network is trained with the whole data. Then, we concatenate each base classification network with reverse augmented network, thus the latter network can help the training of former network. Finally, we integrate the outputs of all base classification network to generate the pseudo-labels, which are used for semi-supervised learning of the augment network. We demonstrated a convincing performance of our semi-supervised learning framework on four bench-marked data-sets. These results show that our augmented semi-supervised learning framework can outperform other optimization strategies on deep learning for the edge computing.

From unsupervised to semi-supervised adversarial domain adaptation in electroencephalography-based sleep staging

Objective. The recent breakthrough of wearable sleep monitoring devices has resulted in large amounts of sleep data. However, as limited labels are available, interpreting these data requires automated sleep stage classification methods with a small need for labeled training data. Transfer learning and domain adaptation offer possible solutions by enabling models to learn on a source dataset and adapt to a target dataset. Approach. In this paper, we investigate adversarial domain adaptation applied to real use cases with wearable sleep datasets acquired from diseased patient populations. Different practical aspects of the adversarial domain adaptation framework are examined, including the added value of (pseudo-)labels from the target dataset and the influence of domain mismatch between the source and target data. The method is also implemented for personalization to specific patients. Main results. The results show that adversarial domain adaptation is effective in the application of sleep staging on wearable data. When compared to a model applied on a target dataset without any adaptation, the domain adaptation method in its simplest form achieves relative gains of 7%–27% in accuracy. The performance in the target domain is further boosted by adding pseudo-labels and real target domain labels when available, and by choosing an appropriate source dataset. Furthermore, unsupervised adversarial domain adaptation can also personalize a model, improving the performance by 1%–2% compared to a non-personalized model. Significance. In conclusion, adversarial domain adaptation provides a flexible framework for semi-supervised and unsupervised transfer learning. This is particularly useful in sleep staging and other wearable electroencephalography applications. (Clinical trial registration number: S64190.)

A Fully Automated Multimodal MRI-Based Multi-Task Learning for Glioma Segmentation and IDH Genotyping.

The accurate prediction of isocitrate dehydrogenase (IDH) mutation and glioma segmentation are important tasks for computer-aided diagnosis using preoperative multimodal magnetic resonance imaging (MRI). The two tasks are ongoing challenges due to the significant inter-tumor and intra-tumor heterogeneity. The existing methods to address them are mostly based on single-task approaches without considering the correlation between the two tasks. In addition, the acquisition of IDH genetic labels is expensive and costly, resulting in a limited number of IDH mutation data for modeling. To comprehensively address these problems, we propose a fully automated multimodal MRI-based multi-task learning framework for simultaneous glioma segmentation and IDH genotyping. Specifically, the task correlation and heterogeneity are tackled with a hybrid CNN-Transformer encoder that consists of a convolutional neural network and a transformer to extract the shared spatial and global information learned from a decoder for glioma segmentation and a multi-scale classifier for IDH genotyping. Then, a multi-task learning loss is designed to balance the two tasks by combining the segmentation and classification loss functions with uncertain weights. Finally, an uncertainty-aware pseudo-label selection is proposed to generate IDH pseudo-labels from larger unlabeled data for improving the accuracy of IDH genotyping by using semi-supervised learning. We evaluate our method on a multi-institutional public dataset. Experimental results show that our proposed multi-task network achieves promising performance and outperforms the single-task learning counterparts and other existing state-of-the-art methods. With the introduction of unlabeled data, the semi-supervised multi-task learning framework further improves the performance of glioma segmentation and IDH genotyping. The source codes of our framework are publicly available at https://github.com/miacsu/MTTU-Net.git.

A novel myocardial infarction localization method using multi-branch DenseNet and spatial matching-based active semi-supervised learning

Individual differences among patients and the high cost of manual labeling are major challenges for electrocardiogram (ECG) diagnosis algorithms. To tackle this problem, we develop a novel deep active semi-supervised learning framework for myocardial infarction localization based on 12-lead ECG signals. First, a new deep learning model named multi-branch densely connected convolutional network (MB-DenseNet) is designed to automatically extract and fuse heartbeat features from 12-lead ECG signals. Moreover, to improve the classification results for new patients, we propose a novel active semi-supervised learning (ASSL) mechanism to update the model. Active learning (AL) is employed to improve the classification ability of the initial model firstly, and then a new semi-supervised learning method named self-training with spatial matching (STSM) is designed to update the model further. Based on the artificial knowledge from AL, STSM combines spatial matching algorithm and the trained model to label valuable unlabeled samples automatically. We conduct experiments based on intra-patient and patient-specific schemes using the PTB database. The MB-DenseNet yields an accuracy of 99.87% under the intra-patient scheme. For the patient-specific scheme, the updated model achieves an accuracy of 96.09%. Compared with state-of-the-art methods, our method can effectively reduce manual labeling while achieving comparable performance.

TBAL: Two-stage batch-mode active learning for image classification

The success of deep learning applications relies on a large number of labeled data. Active learning aims at identifying most informative unlabeled samples for labeling so as to achieve comparable performance with as few labeled data as possible. A two-stage batch-mode active learning (TBAL) method is proposed in this work. In the first stage of TBAL, we propose a new semi-supervised learning framework that offers accurate uncertainty estimation and finds a high-dimensional feature representation for unlabeled samples. In other words, it leverages a small number of labeled data as well as a large number of unlabeled data. In the second stage of TBAL, we propose a novel technique called cluster re-balancing (CRB). It takes correlations within a batch into account and fits a query strategy in batch mode. The TBAL method can obtain informative batches of samples by considering uncertainty and diversity jointly in a high-dimensional feature space. Moreover, in order to compare the robustness of semi-supervised learning based active learning methods, a new evaluation protocol based on transfer learning is introduced. Experiments on SVHN, CIFAR-10 and CUB-200-2011 datasets show that our proposed CRB based query strategy that employs a semi-supervised model with a good balance between uncertainty and diversity outperforms various baselines as well as other state-of-the-art methods by a clear margin.

Semi-supervised adapted HMMs for P2P credit scoring systems with reject inference.

The majority of current credit-scoring models, used for loan approval processing, are generally built on the basis of the information from the accepted credit applicants whose ability to repay the loan is known. This situation generates what is called the selection bias, presented by a sample that is not representative of the population of applicants, since rejected applications are excluded. Thus, the impact on the eligibility of those models from a statistical and economic point of view. Especially for the models used in the peer-to-peer lending platforms, since their rejection rate is extremely high. The method of inferring rejected applicants information in the process of construction of the credit scoring models is known as reject inference. This study proposes a semi-supervised learning framework based on hidden Markov models (SSHMM), as a novel method of reject inference. Real data from the Lending Club platform, the most used online lending marketplace in the United States as well as the rest of the world, is used to experiment the effectiveness of our method over existing approaches. The results of this study clearly illustrate the proposed method’s superiority, stability, and adaptability.

Semi-Supervised Ensemble Learning Framework for Accelerating Power System Transient Stability Knowledge Base Generation

Machine learning approaches have exhibited high potential in power system transient stability assessment (TSA), yet their initial preparation stages of stability knowledge base generation (SKBG) based on time-domain simulations often undergo high computational costs. In fact, how to ease the heavy computational burden of SKBG without sacrificing the reliability is still a significant challenge. To address this problem, this paper develops a semi-supervised ensemble learning (SSEL) framework for reliable SKBG acceleration, without additional need for computing hardware upgrading. In particular, it performs detailed simulations for a minority of cases and fast simulations for the majority ones to reduce the total computation time. Considering the absence of stability status information of those fast simulated cases, an SSEL scheme is systematically designed to reliably label their stability status. Before implementing SSEL, two concise feature descriptors are first introduced to efficiently extract transient features from multiplex system trajectories. Then, all the cases are characterized in a unified feature space, whereby a series of semi-supervised support vector machines are trained in randomly formed subspaces. Afterward, these single machines are systematically combined to derive an enhanced SSEL model, which is able to make reliable and robust labeling decisions. Further, a backtrace strategy is carefully devised for SSEL, so as to maintain the high reliability of SKBG. Test results on the IEEE 39-bus system and the realistic GD Power Grid in South China illustrate the excellent performances of the proposed framework on SKBG acceleration.

NAS-SGAN: A Semi-Supervised Generative Adversarial Network Model for Atypia Scoring of Breast Cancer Histopathological Images.

Nuclear atypia scoring (NAS), forms a significant factor in determining individualized treatment plans and also for the prognosis of the disease. Automation of cancer grading using quantitative image-based analysis of histopathological images can circumvent the shortcomings of the prevailing manual grading and can assist the pathologists in cancer diagnosis. However, developing such a robust classifier model require sufficient amount of annotated data, while the labeled histopathological images are scarce and expensive to procure as annotation forms a time-consuming and laborious task. Hence, a semi-supervised learning framework combined with the deep neural network based generative adversarial training, that can improve the performance of the classification model with limited annotated data, is proposed in this paper. The proposed NAS-SGAN model consists of discriminator and generator models that are trained in an adversarial manner using both labeled and unlabeled samples. The discriminator model is designed as an unsupervised model stacked over the supervised model sharing the model parameters and learns the data distribution by extracting the discriminative features. The generator model is trained over a stable feature matching objective function following a composite GAN architecture, and its for the first time the semi-supervised GAN model is explored for the grading of breast cancer. Experimental analysis shows that the proposed model could better discriminate different cancer grades thereby improving the robustness and accuracy of the system, even with limited amount of labeled samples.

Endoscopy image enhancement method by generalized imaging defect models based adversarial training

Objective. Smoke, uneven lighting, and color deviation are common issues in endoscopic surgery, which have increased the risk of surgery and even lead to failure. Approach. In this study, we present a new physics model driven semi-supervised learning framework for high-quality pixel-wise endoscopic image enhancement, which is generalizable for smoke removal, light adjustment, and color correction. To improve the authenticity of the generated images, and thereby improve the network performance, we integrated specific physical imaging defect models with the CycleGAN framework. No ground-truth data in pairs are required. In addition, we propose a transfer learning framework to address the data scarcity in several endoscope enhancement tasks and improve the network performance. Main results. Qualitative and quantitative studies reveal that the proposed network outperforms the state-of-the-art image enhancement methods. In particular, the proposed method performs much better than the original CycleGAN, for example, the structural similarity improved from 0.7925 to 0.8648, feature similarity for color images from 0.8917 to 0.9283, and quaternion structural similarity from 0.8097 to 0.8800 in the smoke removal task. Experimental results of the proposed transfer learning method also reveal its superior performance when trained with small datasets of target tasks. Significance. Experimental results on endoscopic images prove the effectiveness of the proposed network in smoke removal, light adjustment, and color correction, showing excellent clinical usefulness.

A framework of active learning and semi-supervised learning for lithology identification based on improved naive Bayes

Lithology identification is the basis of energy exploration and reservoir evaluation, intelligent and accurate identification of underground lithology is a key issue. The establishment of a machine learning lithology identification model using logging data is a hot research direction in recent years. However, the logging data has a high degree of non-linearity and multi-response characteristics, and there are insufficient numbers of labeled samples in the training data set. These will eventually affect the modeling accuracy and may cause over-fitting. Therefore, a framework of active learning and semi-supervised learning for lithology identification based on improved naive Bayes (ALSLINB) is proposed. The contributions are fourfold: (i) The Gaussian mixture model (GMM) based on the EM algorithm is used to estimate the probability density of the log data, which fits the probability distribution of the nonlinear multi-response log data. (ii) A framework combining active learning (AL) and semi-supervised learning is proposed for the expansion of labeled samples in the training data set. (iii) The application of pseudo-labeling detection technology can effectively improve the authenticity of pseudo-label samples. (iv) Different from the general deterministic lithology identification method, the result of the ALSLINB algorithm corresponds to the probability score, which provides an auxiliary basis for the prediction result. Finally, the ALSLINB algorithm is applied to two different data sets for a large number of experiments and compared with the related baseline methods to verify its effectiveness and generalization ability. The result proves that the ALSLINB algorithm can complete the lithology recognition task well and has high accuracy and robustness, which provides a new direction for intelligent lithology identification.

Semi-Supervised Adversarial Semantic Segmentation Network Using Transformer and Multiscale Convolution for High-Resolution Remote Sensing Imagery

Semantic segmentation is a crucial approach for remote sensing interpretation. High-precision semantic segmentation results are obtained at the cost of manually collecting massive pixelwise annotations. Remote sensing imagery contains complex and variable ground objects and obtaining abundant manual annotations is expensive and arduous. The semi-supervised learning (SSL) strategy can enhance the generalization capability of a model with a small number of labeled samples. In this study, a novel semi-supervised adversarial semantic segmentation network is developed for remote sensing information extraction. A multiscale input convolution module (MICM) is designed to extract sufficient local features, while a Transformer module (TM) is applied for long-range dependency modeling. These modules are integrated to construct a segmentation network with a double-branch encoder. Additionally, a double-branch discriminator network with different convolution kernel sizes is proposed. The segmentation network and discriminator network are jointly trained under the semi-supervised adversarial learning (SSAL) framework to improve its segmentation accuracy in cases with small amounts of labeled data. Taking building extraction as a case study, experiments on three datasets with different resolutions are conducted to validate the proposed network. Semi-supervised semantic segmentation models, in which DeepLabv2, the pyramid scene parsing network (PSPNet), UNet and TransUNet are taken as backbone networks, are utilized for performance comparisons. The results suggest that the approach effectively improves the accuracy of semantic segmentation. The F1 and mean intersection over union (mIoU) accuracy measures are improved by 0.82–11.83% and 0.74–7.5%, respectively, over those of other methods.

A Semisupervised Learning Framework for Recognition and Classification of Defects in Transient Thermography Detection

The defect classification task is of great benefit to evaluating the safety performance of equipment and providing useful feedback information for discovering production process problems. In this article, we present a semisupervised learning (SSL) framework for transient thermography detection to employ the temporal and spatial information encoded into the three-dimensional transient thermal tensor data and provide pixel-level classification results for defect types. The time- and frequency-domain physical models for the transient thermal evolution of different kinds of defects are established to illustrate the theoretical foundation of defects classification based on transient thermography. The semisupervised multiclass Laplacian support vector machine is proposed to enable involving the abundant unlabeled data for enhancing learning performance in practical industrial applications where labeled samples are insufficient and labeling work is costly and laborious. A case study on silicone insulating materials with various types of artificial simulated internal defects validates the stronger generalized ability of the proposed method. This work, for the first time, proposes an SSL framework in transient thermography-based defect detection studies. It is believed that our proposed method is quite inspired for introducing SSL techniques to transient thermography for preferable performance in practical industrial applications.

Semi-Supervised Deep Transfer Learning for Benign-Malignant Diagnosis of Pulmonary Nodules in Chest CT Images.

Lung cancer is the leading cause of cancer deaths worldwide. Accurately diagnosing the malignancy of suspected lung nodules is of paramount clinical importance. However, to date, the pathologically-proven lung nodule dataset is largely limited and is highly imbalanced in benign and malignant distributions. In this study, we proposed a Semi-supervised Deep Transfer Learning (SDTL) framework for benign-malignant pulmonary nodule diagnosis. First, we utilize a transfer learning strategy by adopting a pre-trained classification network that is used to differentiate pulmonary nodules from nodule-like tissues. Second, since the size of samples with pathological-proven is small, an iterated feature-matching-based semi-supervised method is proposed to take advantage of a large available dataset with no pathological results. Specifically, a similarity metric function is adopted in the network semantic representation space for gradually including a small subset of samples with no pathological results to iteratively optimize the classification network. In this study, a total of 3,038 pulmonary nodules (from 2,853 subjects) with pathologically-proven benign or malignant labels and 14,735 unlabeled nodules (from 4,391 subjects) were retrospectively collected. Experimental results demonstrate that our proposed SDTL framework achieves superior diagnosis performance, with accuracy = 88.3%, AUC = 91.0% in the main dataset, and accuracy = 74.5%, AUC = 79.5% in the independent testing dataset. Furthermore, ablation study shows that the use of transfer learning provides 2% accuracy improvement, and the use of semi-supervised learning further contributes 2.9% accuracy improvement. Results implicate that our proposed classification network could provide an effective diagnostic tool for suspected lung nodules, and might have a promising application in clinical practice.

Consistency-based Semi-supervised Learning Framework for Power Line Insulators Detection

Abstract It is commonly believed that fast and accurate object detection is the prerequisite for power line insulators automatic fault diagnosis. While the insulators detection task requires a huge number of labeled images to guarantee its performance, placing bounding boxes for every object in each image is time-consuming and requires professional knowledge. To alleviate this problem, we propose CASD, a two-stage semi-supervised learning framework for power line insulators detection based on consistency regularization and data augmentation. Firstly, we train a teacher model on labeled images, then generate pseudo labels for unlabeled images using the trained model, and the confidence-based thresholding is introduced to control the quality of pseudo labels. In the second stage, we apply strong data augmentation to unlabeled images and use both the labeled and unlabeled images to train the CASD. Consistency regularization is introduced in the calculation of unsupervised loss to improve the accuracy and robustness of the model. Finally, we compare CASD with supervised learning, data augmentation and self-training methods on insulators aerial image dataset. The results show that the proposed method performs better than all the three baseline methods, and the detection accuracy of CASD is far better than the supervised learning method in the case of less available images. The comparison results further prove the efficacy of the proposed method.

Semi-supervised machine learning framework for network intrusion detection

Semi-supervised machine learning framework for network intrusion detection

Fine-grained interactive attention learning for semi-supervised white blood cell classification

White blood cell (WBC) is an essential part of the human immune system. To diagnose blood diseases, hematologists have to think about the WBC information. For instance, the number of each type of WBCs often implies the health condition of the human body. Thus, the classification of white blood cell images plays a significant role in the medical diagnosis process. However, manual WBC inspection is time-consuming and labor-intensive for experts, which means automated classification methods are needed for WBC recognition. Another problem is that the traditional automatic recognition system needs a large amount of annotated medical images for training, which is highly costly. In this respect, the semi-supervised learning framework has recently been widely used for medical diagnosis due to its specificity, which can explore relevant information from massive unlabeled data. In this study, a novel semi-supervised white blood cell classification method is proposed, named by Fine-grained Interactive Attention Learning (FIAL). It consists of a Semi-Supervised Teacher-Student (SSTS) module and a Fine-Grained Interactive Attention (FGIA) mechanism. In detail, SSTS employs limited labeled WBC images and generates predicted probability vectors for a large amount of unlabeled WBC samples, like a human. After top-k selection in predicted probabilities, the efficient data can be exploited from unlabeled WBC images for training. With a very small amount of annotated WBC images, FIAL achieves an average accuracy of 93.2% on BCCD dataset when giving 75 labeled images for each category, which sufficiently elaborates our excellent capability on semi-supervised white blood cell image classification task.