Virtual Adversarial Training Research Articles

A Semi-Supervised Approach for Partial Discharge Recognition Combining Graph Convolutional Network and Virtual Adversarial Training

A Semi-Supervised Approach for Partial Discharge Recognition Combining Graph Convolutional Network and Virtual Adversarial Training

Enhanced Radar Emitter Recognition with Virtual Adversarial Training: A Semi-Supervised Framework

Enhanced Radar Emitter Recognition with Virtual Adversarial Training: A Semi-Supervised Framework

Diversity matters: Cross-head mutual mean-teaching for semi-supervised medical image segmentation

Semi-supervised medical image segmentation (SSMIS) has witnessed substantial advancements by leveraging limited labeled data and abundant unlabeled data. Nevertheless, existing state-of-the-art (SOTA) methods encounter challenges in accurately predicting labels for the unlabeled data, giving rise to disruptive noise during training and susceptibility to erroneous information overfitting. Moreover, applying perturbations to inaccurate predictions further impedes consistent learning. To address these concerns, we propose a novel cross-head mutual mean-teaching network (CMMT-Net) incorporated weak-strong data augmentations, thereby benefiting both co-training and consistency learning. More concretely, our CMMT-Net extends the cross-head co-training paradigm by introducing two auxiliary mean teacher models, which yield more accurate predictions and provide supplementary supervision. The predictions derived from weakly augmented samples generated by one mean teacher are leveraged to guide the training of another student with strongly augmented samples. Furthermore, two distinct yet synergistic data perturbations at the pixel and region levels are introduced. We propose mutual virtual adversarial training (MVAT) to smooth the decision boundary and enhance feature representations, and a cross-set CutMix strategy to generate more diverse training samples for capturing inherent structural data information. Notably, CMMT-Net simultaneously implements data, feature, and network perturbations, amplifying model diversity and generalization performance. Experimental results on three publicly available datasets indicate that our approach yields remarkable improvements over previous SOTA methods across various semi-supervised scenarios. The code is available at https://github.com/Leesoon1984/CMMT-Net.

Semisupervised Radar Intrapulse Signal Modulation Classification With Virtual Adversarial Training

Semisupervised Radar Intrapulse Signal Modulation Classification With Virtual Adversarial Training

Refined Semi-Supervised Modulation Classification: Integrating Consistency Regularization and Pseudo-Labeling Techniques

Automatic modulation classification (AMC) plays a crucial role in wireless communication by identifying the modulation scheme of received signals, bridging signal reception and demodulation. Its main challenge lies in performing accurate signal processing without prior information. While deep learning has been applied to AMC, its effectiveness largely depends on the availability of labeled samples. To address the scarcity of labeled data, we introduce a novel semi-supervised AMC approach combining consistency regularization and pseudo-labeling. This method capitalizes on the inherent data distribution of unlabeled data to supplement the limited labeled data. Our approach involves a dual-component objective function for model training: one part focuses on the loss from labeled data, while the other addresses the regularized loss for unlabeled data, enhanced through two distinct levels of data augmentation. These combined losses concurrently refine the model parameters. Our method demonstrates superior performance over established benchmark algorithms, such as decision trees (DTs), support vector machines (SVMs), pi-models, and virtual adversarial training (VAT). It exhibits a marked improvement in the recognition accuracy, particularly when the proportion of labeled samples is as low as 1–4%.

Mixup Virtual Adversarial Training for Robust Vision Transformers

Mixup Virtual Adversarial Training for Robust Vision Transformers

MLC: Multi-level consistency learning for semi-supervised left atrium segmentation

Atrial fibrillation is the most common type of arrhythmia associated with a high mortality rate. Left atrium segmentation is crucial for the diagnosis and treatment of atrial fibrillation. Accurate left atrium segmentation with limited labeled data is a tricky problem. In this paper, a novel multi-level consistency semi-supervised learning method is proposed for left atrium segmentation from 3D magnetic resonance images. The proposed framework can efficiently utilize limited labeled data and large amounts of unlabeled data by performing consistency predictions under task level, data level, and feature level perturbations. For task consistency, the segmentation results and signed distance maps were used for both segmentation and distance estimation tasks. For data level perturbation, random flips (horizontal or vertical) were introduced for unlabeled data. Moreover, based on virtual adversarial training, we design a multi-layer feature perturbation in the structure of skipping connection. Our method is evaluated on the publicly available Left Atrium Segmentation Challenge dataset version 2018. For the model trained with a label rate of 20%, the evaluation metrics Dice, Jaccard, ASD, and 95HD are 91.69%, 84.71%, 1.43 voxel, and 5.44 voxel, respectively. The experimental results show that the proposed method outperforms other semi-supervised learning methods and even achieves better performance than the fully supervised V-Net.

A domain knowledge powered hybrid regularization strategy for semi-supervised breast cancer diagnosis

Semi-supervised learning has attracted much attention in medical image analysis as annotating medical images is substantially difficult. Existing semi-supervised learning methods mainly utilize some regularization strategies to encourage a network to make similar predictions for each sample under various perturbations (e.g. rotating and flipping). However, as these strategies are originally designed for natural images, they do not fully utilize the specific attributes in the medical domain. In this paper, we propose DK-HRS, a domain knowledge-powered hybrid regularization strategy for semi-supervised breast cancer diagnosis. More specifically, DK-HRS generates four types of perturbed samples, namely, domain knowledge-augmented samples, weakly and strongly-augmented samples, and virtual adversarial samples. These perturbed samples represent transformations of medical images from different aspects. DK-HRS utilizes the FixMatch, a popular consistency regularization method as the backbone, and integrates two additional modules: a virtual adversarial training module, and a domain knowledge-based contrastive learning module. Experimental results on four breast ultrasound datasets demonstrate that, by incorporating medical domain knowledge, DK-HRS achieves superior performance and outperforms some state-of-the-art semi-supervised methods by a large margin.

Supervised Contrastive Learning for RFF Identification With Limited Samples

Radio frequency fingerprint (RFF), which comes from the imperfect hardware, is a potential feature to ensure the security of communication. With the development of deep learning (DL), DL-based RFF identification methods have made excellent and promising achievements. However, on one hand, existing DL-based methods require a large amount of samples for model training. On the other hand, the RFF identification method is generally less effective with limited amount of samples, while the auxiliary dataset and the target dataset often needs to have similar data distribution. To address the data-hungry problems in the absence of auxiliary datasets, in this paper, we propose a supervised contrastive learning (SCL)-based RFF identification method using data augmentation and virtual adversarial training (VAT), which is called “SCACNN”. First, we analyze the causes of RFF, and model the RFF identification problem with augmented dataset. A non-auxiliary data augmentation method is proposed to acquire an extended dataset, which consists of rotation, flipping, adding Gaussian noise, and shifting. Second, a novel similarity radio frequency fingerprinting encoder (SimRFE) is used to map the RFF signal to the feature coding space, which is based on the convolution, long-short-term-memory, and a fully connected deep neural network (CLDNN). Finally, several secondary classifiers are employed to identify the RFF feature coding. The simulation results show that the proposed SCACNN has greater identification ratio than the other classical RFF identification methods. Moreover, the identification ratio of the proposed SCACNN achieves an accuracy of 92.68% with only 5% samples.

Leveraging Unlabelled Data in Multiple-Instance Learning Problems for Improved Detection of Parkinsonian Tremor in Free-Living Conditions.

Data-driven approaches for remote detection of Parkinson's Disease and its motor symptoms have proliferated in recent years, owing to the potential clinical benefits of early diagnosis. The holy grail of such approaches is the free-living scenario, in which data are collected continuously and unobtrusively during every day life. However, obtaining fine-grained ground-truth and remaining unobtrusive is a contradiction and therefore, the problem is usually addressed via multiple-instance learning. Yet for large scale studies, obtaining even the necessary coarse ground-truth is not trivial, as a complete neurological evaluation is required. In contrast, large scale collection of data without any ground-truth is much easier. Nevertheless, utilizing unlabelled data in a multiple-instance setting is not straightforward, as the topic has received very little research attention. Here we try to fill this gap by introducing a new method for combining semi-supervised with multiple-instance learning. Our approach builds on the Virtual Adversarial Training principle, a state-of-the-art approach for regular semi-supervised learning, which we adapt and modify appropriately for the multiple-instance setting. We first establish the validity of the proposed approach through proof-of-concept experiments on synthetic problems generated from two well-known benchmark datasets. We then move on to the actual task of detecting PD tremor from hand acceleration signals collected in-the-wild, but in the presence of additional completely unlabelled data. We show that by leveraging the unlabelled data of 454 subjects we can achieve large performance gains (up to 9% increase in F1-score) in per-subject tremor detection for a cohort of 45 subjects with known tremor ground-truth. In doing so, we confirm the validity of our approach on a real-world problem where the need for semi-supervised and multiple-instance learning arises naturally.

Semi-Supervised Specific Emitter Identification Method Using Metric-Adversarial Training

Specific emitter identification (SEI) plays an increasingly crucial and potential role in both military and civilian scenarios. It refers to a process to discriminate individual emitters from each other by analyzing extracted characteristics from given radio signals. Deep learning (DL) and deep neural networks (DNNs) can learn the hidden features of data and build the classifier automatically for decision making, which have been widely used in the SEI research. Considering the insufficiently labeled training samples and large unlabeled training samples, semi-supervised learning-based SEI (SS-SEI) methods have been proposed. However, there are few SS-SEI methods focusing on extracting the discriminative and generalized semantic features of radio signals. In this paper, we propose an SS-SEI method using metric-adversarial training (MAT). Specifically, pseudo labels are innovatively introduced into metric learning to enable semi-supervised metric learning (SSML), and an objective function alternatively regularized by SSML and virtual adversarial training (VAT) is designed to extract discriminative and generalized semantic features of radio signals. The proposed MAT-based SS-SEI method is evaluated on an open-source large-scale real-world automatic-dependent surveillance-broadcast (ADS-B) dataset and WiFi dataset and is compared with state-of-the-art methods. The simulation results show that the proposed method achieves better identification performance than existing state-of-the-art methods. Specifically, when the ratio of the number of labeled training samples to the number of all training samples is 10%, the identification accuracy is 84.80% under the ADS-B dataset and 80.70% under the WiFi dataset. Our code can be downloaded from https://github.com/lovelymimola/MAT-based-SS-SEI.

Transformer-Based Network with Optimization for Cross-Subject Motor Imagery Identification.

Exploring the effective signal features of electroencephalogram (EEG) signals is an important issue in the research of brain-computer interface (BCI), and the results can reveal the motor intentions that trigger electrical changes in the brain, which has broad research prospects for feature extraction from EEG data. In contrast to previous EEG decoding methods that are based solely on a convolutional neural network, the traditional convolutional classification algorithm is optimized by combining a transformer mechanism with a constructed end-to-end EEG signal decoding algorithm based on swarm intelligence theory and virtual adversarial training. The use of a self-attention mechanism is studied to expand the receptive field of EEG signals to global dependence and train the neural network by optimizing the global parameters in the model. The proposed model is evaluated on a real-world public dataset and achieves the highest average accuracy of 63.56% in cross-subject experiments, which is significantly higher than that found for recently published algorithms. Additionally, good performance is achieved in decoding motor intentions. The experimental results show that the proposed classification framework promotes the global connection and optimization of EEG signals, which can be further applied to other BCI tasks.

Adversarial dense graph convolutional networks for single-cell classification.

In single-cell transcriptomics applications, effective identification of cell types in multicellular organisms and in-depth study of the relationships between genes has become one of the main goals of bioinformatics research. However, data heterogeneity and random noise pose significant difficulties for scRNA-seq data analysis. We have proposed an adversarial dense graph convolutional network architecture for single-cell classification. Specifically, to enhance the representation of higher-order features and the organic combination between features, dense connectivity mechanism and attention-based feature aggregation are introduced for feature learning in convolutional neural networks. To preserve the features of the original data, we use a feature reconstruction module to assist the goal of single-cell classification. In addition, HNNVAT uses virtual adversarial training to improve the generalization and robustness. Experimental results show that our model outperforms the existing classical methods in terms of classification accuracy on benchmark datasets. The source code of HNNVAT is available at https://github.com/DisscLab/HNNVAT. Supplementary data are available at Bioinformatics online.

Unsupervised Domain-Adaptation Based Nonlinear Equalizer With Improved Dynamic Impairment Tracking

We propose an unsupervised domain adaptation (UDA) based adaptive equalizer to compensate nonlinearity and dynamic timing jitter in intensity-modulation and direct-detection (IM/DD) systems. The proposed method employs generative adversarial training to update the equalizer on a block-by-block basis without the need for decision feedback. 56-Gbit/s PAM-4 experiments over 30-km fiber transmission under residual timing jitter, either virtually added using the mathematical model or resulted from Gardner-algorithm based digital phase-locked loop, show that the proposed UDA method has a faster convergence speed and better tracking performance than decision-directed neural network (NN) and virtual adversarial training (VAT) based NN, while exhibiting a lower complexity than VAT.

Semisupervised Fault Diagnosis of Gearbox Using Weighted Graph-Based Label Propagation and Virtual Adversarial Training

Fault diagnosis of gearbox is essential for the safe operation of the modern industrial system. Recently, supervised deep learning (DL)-based approaches have rapidly developed because they can achieve an automated fault-sensitive feature extraction process. However, most DL-based methods work on the assumption that labeled data are sufficient, and the field data recorded with label information are difficult to obtain in practical industrial applications. Moreover, there is a huge amount of unlabeled data that are collected in industrial practice and they can be utilized by semisupervised learning-based methods to enhance the identification ability of the model. Therefore, a novel semisupervised method for gearboxes using weighted label propagation and virtual adversarial training (WLP-VAT) is proposed in this article. In this study, the transductive label propagation method is introduced to infer pseudo labels for the unlabeled samples, which is based on the assumption that similar samples are likely to have the same labels. Meanwhile, a sample weight reflecting the uncertainty of each pseudo label is proposed to reduce the negative effect caused by noisy labels. Moreover, VAT is adopted to further enhance the identification capability of the model under limited labeled data. The effectiveness of the proposed semisupervised approach is verified by an experimental gearbox dataset and a wind turbine gearbox dataset. The results of the above two experiments indicate that our proposed semisupervised can leverage unlabeled data to solve the labeled data shortage.

Improving Non-Negative Positive-Unlabeled Learning for News Headline Classification

With the development of Internet technology, network platforms have gradually become a tool for people to obtain hot news. How to filter out the current hot news from a large number of news collections and push them to users has important application value. In supervised learning scenarios, each piece of news needs to be labeled manually, which takes a lot of time and effort. From the perspective of semi-supervised learning, on the basis of the non-negative Positive-Unlabeled (nnPU) learning, this paper proposes a novel algorithm, called ’Enhanced nnPU with Focal Loss’ (FLPU), for news headline classification, which uses the Focal Loss to replace the way the classical nnPU calculates the empirical risk of positive and negative samples. Then, by introducing the Virtual Adversarial Training (VAT) of the Adversarial training for large neural LangUage Models (ALUM) into FLPU, another (and better) algorithm, called ’FLPU+ALUM’, is proposed for the same purpose, aiming to label only a small number of positive samples. The superiority of both algorithms to state-of-the-art PU algorithms considered is demonstrated by means of experiments, conducted on two datasets for performance comparison. Moreover, through another set of experiments, it is shown that, if enriched by the proposed algorithms, the RoBERTa-wwm-ext model can achieve better classification performance than the state-of-the-art binary classification models included in the comparison. In addition, a ’Ratio Batch’ method is elaborated and proposed as more stable for use in scenarios involving only a small number of labeled positive samples, which is also experimentally demonstrated.

Batch virtual adversarial training for graph convolutional networks

We present batch virtual adversarial training (BVAT), a novel regularization method for graph convolutional networks (GCNs). BVAT addresses the issue that GCNs do not ensure the smoothness of the model’s output distribution against local perturbations around the input node features. We propose two algorithms, sampling-based BVAT and optimization-based BVAT, which promote the output smoothness of GCN classifiers based on the generated virtual adversarial perturbations for either a subset of independent nodes or all nodes via an elaborate optimization process. Extensive experiments on three citation network datasets Cora, Citeseer and Pubmed and a knowledge graph dataset Nell validate the efficacy of the proposed method in semi-supervised node classification tasks.

Making attention mechanisms more robust and interpretable with virtual adversarial training

Although attention mechanisms have become fundamental components of deep learning models, they are vulnerable to perturbations, which may degrade the prediction performance and model interpretability. Adversarial training (AT) for attention mechanisms has successfully reduced such drawbacks by considering adversarial perturbations. However, this technique requires label information, and thus, its use is limited to supervised settings. In this study, we explore the concept of incorporating virtual AT (VAT) into the attention mechanisms, by which adversarial perturbations can be computed even from unlabeled data. To realize this approach, we propose two general training techniques, namely VAT for attention mechanisms (Attention VAT) and "interpretable" VAT for attention mechanisms (Attention iVAT), which extend AT for attention mechanisms to a semi-supervised setting. In particular, Attention iVAT focuses on the differences in attention; thus, it can efficiently learn clearer attention and improve model interpretability, even with unlabeled data. Empirical experiments based on six public datasets revealed that our techniques provide better prediction performance than conventional AT-based as well as VAT-based techniques, and stronger agreement with evidence that is provided by humans in detecting important words in sentences. Moreover, our proposal offers these advantages without needing to add the careful selection of unlabeled data. That is, even if the model using our VAT-based technique is trained on unlabeled data from a source other than the target task, both the prediction performance and model interpretability can be improved.

A regularization-driven Mean Teacher model based on semi-supervised learning for medical image segmentation

Objective. A semi-supervised learning method is an essential tool for applying medical image segmentation. However, the existing semi-supervised learning methods rely heavily on the limited labeled data. The generalization performance of image segmentation is improved to reduce the need for the number of labeled samples and the difficulty of parameter tuning by extending the consistency regularization. Approach. We propose a new regularization-driven Mean Teacher model based on semi-supervised learning for medical image segmentation in this work. We introduce a regularization-driven strategy with virtual adversarial training to improve segmentation performance and the robustness of the Mean Teacher model. We optimize the unsupervised loss function and the regularization term with an entropy minimum to smooth the decision boundary. Main results. We extensively evaluate the proposed method on the International Skin Imaging Cooperation 2017(ISIC2017) and COVID-19 CT segmentation datasets. Our proposed approach gains more accurate results on challenging 2D images for semi-supervised medical image segmentation. Compared with the state-of-the-art methods, the proposed approach has significantly improved and is superior to other semi-supervised segmentation methods. Significance. The proposed approach can be extended to other medical segmentation tasks and can reduce the burden of physicians to some extent.

Handling Imbalanced Data: Uncertainty-Guided Virtual Adversarial Training With Batch Nuclear-Norm Optimization for Semi-Supervised Medical Image Classification.

In manyclinical settings, a lot of medical image datasets suffer from imbalance problems, which makes predictions of trained models to be biased toward majority classes. Semi-supervised Learning (SSL) algorithms trained with such imbalanced datasets become more problematic since pseudo-supervision of unlabeled data are generated from the model's biased predictions. To address these issues, in this work, we propose a novel semi-supervised deep learning method, i.e., uncertainty-guided virtual adversarial training (VAT) with batch nuclear-norm (BNN) optimization, for large-scale medical image classification. To effectively exploit useful information from both labeled and unlabeled data, we leverage VAT and BNN optimization to harness the underlying knowledge, which helps to improve discriminability, diversity and generalization of the trained models. More concretely, our network is trained by minimizing a combination of four types of losses, including a supervised cross-entropy loss, a BNN loss defined on the output matrix of labeled data batch (lBNN loss), a negative BNN loss defined on the output matrix of unlabeled data batch (uBNN loss), and a VAT loss on both labeled and unlabeled data. We additionally propose to use uncertainty estimation to filter out unlabeled samples near the decision boundary when computing the VAT loss. We conduct comprehensive experiments to evaluate the performance of our method on two publicly available datasets and one in-house collected dataset. The experimental results demonstrated that our method achieved better results than state-of-the-art SSL methods.