Adversarial Samples Research Articles

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness

To obtain a near-optimal policy with fewer interactions in Reinforcement Learning (RL), a promising approach involves the combination of offline RL, which enhances sample efficiency by leveraging offline datasets, and online RL, which explores informative transitions by interacting with the environment. Offline-to-Online RL provides a paradigm for improving an offline-trained agent within limited online interactions. However, due to the significant distribution shift between online experiences and offline data, most offline RL algorithms suffer from performance drops and fail to achieve stable policy improvement in offline-to-online adaptation. To address this problem, we propose the Robust Offlineto-Online (RO2O) algorithm, designed to enhance offline policies through uncertainty and smoothness, and to mitigate the performance drop in online adaptation. Specifically, RO2O incorporates Q-ensemble for uncertainty penalty and adversarial samples for policy and value smoothness, which enable RO2O to maintain a consistent learning procedure in online adaptation without requiring special changes to the learning objective. Theoretical analyses in linear MDPs demonstrate that the uncertainty and smoothness lead to tighter optimality bound in offline-to-online against distribution shift. Experimental results illustrate the superiority of RO2O in facilitating stable offline-to-online learning and achieving significant improvement with limited online interactions.

Research on adversarial attack and defense of large language models

Abstract. Large language models (LLMs) have made excellent progress in text and image understanding and generation. However, with the wide range of applications of these models in various industries, the issue of their security, especially the defense against adversarial attacks, has become a focus of research. This study focuses on exploring the adversarial attacks faced by LLMs and their defense strategies, especially the design and optimization of defense mechanisms. Through literature review and case studies, this paper analyzes in detail the white-box and black-box attack patterns against LLMs, including model inversion, backdoor attacks, and token-based strategies. In response to these attacks, this paper proposes a series of defense strategies, including preventive measures such as data augmentation, adversarial training and model regularization, as well as real-time attack detection and response strategies such as anomaly detection and adversarial sample detection techniques. The core of this research is to improve the robustness and trustworthiness of LLMs, providing the necessary guarantees for their integration and sustainability in multiple industrial applications. In addition, this paper proposes future research directions, highlighting the importance of developing advanced defense systems, promoting interdisciplinary research and exploring new applications for LLMs. This research provides valuable insights into understanding and improving the security defense mechanisms of LLMs, which is essential for maintaining the security and user trust of these models.

Few‐Shot Contrastive Learning‐Based Multi‐Round Dialogue Intent Classification Method

ABSTRACTTraditional text classification models face challenges in handling long texts and understanding topic transitions in dialogue scenarios, leading to suboptimal performance in automatic speech recognition (ASR)‐based multi‐round dialogue intent classification. In this article, we propose a few‐shot contrastive learning‐based multi‐round dialogue intent classification method. First, the ASR texts are partitioned, and role‐based features are extracted using a Transformer encoder. Second, refined sample pairs are forward‐propagated, adversarial samples are generated by perturbing word embedding matrices and contrastive loss is applied to positive sample pairs. Then, positive sample pairs are input into a multi‐round reasoning module to learn semantic clues from the entire scenario through multiple dialogues, obtain reasoning features, input them into a classifier to obtain classification results, and calculate multi‐task loss. Finally, a prototype update module (PUM) is introduced to rectify the biased prototypes by using gated recurrent unit (GRU) to update the prototypes stored in the memory bank and few‐shot learning (FSL) task. Experimental evaluations demonstrate that the proposed method outperforms state‐of‐the‐art methods on two public datasets (DailyDialog and CM) and a private real‐world dataset.

Classification and Identification of Frequency-Hopping Signals Based on Jacobi Salient Map for Adversarial Sample Attack Approach.

Frequency-hopping (FH) communication adversarial research is a key area in modern electronic countermeasures. To address the challenge posed by interfering parties that use deep neural networks (DNNs) to classify and identify multiple intercepted FH signals-enabling targeted interference and degrading communication performance-this paper presents a batch feature point targetless adversarial sample generation method based on the Jacobi saliency map (BPNT-JSMA). This method builds on the traditional JSMA to generate feature saliency maps, selects the top 8% of salient feature points in batches for perturbation, and increases the perturbation limit to restrict the extreme values of single-point perturbations. Experimental results in a white-box environment show that, compared with the traditional JSMA method, BPNT-JSMA not only maintains a high attack success rate but also enhances attack efficiency and improves the stealthiness of the adversarial samples.

Adversarial Sample Generation and Training using Neural Network

Adversarial Sample Generation and Training using Neural Network

Malware Detection and Classification using Generative Adversarial Network

The Generative Adversarial Networks (GANs) are playing a crucial role in deep-learning-based malware classification to overcome the dataset imbalance and unseen malware. The Generative AI is preferably used in many applications, such as improving image resolution and generating audio, video, and text. The cybercriminals are also using the Generative AI for generating the malware and deepfake videos to harm the targeted person or device. By generating the synthetic data, it makes the deep learning model more robust to detect such types of unseen and adversarial attacks. This work utilizes GANs for generating adversarial malware samples to train a classification and detection model, improving the model’s ability to identify sophisticated malware variants. The performance of the proposed Conditional Generative Adversarial Network (CGAN) model is evaluated on a multiclass malware grayscale image dataset and a binary class malware RGB image dataset. The performance of proposed model is compared with current state-of-the-art. Results indicate a significant improvement in classification accuracy and a reduction in training time and false positives, showcasing GAN’s potential in the dynamic cybersecurity landscape.

Efficient Ensemble Adversarial Attack for a Deep Neural Network (DNN)-Based Unmanned Aerial Vehicle (UAV) Vision System

In recent years, unmanned aerial vehicles (UAVs) vision systems based on deep neural networks (DNNs) have made remarkable advancements, demonstrating impressive performance. However, due to the inherent characteristics of DNNs, these systems have become increasingly vulnerable to adversarial attacks. Traditional black-box attack methods typically require a large number of queries to generate adversarial samples successfully. In this paper, we propose a novel adversarial attack technique designed to achieve efficient black-box attacks with a minimal number of queries. We define a perturbation generator that first decomposes the image into four frequency bands using wavelet decomposition and then searches for adversarial perturbations across these bands by minimizing a weighted loss function on a set of fixed surrogate models. For the target victim model, the perturbation images generated by the perturbation generator are used to query and update the weights in the loss function, as well as the weights for different frequency bands. Experimental results show that, compared to state-of-the-art methods on various image classifiers trained on ImageNet (such as VGG-19, DenseNet-121, and ResNext-50), our method achieves a success rate over 98% for targeted attacks and nearly a 100% success rate for non-targeted attacks with only 1–2 queries per image.

Data Quality-Aware Client Selection in Heterogeneous Federated Learning

Federated Learning (FL) enables decentralized data utilization while maintaining edge user privacy, but it faces challenges due to statistical heterogeneity. Existing approaches address client drift and data heterogeneity issues. However, real-world settings often involve low-quality data with noisy features, such as covariate drift or adversarial samples, which are usually ignored. Noisy samples significantly impact the global model’s accuracy and convergence rate. Assessing data quality and selectively aggregating updates from high-quality clients is crucial, but dynamically perceiving data quality without additional computations or data exchanges is challenging. In this paper, we introduce the FedDQA (Federated learning via Data Quality-Aware) (FedDQA) framework. We discover increased data noise leads to slower loss reduction during local model training. We propose a loss sharpness-based Data-Quality-Awareness (DQA) metric to differentiate between high-quality and low-quality data. Based on the DQA, we design a client selection algorithm that strategically selects participant clients to reduce the negative impact of noisy clients. Experiment results indicate that FedDQA significantly outperforms the baselines. Notably, it achieves up to a 4% increase in global model accuracy and demonstrates faster convergence rates.

A3GT: An Adaptive Asynchronous Generalized Adversarial Training Method

Adversarial attack methods can significantly improve the classification accuracy of deep learning models, but research has found that although most deep learning models with defense methods still show good classification accuracy in the face of various adversarial attack attacks, the improved robust models have a significantly lower classification accuracy when facing clean samples compared to themselves without using defense methods. This means that while improving the model’s adversarial robustness, it is necessary to find a defense method to balance the accuracy of clean samples (clean accuracy) and the accuracy of adversarial samples (robust accuracy). Therefore, in this work, we propose an Adaptive Asynchronous Generalized Adversarial Training (A3GT) method, which is an improvement over the existing Generalist method. It employs an adaptive update strategy without the need for extensive experiments to determine the optimal starting iteration for global updates. The experimental results show that compared with other advanced methods, A3GT can achieve a balance between clean sample classification accuracy and robust classification accuracy while improving the model’s adversarial robustness.

Enhancing Adversarial Robustness through Stable Adversarial Training

Deep neural network models are vulnerable to attacks from adversarial methods, such as gradient attacks. Evening small perturbations can cause significant differences in their predictions. Adversarial training (AT) aims to improve the model’s adversarial robustness against gradient attacks by generating adversarial samples and optimizing the adversarial training objective function of the model. Existing methods mainly focus on improving robust accuracy, balancing natural and robust accuracy and suppressing robust overfitting. They rarely consider the AT problem from the characteristics of deep neural networks themselves, such as the stability properties under certain conditions. From a mathematical perspective, deep neural networks with stable training processes may have a better ability to suppress overfitting, as their training process is smoother and avoids sudden drops in performance. We provide a proof of the existence of Ulam stability for deep neural networks. Ulam stability not only determines the existence of the solution for an operator inequality, but it also provides an error bound between the exact and approximate solutions. The feature subspace of a deep neural network with Ulam stability can be accurately characterized and constrained by a function with special properties and a controlled error boundary constant. This restricted feature subspace leads to a more stable training process. Based on these properties, we propose an adversarial training framework called Ulam stability adversarial training (US-AT). This framework can incorporate different Ulam stability conditions and benchmark AT models, optimize the construction of the optimal feature subspace, and consistently improve the model’s robustness and training stability. US-AT is simple and easy to use, and it can be easily integrated with existing multi-class AT models, such as GradAlign and TRADES. Experimental results show that US-AT methods can consistently improve the robust accuracy and training stability of benchmark models.

Vibrational Spectroscopy Can Be Vulnerable to Adversarial Attacks.

Nondestructive detection methods based on vibrational spectroscopy have been widely used in many critical applications in a variety of fields such as the chemical industry, pharmacy, national defense, security, and so on. As these methods/applications rely on machine learning models for data analysis, studying the threats associated with adversarial examples in vibrational spectroscopy and defenses against them is of great importance. In this paper, we propose a novel adversarial method to attack vibrational spectroscopy, named SynPat, where synthetic peaks produced by a physical model are placed at key locations to form adversarial perturbations. Our new attack generates perturbations that successfully deceive machine learning models for Raman and infrared spectrum analysis while they blend much better into the spectra and hence are unnoticeable to human operators, unlike the existing state-of-the-art adversarial attacking methods, e.g., images and audio. We verified the superiority of the proposed SynPat by an imperceptibility test conducted by human experts and of defense experiments by an AI detector. To the best of our knowledge, this is a first thorough study on the robustness of vibrational spectroscopic techniques against adversarial samples and defense mechanisms. Our extensive experiments show that machine learning models for vibrational spectroscopy, including conventional and deep models for Raman or infrared classification and regression, are all vulnerable to adversarial perturbations and thus may pose serious security threats to our society.

Adversarial sample attack method based on loss smoothing

Deep neural networks (DNNs) are vulnerable to adversarial examples.Although the existing momentum-based adversarial example generation method can achieve a close 100%white-box attack success rate, it is still not ideal when attacking other models, and the black- box attack success rate is low. To address this, an adversarial example attack method based on loss smoothing is proposed to improve the transferability of adversarial examples. In the iterative process of calculating the gradient at each step, the current gradient is not used directly, but the local average gradient is used to accumulate momentum, so as to suppress the local oscillation phenomenon on the loss function surface, thereby stabilizing the update direction and escaping the local extreme point. A large number of experimental results on the ImageNet dataset show that compared with the existing momentum-based method, the average black-box attack success rate of the proposed method in single model attack experiments is improved by 38.07%and 27.77%, and the average black-box attack success rate in integrated model attack experiments is improved by and.Rising32.50%and28.63%

Adaptive feature alignment for adversarial training

Recent studies reveal that Convolutional Neural Networks (CNNs) are typically vulnerable to adversarial attacks. Many adversarial defense methods have been proposed to improve the robustness against adversarial samples. Moreover, these methods can only defend adversarial samples of a specific strength, reducing their flexibility against attacks of varying strengths. Moreover, these methods often enhance adversarial robustness at the expense of accuracy on clean samples. In this paper, we first observed that features of adversarial images change monotonically and smoothly w.r.t the rising of attacking strength. This intriguing observation suggests that features of adversarial images with various attacking strengths can be approximated by interpolating between the features of adversarial images with the strongest and weakest attacking strengths. Due to the monotonicity property, the interpolation weight can be easily learned by a neural network. Based on the observation, we proposed the adaptive feature alignment (AFA) that automatically align features to defense adversarial attacks of various attacking strengths. During training, our method learns the statistical information of adversarial samples with various attacking strengths using a dual batchnorm architecture. In this architecture, each batchnorm process handles samples of a specific attacking strength. During inference, our method automatically adjusts to varying attacking strengths by linearly interpolating the dual-BN features. Unlike previous methods that need to either retrain the model or manually tune hyper-parameters for a new attacking strength, our method can deal with arbitrary attacking strengths with a single model without introducing any hyper-parameter. Additionally, our method improves the model robustness against adversarial samples without incurring much loss of accuracy on clean images. Experiments on CIFAR-10, SVHN and tiny-ImageNet datasets demonstrate that our method outperforms the state-of-the-art under various attacking strengths and even improve accuracy on clean samples. Code will be made open available upon acceptance.

GBKPA and AuxShield: Addressing adversarial robustness and transferability in android malware detection

Android stands as the predominant operating system within the mobile ecosystem. Users can download applications from official sources like Google Play Store and other third-party platforms. However, malicious actors can attempt to compromise user device integrity through malicious applications. Traditionally, signatures, rules, and other methods have been employed to detect malware attacks and protect device integrity. However, the growing number and complexity of malicious applications have prompted the exploration of newer techniques like machine learning (ML) and deep learning (DL). Many recent studies have demonstrated promising results in detecting malicious applications using ML and DL solutions. However, research in other fields, such as computer vision, has shown that ML and DL solutions are vulnerable to targeted adversarial attacks. Malicious actors can develop malicious adversarial applications that can bypass ML and DL based anti-viruses. The study of adversarial techniques related to malware detection has now captured the security community’s attention. In this work, we utilise android permissions and intents to construct 28 distinct malware detection models using 14 classification algorithms. Later, we introduce a novel targeted false-negative evasion attack, Gradient Based K Perturbation Attack (GBKPA), designed for grey-box knowledge scenarios to assess the robustness of these models. The GBKPA attempts to craft malicious adversarial samples by making minimal perturbations without violating the syntactic and functional structure of the application. GBKPA achieved an average fooling rate (FR) of 77 % with only five perturbations across the 28 detection models. Additionally, we identified the most vulnerable android permissions and intents that malicious actors can exploit for evasion attacks. Furthermore, we analyse the transferability of adversarial samples across different classes of models and provide explanations for the same. Finally, we proposed AuxShield defence mechanism to develop robust detection models. AuxShield reduced the average FR to 3.25 % against 28 detection models. Our findings underscore the need to understand the causation of adversarial samples, their transferability, and robust defence strategies before deploying ML and DL solutions in the real world.

Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks.

Despite providing high-performance solutions for computer vision tasks, the deep neural network (DNN) model has been proved to be extremely vulnerable to adversarial attacks. Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. Besides, commonly used adaptive learning and fine-tuning technique is unsuitable for adversarial defense since it is essentially a zero-shot problem when deployed. Thus, to tackle this challenge, we propose an attack-agnostic defense method named Meta Invariance Defense (MID). Specifically, various combinations of adversarial attacks are randomly sampled from a manually constructed Attacker Pool to constitute different defense tasks against unknown attacks, in which a student encoder is supervised by multi-consistency distillation to learn the attack-invariant features via a meta principle. The proposed MID has two merits: 1) Full distillation from pixel-, feature- and prediction-level between benign and adversarial samples facilitates the discovery of attack-invariance. 2) The model simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration. Theoretical and empirical studies on numerous benchmarks such as ImageNet verify the generalizable robustness and superiority of MID under various attacks.

Gradient correlation based detection of adversarial attacks on vehicular networks

The controller area network (CAN) bus, which controls real-time communication and data transmission among vehicle electronic control units, lacks security mechanisms and is highly vulnerable to attacks. Existing in-vehicle network intrusion detection systems (IDSs) typically rely on deep learning models for detection, which are susceptible to interference from adversarial attacks owing to the vulnerability of the models themselves, thereby compromising the detection performance. In this study, we propose an adversarial attack detection method based on gradient correlation that achieves a high accuracy rate using a linear approach to detect adversarial samples. The experimental results show that the proposed model does not require retraining of the original detection model and demonstrates better detection performance for multiple adversarial attacks.

Robust Remote Sensing Scene Interpretation Based on Unsupervised Domain Adaptation

Deep learning models excel in interpreting the exponentially growing amounts of remote sensing data; however, they are susceptible to deception and spoofing by adversarial samples, posing catastrophic threats. The existing methods to combat adversarial samples have limited performance in robustness and efficiency, particularly in complex remote sensing scenarios. To tackle these challenges, an unsupervised domain adaptation algorithm is proposed for the accurate identification of clean images and adversarial samples by exploring a robust generative adversarial classification network that can harmonize the features between clean images and adversarial samples to minimize distribution discrepancies. Furthermore, linear polynomial loss as a replacement for cross-entropy loss is integrated to guide robust representation learning. Additionally, we leverage the fast gradient sign method (FGSM) and projected gradient descent (PGD) algorithms to generate adversarial samples with varying perturbation amplitudes to assess model robustness. A series of experiments was performed on the RSSCN7 dataset and SIRI-WHU dataset. Our experimental results illustrate that the proposed algorithm performs exceptionally well in classifying clean images while demonstrating robustness against adversarial perturbations.

Decoupling visual and identity features for adversarial palm-vein image attack

Palm-vein has been widely used for biometric recognition due to its resistance to theft and forgery. However, with the emergence of adversarial attacks, most existing palm-vein recognition methods are vulnerable to adversarial image attacks, and to the best of our knowledge, there is still no study specifically focusing on palm-vein image attacks. In this paper, we propose an adversarial palm-vein image attack network that generates highly similar adversarial palm-vein images to the original samples, but with altered palm-identities. Unlike most existing generator-oriented methods that directly learn image features via concatenated convolutional layers, our proposed network first maps palm-vein images into multi-scale high-dimensional shallow representation, and then develops attention-based dual-path feature learning modules to extensively exploit diverse palm-vein-specific features. After that, we design visual-consistency and identity-aware loss functions to specially decouple the visual and identity features to reconstruct the adversarial palm-vein images. By doing this, the visual characteristics of palm-vein images can be largely preserved while the identity information is removed in the adversarial palm-vein images, such that high-aggressive adversarial palm-vein samples can be obtained. Extensive white-box and black-box attack experiments conducted on three widely used databases clearly show the effectiveness of the proposed network.

Label-Only Membership Inference Attack Based on Model Explanation

It is well known that machine learning models (e.g., image recognition) can unintentionally leak information about the training set. Conventional membership inference relies on posterior vectors, and this task becomes extremely difficult when the posterior is masked. However, current label-only membership inference attacks require a large number of queries during the generation of adversarial samples, and thus incorrect inference generates a large number of invalid queries. Therefore, we introduce a label-only membership inference attack based on model explanations. It can transform a label-only attack into a traditional membership inference attack by observing neighborhood consistency and perform fine-grained membership inference for vulnerable samples. We use feature attribution to simplify the high-dimensional neighborhood sampling process, quickly identify decision boundaries and recover a posteriori vectors. It also compares different privacy risks faced by different samples through finding vulnerable samples. The method is validated on CIFAR-10, CIFAR-100 and MNIST datasets. The results show that membership attributes can be identified even using a simple sampling method. Furthermore, vulnerable samples expose the model to greater privacy risks.

Adversarial Sample Synthesis for Visual Question Answering

Language prior is a major block to improving the generalization of visual question answering (VQA) models. Recent work has revealed that synthesizing extra training samples to balance training sets is a promising way to alleviate language priors. However, most existing methods synthesize extra samples in a manner independent of training processes, which neglect the fact that the language priors memorized by VQA models are changing during training, resulting in insufficient synthesized samples. In this paper, we propose an adversarial sample synthesis method, which synthesizes different adversarial samples by adversarial masking at different training epochs to cope with the changing memorized language priors. The basic idea behind our method is to use adversarial masking to synthesize adversarial samples that will cause the model to make wrong answers. To this end, we design a generative module to carry out adversarial masking by attacking the VQA model, and introduce a bias-oriented objective to supervise the training of the generative module. We couple the sample synthesis with the training process of the VQA model, which ensures that the synthesized samples at different training epochs are beneficial to the VQA model. We incorporated the proposed method into three VQA models including UpDn, LMH and LXMERT, and conducted experiments on three datasets including VQA-CP v1, VQA-CP v2 and VQA v2. Experimental results demonstrate that a large improvement of our method, such as 16.22% gains on LXMERT in the overall accuracy of VQA-CP v2.