Black-box Attack Research Articles

Random transformations to improve mitigation of query-based black-box attacks

This paper proposes methods to upstage the best-known defences against query-based black-box attacks. These benchmark defences incorporate gaussian noise into input data during inference to achieve state-of-the-art performance in protecting image classification models against the most advanced query-based black-box attacks. Even so there is a need to improve upon them; for example, the widely benchmarked Random noise defense (RND) method has demonstrated limited robustness – achieving only 53.5% and 18.1% with a ResNet-50 model on the CIFAR-10 and ImageNet datasets, respectively – against the square attack, which is commonly regarded as the state-of-the-art black-box attack. Therefore, in this work, we propose two alternatives to gaussian noise addition at inference time: random crop-resize and random rotation of the input images. Although these transformations are generally used for data augmentation while training to improve model invariance and generalisation, their protective potential against query-based black-box attacks at inference time is unexplored. Therefore, for the first time, we report that for such well-trained models either of the two transformations can also blunt powerful query-based black-box attacks when used at inference time on three popular datasets. The results show that the proposed randomised transformations outperform RND in terms of robust accuracy against a strong adversary that uses a high budget of 100,000 queries based on expectation over transformation (EOT) of 10, by 0.9% on the CIFAR-10 dataset, 9.4% on the ImageNet dataset and 1.6% on the Tiny ImageNet dataset. Crucially, in two even tougher attack settings, that is, high-confidence adversarial examples and EOT-50 adversary, these transformations are even more effective as the margin of improvement over the benchmarks increases further.

Semantic Web-Driven Targeted Adversarial Attack on Black Box Automatic Speech Recognition Systems

The susceptibility of Deep Neural Networks (DNNs) to adversarial attacks in Automatic Speech Recognition (ASR) systems has drawn significant attention. Most work focuses on white-box methods, but the assumption of full transparency of model architecture and parameters is unrealistic in real-world scenarios. Although several targeted black-box attack methods have been proposed in recent years, due to the complexity of ASR systems, they primarily rely on query-based approaches with limited search capabilities, leading to low success rates and noticeable noise. To address this, we propose DE-gradient, a new black-box approach using differential evolution (DE), a population-based search algorithm. Inspired by Semantic Web ideas, we introduce modulation noise to preserve semantic coherence while enhancing imperceptibility. In experiments on two public datasets, DE-gradient improved attack success rates by 19% and increased the signal-to-noise ratio (SNR) of silent parts from 27 dB to 54 dB, establishing a strong baseline for evaluating black-box adversarial attacks in ASR systems.

Adversarial Machine Learning In Network Security: A Systematic Review Of Threat Vectors And Defense Mechanisms

Adversarial Machine Learning (AML) has emerged as a critical area of research within network security, addressing the evolving challenge of adversaries exploiting machine learning (ML) models. This systematic review adopts the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology to comprehensively examine threat vectors and defense mechanisms in AML. The study identifies, categorizes, and evaluates existing research focused on adversarial attacks targeting ML algorithms in network security applications, including evasion, poisoning, and model extraction attacks. By rigorously following the PRISMA guidelines, a systematic search across multiple scholarly databases yielded a robust dataset of peer-reviewed articles that were screened, reviewed, and analyzed for inclusion. The review outlines key adversarial techniques employed against ML systems, such as gradient-based attack strategies and black-box attacks and explores the underlying vulnerabilities in network security architectures. Additionally, it highlights defense mechanisms, including adversarial training, input preprocessing, and robust model design, discussing their efficacy and limitations in mitigating adversarial threats. The study also identifies critical gaps in current research, such as the lack of standardized benchmarking for adversarial defenses and the need for scalable and real-time AML solutions.

Enhancing adversarial transferability with local transformation

Robust deep learning models have demonstrated significant applicability in real-world scenarios. The utilization of adversarial attacks plays a crucial role in assessing the robustness of these models. Among such attacks, transfer-based attacks, which leverage white-box models to generate adversarial examples, have garnered considerable attention. These transfer-based attacks have demonstrated remarkable efficiency, particularly under the black-box setting. Notably, existing transfer attacks often exploit input transformations to amplify their effectiveness. However, prevailing input transformation-based methods typically modify input images indiscriminately, overlooking regional disparities. To bolster the transferability of adversarial examples, we propose the Local Transformation Attack (LTA) based on forward class activation maps. Specifically, we first obtain future examples through accumulated momentum and compute forward class activation maps. Subsequently, we utilize these maps to identify crucial areas and apply pixel scaling for transformation. Finally, we update the adversarial examples by using the average gradient of the transformed image. Extensive experiments convincingly demonstrate the effectiveness of our proposed LTA. Compared to the current state-of-the-art attack approaches, LTA achieves an increase of 7.9% in black-box attack performance. Particularly, in the case of ensemble attacks, our method achieved an average attack success rate of 98.3%.

Research on adversarial attack and defense of large language models

Abstract. Large language models (LLMs) have made excellent progress in text and image understanding and generation. However, with the wide range of applications of these models in various industries, the issue of their security, especially the defense against adversarial attacks, has become a focus of research. This study focuses on exploring the adversarial attacks faced by LLMs and their defense strategies, especially the design and optimization of defense mechanisms. Through literature review and case studies, this paper analyzes in detail the white-box and black-box attack patterns against LLMs, including model inversion, backdoor attacks, and token-based strategies. In response to these attacks, this paper proposes a series of defense strategies, including preventive measures such as data augmentation, adversarial training and model regularization, as well as real-time attack detection and response strategies such as anomaly detection and adversarial sample detection techniques. The core of this research is to improve the robustness and trustworthiness of LLMs, providing the necessary guarantees for their integration and sustainability in multiple industrial applications. In addition, this paper proposes future research directions, highlighting the importance of developing advanced defense systems, promoting interdisciplinary research and exploring new applications for LLMs. This research provides valuable insights into understanding and improving the security defense mechanisms of LLMs, which is essential for maintaining the security and user trust of these models.

Negatively correlated ensemble against transfer adversarial attacks

Deep neural networks (DNNs) have become a popular tool in various fields, but their susceptibility to adversarial attacks poses a significant threat to the security of machine learning systems. Adversarial examples, which are intentionally crafted inputs to deceive a DNN, can easily transfer between different machine learning models, further exacerbating the problem. To address the transfer adversarial vulnerability issue, ensemble methods have been developed to promote diversity among members, thereby impeding the transfer of adversarial examples among them. However, we observed that the diversity among ensemble members tends to diminish rapidly as the magnitude of adversarial perturbations increases. All ensemble members tend to be deceived by the same adversarial example, resulting in a poor performance against slightly larger transfer adversarial perturbations. To overcome this challenge, we introduce NCRE (negative correlation robust ensemble) and NCRE+(negative correlation robust ensemble on adversarial vulnerabilities) in this work. Our approach leverages the explicit maximization of negative correlation among ensemble member outputs to enhance ensemble diversity and robustness against adversarial perturbations, particularly transfer black-box attacks. Extensive experimental studies and comparisons with state-of-the-art algorithms demonstrate the effectiveness of our approach.

An auxiliary correction model implemented by the Correction Property of intermediate layer against adversarial examples

In powerful adversarial attacks against deep neural networks (DNN), the generated adversarial example will mislead the DNN-implemented classifier by destroying the features of the last layer. To enhance the robustness of the classifier, in our paper, a Feature Analysis and Conditional Matching prediction distribution (FACM) model is proposed to utilize the features of intermediate layers to correct the misclassification. Specifically, we first prove that the intermediate layers of the classifier still retain effective features for the original category when the classifier is subjected to adversarial attacks, which is defined as the Correction Property in our paper. According to this, we propose the FACM model consisting of Feature Analysis (FA) correction module, Conditional Matching Prediction Distribution (CMPD) correction module and decision module. Specifically, the FA correction module is comprised of fully connected layers, which takes the features of the intermediate layers as the input to correct the misclassification of the classifier. The CMPD correction module is based on a conditional autoencoder, which not only uses the features of intermediate layers as the condition to accelerate convergence but also mitigates the negative effect of adversarial examples, trained with the Kullback–Leibler loss to match prediction distribution. Through the empirically verified Diversity Property among the individual correction modules, the decision module is proposed to integrate the proposed correction modules to enhance the DNN-implemented classifier’s robustness by reducing the dimensionality of adversarial subspace. That is, the input perturbed in certain directions (i.e., dimensions) that lead to misclassifications for the classifier can be correctly classified by the proposed correction modules. The extended experiments demonstrate our FACM model outperforms the existing methods against adversarial attacks, especially optimization-based white-box attacks and query-based black-box attacks.

Efficient Ensemble Adversarial Attack for a Deep Neural Network (DNN)-Based Unmanned Aerial Vehicle (UAV) Vision System

In recent years, unmanned aerial vehicles (UAVs) vision systems based on deep neural networks (DNNs) have made remarkable advancements, demonstrating impressive performance. However, due to the inherent characteristics of DNNs, these systems have become increasingly vulnerable to adversarial attacks. Traditional black-box attack methods typically require a large number of queries to generate adversarial samples successfully. In this paper, we propose a novel adversarial attack technique designed to achieve efficient black-box attacks with a minimal number of queries. We define a perturbation generator that first decomposes the image into four frequency bands using wavelet decomposition and then searches for adversarial perturbations across these bands by minimizing a weighted loss function on a set of fixed surrogate models. For the target victim model, the perturbation images generated by the perturbation generator are used to query and update the weights in the loss function, as well as the weights for different frequency bands. Experimental results show that, compared to state-of-the-art methods on various image classifiers trained on ImageNet (such as VGG-19, DenseNet-121, and ResNext-50), our method achieves a success rate over 98% for targeted attacks and nearly a 100% success rate for non-targeted attacks with only 1–2 queries per image.

Adversarial sample attack method based on loss smoothing

Deep neural networks (DNNs) are vulnerable to adversarial examples.Although the existing momentum-based adversarial example generation method can achieve a close 100%white-box attack success rate, it is still not ideal when attacking other models, and the black- box attack success rate is low. To address this, an adversarial example attack method based on loss smoothing is proposed to improve the transferability of adversarial examples. In the iterative process of calculating the gradient at each step, the current gradient is not used directly, but the local average gradient is used to accumulate momentum, so as to suppress the local oscillation phenomenon on the loss function surface, thereby stabilizing the update direction and escaping the local extreme point. A large number of experimental results on the ImageNet dataset show that compared with the existing momentum-based method, the average black-box attack success rate of the proposed method in single model attack experiments is improved by 38.07%and 27.77%, and the average black-box attack success rate in integrated model attack experiments is improved by and.Rising32.50%and28.63%

A hypothetical defenses-based training framework for generating transferable adversarial examples

Transfer-based attacks utilize the proxy model to craft adversarial examples against the target model and make significant advancements in the realm of black-box attacks. Recent research suggests that these attacks can be enhanced by incorporating adversarial defenses into the training process of adversarial examples. Specifically, adversarial defenses supervise the training process, forcing the attacker to overcome greater challenges and produce more robust adversarial examples with enhanced transferability. However, current methods mainly rely on limited input transformation defenses, which apply only linear affine changes. These defenses are insufficient for effectively removing harmful content from adversarial examples, resulting in restricted improvements in their transferability. To address this issue, we propose a novel training framework named Transfer-based Attacks through Hypothesis Defense (TA-HD). This framework enhances the generalization of adversarial examples by integrating a hypothesis defense mechanism into the proxy model. Specifically, we propose an input denoising network as the hypothesis defense to effectively remove harmful noise from adversarial examples. Furthermore, we introduce an adversarial training strategy and design specific adversarial loss functions to optimize the input denoising network’s parameters. The visualization of the training process demonstrates the effective denoising capability of the hypothesized defense mechanism and the stability of the training process. Extensive experiments show that the proposed training framework significantly improves the success rate of transfer-based attacks by up to 19.9%. The code is available at https://github.com/haolingguang/TA-HD.

Stealthy Black-Box Attack With Dynamic Threshold Against MARL-Based Traffic Signal Control System

Stealthy Black-Box Attack With Dynamic Threshold Against MARL-Based Traffic Signal Control System

Dynamic Routing and Knowledge Re-Learning for Data-Free Black-Box Attack.

Deep learning models have emerged as strong and efficient tools that can be applied to a broad spectrum of complex learning problems and many real-world applications. However, more and more works show that deep models are vulnerable to adversarial examples. Compared to vanilla attack settings, this paper advocates a more practical setting of data-free black-box attack, for which the attackers can completely not access the structures and parameters of the target model, as well as the intermediate features and any training data associated with the model. To tackle this task, previous methods generate transferable adversarial examples from a transparent substitute model to the target model. However, we found that these works have the limitations of taking static substitute model structure for different targets, only using hard synthesized examples once, and still relying on data statistics of the target model. This may potentially harm the performance of attacking the target model. To this end, we propose a novel Dynamic Routing and Knowledge Re-Learning framework (DraKe) to effectively learn a dynamic substitute model from the target model. Specifically, given synthesized training samples, a dynamic substitute structure learning strategy is proposed to adaptively generate optimal substitute model structure via a policy network according to different target models and tasks. To facilitate the substitute training, we present a graph-based structure information learning to capture the structural knowledge learned from the target model. For the inherent limitation that online data generation can only be learned once, a dynamic knowledge re-learning strategy is proposed to adjust the weights of optimization objectives and re-learn hard samples. Extensive experiments on four public image classification datasets and one face recognition benchmark are conducted to evaluate the efficacy of our Drake. We can obtain significant improvement compared with state-of-the-art competitors. More importantly, our DraKe consistently achieves attack superiority for different target models (e.g., residual networks, and vision transformers), showing great potential for complex real-world applications.

Black-box adversarial attacks against image quality assessment models

The problem of No-Reference Image Quality Assessment (NR-IQA) is to predict the perceptual quality of an image in line with its subjective evaluation. However, the vulnerabilities of NR-IQA models to the adversarial attacks have not been thoroughly studied for model refinement. This paper aims to investigate the potential loopholes of NR-IQA models via black-box adversarial attacks. Specifically, we first formulate the attack problem as maximizing the deviation between the estimated quality scores of original and perturbed images, while restricting the perturbed image distortions for visual quality preservation. Under such formulation, we then design a Bi-directional loss function to mislead the estimated quality scores of adversarial examples towards an opposite direction with maximum deviation. On this basis, we finally develop an efficient and effective black-box attack method for NR-IQA models based on a random search paradigm. Comprehensive experiments on three benchmark datasets show that all evaluated NR-IQA models are significantly vulnerable to the proposed attack method. After being attacked, the average change rates in terms of two well-known IQA performance metrics achieved by victim models reach 97% and 101%, respectively. In addition, our attack method also outperforms a newly introduced black-box attack approach on IQA models. We also observe that the generated perturbations are not transferable, which points out a new research direction in NR-IQA community. The source code is available at https://github.com/GZHU-DVL/AttackIQA.

Decoupling visual and identity features for adversarial palm-vein image attack

Palm-vein has been widely used for biometric recognition due to its resistance to theft and forgery. However, with the emergence of adversarial attacks, most existing palm-vein recognition methods are vulnerable to adversarial image attacks, and to the best of our knowledge, there is still no study specifically focusing on palm-vein image attacks. In this paper, we propose an adversarial palm-vein image attack network that generates highly similar adversarial palm-vein images to the original samples, but with altered palm-identities. Unlike most existing generator-oriented methods that directly learn image features via concatenated convolutional layers, our proposed network first maps palm-vein images into multi-scale high-dimensional shallow representation, and then develops attention-based dual-path feature learning modules to extensively exploit diverse palm-vein-specific features. After that, we design visual-consistency and identity-aware loss functions to specially decouple the visual and identity features to reconstruct the adversarial palm-vein images. By doing this, the visual characteristics of palm-vein images can be largely preserved while the identity information is removed in the adversarial palm-vein images, such that high-aggressive adversarial palm-vein samples can be obtained. Extensive white-box and black-box attack experiments conducted on three widely used databases clearly show the effectiveness of the proposed network.

CARL: Unsupervised Code-Based Adversarial Attacks for Programming Language Models via Reinforcement Learning

Code based adversarial attacks play a crucial role in revealing vulnerabilities of software system. Recently, pre-trained programming language models (PLMs) have demonstrated remarkable success in various significant software engineering tasks, progressively transforming the paradigm of software development. Despite their impressive capabilities, these powerful models are vulnerable to adversarial attacks. Therefore, it is necessary to carefully investigate the robustness and vulnerabilities of the PLMs by means of adversarial attacks. Adversarial attacks entail imperceptible input modifications that cause target models to make incorrect predictions. Existing approaches for attacking PLMs often employ either identifier renaming or the greedy algorithm, which may yield sub-optimal performance or lead to high inference times. In response to these limitations, we propose CARL, an unsupervised black-box attack model that leverages reinforcement learning to generate imperceptible adversarial examples. Specifically, CARL comprises a programming language encoder and a perturbation prediction layer. In order to achieve more effective and efficient attack, we cast the task as a sequence decision-making process, optimizing through policy gradient with a suite of reward functions. We conduct extensive experiments to validate the effectiveness of CARL on code summarization, code translation, and code refinement tasks, covering various programming languages and PLMs. The experimental results demonstrate that CARL surpasses state-of-the-art code attack models, achieving the highest attack success rate across multiple tasks and PLMs while maintaining high attack efficiency, imperceptibility, consistency, and fluency.

Spatial-frequency gradient fusion based model augmentation for high transferability adversarial attack

In recent years, deep learning has gained widespread application across diverse fields, including image classification and machine translation. Nevertheless, the emergence of adversarial examples has revealed a vulnerability of deep learning techniques to potential attacks. Despite the introduction of diverse adversarial attack methods, they are still constrained by certain limitations. Specifically, global perturbations are easily discernible by humans, resulting in poor imperceptibility. Additionally, current methods encounter limited transferability due to their reliance on attacking specific models. To address these challenges, this paper proposed a spatial-frequency gradient fusion based model augmentation for adversarial attack. First, we utilize a Gaussian convolution kernel to pinpoint regions in images that exhibit significant pixel variation, aiming to generate locally imperceptible perturbations undetectable by humans. These areas, which we consider as complex texture regions, are ideal for adding perturbations. Then, we design a perceptual similarity constraint to regulate the generation of perturbations in smooth texture regions. Subsequently, to further enhance the transferability of our method, we propose a spatial-frequency gradient fusion based model augmentation, applying random spectral transformation to shift into the frequency domain for narrowing the differences between models. Additionally, we design complex region scaling transformations in the spatial domain, aimed at capturing common features shared across models. Finally, we integrate the gradients from both the spatial and frequency domains, leveraging the strengths of both to empower attack models in effectively simulating the target model. Extensive experiments conducted on ImageNet and CIFAR-10 datasets have shown that our method attains a remarkable black-box attack success rate of up to 93.1%, with a perceptual loss reduction of approximately 8.39%, while also exhibiting stronger robustness.

Boosting the transferability of adversarial attacks with global momentum initialization

Deep Neural Networks (DNNs) are vulnerable to adversarial examples, which are crafted by adding human-imperceptible perturbations to the benign inputs. Simultaneously, adversarial examples exhibit transferability across models, enabling practical black-box attacks. However, existing methods are still incapable of achieving the desired transfer attack performance. In this work, focusing on gradient optimization and consistency, we analyze the gradient elimination phenomenon as well as the local momentum optimum dilemma. To tackle these challenges, we introduce Global Momentum Initialization (GI), providing global momentum knowledge to mitigate gradient elimination. Specifically, we perform gradient pre-convergence before the attack and a global search during this stage. GI seamlessly integrates with existing transfer methods, significantly improving the success rate of transfer attacks by an average of 6.4% under various advanced defense mechanisms compared to the state-of-the-art method. Ultimately, GI demonstrates strong transferability in both image and video attack domains. Particularly, when attacking advanced defense methods in the image domain, it achieves an average attack success rate of 95.4%. The code is available at https://github.com/Omenzychen/Global-Momentum-Initialization.

DBI-Attack: Dynamic Bi-Level Integrated Attack for Intensive Multi-Scale UAV Object Detection

Benefiting from the robust feature representation capability of convolutional neural networks (CNNs), the object detection technology of intelligent high-altitude UAV remote sensing has been developed rapidly. In this field, the adversarial examples (AEs) pose serious security risks and vulnerabilities to deep learning-based systems. Due to the limitation of object size, image degradation, and scene brightness, adding adversarial disturbances to small and dense objects is extremely challenging. To study the threat of AE for UAV object detection, a dynamic bi-level integrated attack (DBI-Attack) is proposed for intensive multi-scale UAV object detection. Firstly, we use the dynamic iterative attack (DIA) method to generate perturbation on the classification level by improving the momentum iterative fast gradient sign method (MIM). Secondly, the bi-level adversarial attack method (BAAM) is constructed to add global perturbation on the decision level for completing the white-box attack. Finally, the integrated black-box attack method (IBAM) is combined to realize the black-box mislabeling and fabrication attacks. We experiment on the real drone traffic vehicle detection datasets to better evaluate the attack effectiveness. The experimental results show that the proposed method can achieve mislabeling and fabrication attacks on the UAV object detectors in black-box conditions. Furthermore, the adversarial training is applied to improve the model robustness. This work aims to call more attention to the adversarial and defensive aspects of UAV target detection models.

Efficient Adversarial Attack Based on Moment Estimation and Lookahead Gradient

Adversarial example generation is a technique that involves perturbing inputs with imperceptible noise to induce misclassifications in neural networks, serving as a means to assess the robustness of such models. Among the adversarial attack algorithms, momentum iterative fast gradient sign Method (MI-FGSM) and its variants constitute a class of highly effective offensive strategies, achieving near-perfect attack success rates in white-box settings. However, these methods’ use of sign activation functions severely compromises gradient information, which leads to low success rates in black-box attacks and results in large adversarial perturbations. In this paper, we introduce a novel adversarial attack algorithm, NA-FGTM. Our method employs the Tanh activation function instead of the sign which can accurately preserve gradient information. In addition, it utilizes the Adam optimization algorithm as well as the Nesterov acceleration, which is able to stabilize gradient update directions and expedite gradient convergence. Above all, the transferability of adversarial examples can be enhanced. Through integration with data augmentation techniques such as DIM, TIM, and SIM, NA-FGTM can further improve the efficacy of black-box attacks. Extensive experiments on the ImageNet dataset demonstrate that our method outperforms the state-of-the-art approaches in terms of black-box attack success rate and generates adversarial examples with smaller perturbations.

Boosting transferability of adversarial samples via saliency distribution and frequency domain enhancement

Deep neural networks (DNNs) perform excellently in various vision tasks, however, they are susceptible to the adversarial samples. These samples are crafted by injecting subtle perturbations into benign images that are barely detectable to humans. Due to the differences between the substitute model and target model, the efficacy of black-box attack still yield suboptimal results. To address this issue, we propose a black-box attack approach, SD-FI2M, based on the momentum iterative fast gradient sign method, which can craft potent adversarial samples by inducing diverse inputs in the frequency domain and the saliency distribution of images. To enhance the diversity of input samples in frequency domain, a spectrum transformation technique utilizing the discrete cosine transform was used to craft adversarial samples with higher transferability. Further, Guided Grad-CAM was employed to obtain the saliency distribution of benign images, reducing indiscriminate damage to image features and alleviating the overfitting problem in adversarial samples. Extensive experiments on ImageNet have verified the superior attack performance and transferability of the proposed method.