Black-box Adversarial Attacks Research Articles

Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey

Network-based Intrusion Detection System (NIDS) forms the frontline defence against network attacks that compromise the security of the data, systems, and networks. In recent years, Deep Neural Networks (DNNs) have been increasingly used in NIDS to detect malicious traffic due to their high detection accuracy. However, DNNs are vulnerable to adversarial attacks that modify an input example with imperceivable perturbation, which causes a misclassification by the DNN. In security-sensitive domains, such as NIDS, adversarial attacks pose a severe threat to network security. However, existing studies in adversarial learning against NIDS directly implement adversarial attacks designed for Computer Vision (CV) tasks, ignoring the fundamental differences in the detection pipeline and feature spaces between CV and NIDS. It remains a major research challenge to launch and detect adversarial attacks against NIDS. This article surveys the recent literature on NIDS, adversarial attacks, and network defences since 2015 to examine the differences in adversarial learning against deep neural networks in CV and NIDS. It provides the reader with a thorough understanding of DL-based NIDS, adversarial attacks and defences, and research trends in this field. We first present a taxonomy of DL-based NIDS and discuss the impact of taxonomy on adversarial learning. Next, we review existing white-box and black-box adversarial attacks on DNNs and their applicability in the NIDS domain. Finally, we review existing defence mechanisms against adversarial examples and their characteristics.

Exploring Diverse Feature Extractions for Adversarial Audio Detection

Although deep learning models have exhibited excellent performance in various domains, recent studies have discovered that they are highly vulnerable to adversarial attacks. In the audio domain, malicious audio examples generated by adversarial attacks can cause significant performance degradation and system malfunctions, resulting in security and safety concerns. However, compared to recent developments in the audio domain, the properties of the adversarial audio examples and defenses against them still remain largely unexplored. In this study, to provide a deeper understanding of the adversarial robustness in the audio domain, we first investigate traditional and recent feature extractions in terms of adversarial attacks. We show that adversarial audio examples generated from different feature extractions exhibit different noise patterns, and thus can be distinguished by a simple classifier. Based on the observation, we extend existing adversarial detection methods by proposing a new detection method that detects adversarial audio examples using an ensemble of diverse feature extractions. By combining the frequency and self-supervised feature representations, the proposed method provides a high detection rate against both white-box and black-box adversarial attacks. Our empirical results demonstrate the effectiveness of the proposed method in speech command classification and speaker recognition.

Consensus Adversarial Defense Method Based on Augmented Examples

Deep learning has been used in many computer-vision-based industrial Internet of Things applications. However, deep neural networks are vulnerable to adversarial examples that have been crafted specifically to fool a system while being imperceptible to humans. In this article, we propose a consensus defense (Cons-Def) method to defend against adversarial attacks. Cons-Def implements classification and detection based on the consensus of the classifications of the augmented examples, which are generated based on an individually implemented intensity exchange on the red, green, and blue components of the input image. We train a CNN using augmented examples together with their original examples. For the test image to be assigned to a specific class, the class occurrence of the classifications on its augmented images should be the maximum and reach a defined threshold. Otherwise, it is detected as an adversarial example. The comparison experiments are implemented on MNIST, CIFAR-10, and ImageNet. The average defense success rate (DSR) against white-box attacks on the test sets of the three datasets is 80.3%. The average DSR against black-box attacks on CIFAR-10 is 91.4%. The average classification accuracies of Cons-Def on benign examples of the three datasets are 98.0%, 78.3%, and 66.1%. The experimental results show that Cons-Def shows a high classification performance on benign examples and is robust against white-box and black-box adversarial attacks.

Research on a Method of Defense Adversarial Samples for Target Detection Model of Driverless Cars

The adversarial examples make the object detection model make a wrong judgment, which threatens the security of driverless cars. In this paper, by improving the Momentum Iterative Fast Gradient Sign Method (MI-FGSM), based on ensemble learning, combined with L∞ perturbation and spatial transformation, a strong transferable black-box adversarial attack algorithm for object detection model of driverless cars is proposed. Through a large number of experiments on the nuScenes driverless dataset, it is proved that the adversarial attack algorithm proposed in this paper have strong transferability, and successfully make the mainstream object detection models such as FasterRcnn, SSD, YOLOv3 make wrong judgments. Based on the adversarial attack algorithm proposed in this paper, the parametric noise injection with adversarial training is performed to generate a defense model with strong robustness. The defense model proposed in this paper significantly improves the robustness of the object detection model. It can effectively alleviate various adversarial attacks against the object detection model of driverless cars, and does not affect the accuracy of clean samples. This is of great significance for studying the application of object detection model of driverless cars in the real physical world.

Natural Weather-Style Black-Box Adversarial Attacks Against Optical Aerial Detectors

Natural Weather-Style Black-Box Adversarial Attacks Against Optical Aerial Detectors

IIoT Deep Malware Threat Hunting: From Adversarial Example Detection to Adversarial Scenario Detection

Protecting widely used deep classifiers against black-box adversarial attacks is a recent research challenge in many security-related areas, including malware classification. This class of attacks relies on optimizing a sequence of highly similar queries to bypass given classifiers. In this article, we leverage this property and propose a history-based method named, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stateful query analysis (SQA)</i> , which analyzes sequences of queries received by a malware classifier to detect black-box adversarial attacks on an industrial Internet of Things (IIoT). In the SQA pipeline, there are two components, namely the similarity encoder and the classifier, both based on convolutional neural networks. Unlike the state-of-the-art methods, which aim to identify individual adversarial examples, tracking the history of queries allows our method to identify adversarial scenarios and abort attacks before their completion. We optimize SQA using different combinations of hyperparameters on an advanced risc machine (ARM)-based IIoT malware dataset, widely adopted for malware threat hunting in industry 4.0. The use of a novel distance metric in calculating the loss function of the similarity encoder results in more disentangled representations and improves the performance of our method. Our evaluations demonstrate the validity of SQA via a detection rate of 93.1% over a wide range of adversarial examples.

Query-Efficient Black-Box Adversarial Attacks Guided by a Transfer-Based Prior.

Adversarial attacks have been extensively studied in recent years since they can identify the vulnerability of deep learning models before deployed. In this paper, we consider the black-box adversarial setting, where the adversary needs to craft adversarial examples without access to the gradients of a target model. Previous methods attempted to approximate the true gradient either by using the transfer gradient of a surrogate white-box model or based on the feedback of model queries. However, the existing methods inevitably suffer from low attack success rates or poor query efficiency since it is difficult to estimate the gradient in a high-dimensional input space with limited information. To address these problems and improve black-box attacks, we propose two prior-guided random gradient-free (PRGF) algorithms based on biased sampling and gradient averaging, respectively. Our methods can take the advantage of a transfer-based prior given by the gradient of a surrogate model and the query information simultaneously. Through theoretical analyses, the transfer-based prior is appropriately integrated with model queries by an optimal coefficient in each method. Extensive experiments demonstrate that, in comparison with the alternative state-of-the-arts, both of our methods require much fewer queries to attack black-box models with higher success rates.

Multiuser Adversarial Attack on Deep Learning for OFDM Detection

Adversarial attack has been widely used to degrade the performance of deep learning (DL), especially in the field of communications. In this letter, we evaluate different white-box and black-box adversarial attack algorithms for a DL-based multiuser orthogonal frequency division multiplexing (OFDM) detector subject to multiuser adversarial attack. The bit error rates under different adversarial attacks are compared. The results show that, the perturbation efficiency of adversarial attack is higher than conventional multiuser interference. Virtual adversarial methods (VAM) and zeroth-order-optimization (ZOO) attacks perform the best among white-box and black-box methods, respectively. They are also effective when the attack changes the starting time. Additionally, adding the number of attackers is found useful to improve the VAM attack but not for ZOO. This letter shows that adversarial attack is powerful to generate adversarial against multiuser OFDM communications.

An autonomous decision-making framework for gait recognition systems against adversarial attack using reinforcement learning

Gait identification based on Deep Learning (DL) techniques has recently emerged as biometric technology for surveillance. We leveraged the vulnerabilities and decision-making abilities of the DL model in gait-based autonomous surveillance systems when attackers have no access to underlying model gradients/structures using a patch-based black-box adversarial attack with Reinforcement Learning (RL). These automated surveillance systems are secured, blocking the attacker’s access. Therefore, the attack can be conducted in an RL framework where the agent’s goal is determining the optimal image location, causing the model to perform incorrectly when perturbed with random pixels. Furthermore, the proposed adversarial attack presents encouraging results (maximum success rate = 77.59%). Researchers should explore system resilience scenarios (e.g., when attackers have no system access) before using these models in surveillance applications.

Practical black-box adversarial attack on open-set recognition: Towards robust autonomous driving

Practical black-box adversarial attack on open-set recognition: Towards robust autonomous driving

Aliasing black box adversarial attack with joint self-attention distribution and confidence probability

Deep neural networks (DNNs) are vulnerable to adversarial attacks, in which a small perturbation to samples can cause misclassification. However, how to select important words for textual attack models is a big challenge. Therefore, in this paper, an innovative score-based attack model is proposed to solve the important words selection problem for textual attack models. To this end, the generation of semantically adversarial examples in this model is adopted to mislead a text classification model. Then, this model integrates the self-attention mechanism and confidence probabilities for the selection of the important words. Moreover, an alternative model similar to the transfer attack is introduced to reflect the correlation degree of words inside the texts. Finally, adversarial training experimental results demonstrate the superiority of the proposed model.

How Resilient Are Deep Learning Models in Medical Image Analysis? The Case of the Moment-Based Adversarial Attack (Mb-AdA).

In the past years, deep neural networks (DNNs) have become popular in many disciplines such as computer vision (CV). One of the most important challenges in the CV area is Medical Image Analysis (MIA). However, adversarial attacks (AdAs) have proven to be an important threat to vision systems by significantly reducing the performance of the models. This paper proposes a new black-box adversarial attack, which is based οn orthogonal image moments named Mb-AdA. Additionally, a corresponding defensive method of adversarial training using Mb-AdA adversarial examples is also investigated, with encouraging results. The proposed attack was applied in classification and segmentation tasks with six state-of-the-art Deep Learning (DL) models in X-ray, histopathology and nuclei cell images. The main advantage of Mb-AdA is that it does not destroy the structure of images like other attacks, as instead of adding noise it removes specific image information, which is critical for medical models’ decisions. The proposed attack is more effective than compared ones and achieved degradation up to 65% and 18% in terms of accuracy and IoU for classification and segmentation tasks, respectively, by also presenting relatively high SSIM. At the same time, it was proved that Mb-AdA adversarial examples can enhance the robustness of the model.

PISA: Pixel skipping-based attentional black-box adversarial attack

The studies on black-box and evolutionary algorithm-based adversarial attacks have become increasingly popular due to the intractable acquisition of the structural knowledge of deep neural networks (DNNs). However, the performance of these emerging attacks is negatively impacted when fooling DNNs tailored for high-resolution images. One of the explanations is that they usually focus on attacking the entire image, regardless of its spatial semantic information, and thereby encounter the notorious curse of dimensionality. To this end, we propose a pixel skipping and evolutionary algorithm-based attentional black-box adversarial attack, termed PISA. In PISA, only one of every two neighboring pixels in the salient region is recognized as the target by leveraging the attention map and pixel skipping, such that the dimension of the black-box attack reduces. After that, PISA allows the embedding of an arbitrary multiobjective evolutionary algorithm, which is employed to traverse the reduced pixels and finally generates effective perturbations that are imperceptible by human vision. Extensive experimental results have demonstrated that the proposed PISA is more competitive in attacking high-resolution images than existing black-box and evolutionary algorithm-based attacks.

An Approximated Gradient Sign Method Using Differential Evolution for Black-Box Adversarial Attack

Recent studies show that deep neural networks are vulnerable to adversarial attacks in the form of subtle perturbations to the input image, which leads the model to output wrong prediction. Such an attack can easily succeed by the existing white-box attack methods, where the perturbation is calculated based on the gradient of the target network. Unfortunately, the gradient is often unavailable in the real-world scenarios, which makes the black-box adversarial attack problems practical and challenging. In fact, they can be formulated as high-dimensional black-box optimization problems at the pixel level. Although evolutionary algorithms are well known for solving black-box optimization problems, they cannot efficiently deal with the high-dimensional decision space. Therefore, we propose an approximated gradient sign method using differential evolution (DE) for solving black-box adversarial attack problems. Unlike most existing methods, it is novel that the proposed method searches the gradient sign rather than the perturbation by a DE algorithm. Also, we transform the pixel-based decision space into a dimension-reduced decision space by combining the pixel differences from the input image to neighbor images, and two different techniques for selecting neighbor images are introduced to build the transferred decision space. In addition, six variants of the proposed method are designed according to the different neighborhood selection and optimization search strategies. Finally, the performance of the proposed method is compared with a number of the state-of-the-art adversarial attack algorithms on CIFAR-10 and ImageNet datasets. The experimental results suggest that the proposed method shows superior performance for solving black-box adversarial attack problems, especially nontargeted attack problems.

Adversarial attacks and adversarial robustness in computational pathology

Artificial Intelligence (AI) can support diagnostic workflows in oncology by aiding diagnosis and providing biomarkers directly from routine pathology slides. However, AI applications are vulnerable to adversarial attacks. Hence, it is essential to quantify and mitigate this risk before widespread clinical use. Here, we show that convolutional neural networks (CNNs) are highly susceptible to white- and black-box adversarial attacks in clinically relevant weakly-supervised classification tasks. Adversarially robust training and dual batch normalization (DBN) are possible mitigation strategies but require precise knowledge of the type of attack used in the inference. We demonstrate that vision transformers (ViTs) perform equally well compared to CNNs at baseline, but are orders of magnitude more robust to white- and black-box attacks. At a mechanistic level, we show that this is associated with a more robust latent representation of clinically relevant categories in ViTs compared to CNNs. Our results are in line with previous theoretical studies and provide empirical evidence that ViTs are robust learners in computational pathology. This implies that large-scale rollout of AI models in computational pathology should rely on ViTs rather than CNN-based classifiers to provide inherent protection against perturbation of the input data, especially adversarial attacks.

An Optimized Black-Box Adversarial Simulator Attack Based on Meta-Learning

Much research on adversarial attacks has proved that deep neural networks have certain security vulnerabilities. Among potential attacks, black-box adversarial attacks are considered the most realistic based on the the natural hidden nature of deep neural networks. Such attacks have become a critical academic emphasis in the current security field. However, current black-box attack methods still have shortcomings, resulting in incomplete utilization of query information. Our research, based on the newly proposed Simulator Attack, proves the correctness and usability of feature layer information in a simulator model obtained by meta-learning for the first time. Then, we propose an optimized Simulator Attack+ based on this discovery. Our optimization methods used in Simulator Attack+ include: (1) a feature attentional boosting module that uses the feature layer information of the simulator to enhance the attack and accelerate the generation of adversarial examples; (2) a linear self-adaptive simulator-predict interval mechanism that allows the simulator model to be fully fine-tuned in the early stage of the attack and dynamically adjusts the interval for querying the black-box model; and (3) an unsupervised clustering module to provide a warm-start for targeted attacks. Results from experiments on the CIFAR-10 and CIFAR-100 datasets clearly show that Simulator Attack+ can further reduce the number of consuming queries to improve query efficiency while maintaining the attack.

Query efficient black-box adversarial attack on deep neural networks

Deep neural networks (DNNs) have demonstrated excellent performance on various tasks, yet they are under the risk of adversarial examples that can be easily generated when the target model is accessible to an attacker (white-box setting). As plenty of machine learning models have been deployed via online services that only provide query outputs from inaccessible models (e.g., Google Cloud Vision API2), black-box adversarial attacks raise critical security concerns in practice rather than white-box ones. However, existing query-based black-box adversarial attacks often require excessive model queries to maintain a high attack success rate. Therefore, in order to improve query efficiency, we explore the distribution of adversarial examples around benign inputs with the help of image structure information characterized by a Neural Process, and propose a Neural Process based black-box adversarial attack (NP-Attack) in this paper. Our proposed NP-Attack could be further boosted when applied with surrogate models or tiling tricks. Extensive experiments show that NP-Attack could greatly decrease the query counts under the black-box setting.

Taking Care of the Discretization Problem: A Comprehensive Study of the Discretization Problem and a Black-Box Adversarial Attack in Discrete Integer Domain

Neural network based (NN-based) classifiers are known vulnerable against adversarial examples, namely, adding slight perturbations to a benign image cause a classifier to make a false prediction. To evaluate the robustness of NN-based classifiers against adversarial examples, numerous adversarial attacks with high success rates have been proposed recently. NN-based image classifiers usually normalize valid images (e.g., RGB image where the value at each coordinate is an integer between 0 and 255) into a real continuous domain (e.g., 3-dimensional matrix where the value at each coordinate is a real number between 0 and 1) and make classification decisions on the normalized images. However, adversarial examples crafted in a real continuous domain may become benign once they are denormalized back into the corresponding discrete integer domain, known as the discretization problem. This problem has been mentioned in some prior works but received relatively limited attention. In this work, we report the first comprehensive study of existing works to understand the impacts of the discretization problem. By analyzing 35 representative methods and empirically studying 20 representative open source tools, we found 29/35 (theoretically) and 14/20 (empirically) are affected by the discretization problem, e.g., the success rate could dramatically drop from 100 to 10 percent after the domain transformation. As the first step towards addressing this problem in a black-box scenario, we propose a novel derivative-free optimization method, which can directly craft adversarial examples in the discrete integer domain. Experimental results show that the method achieves nearly 100 percent attack success rates for both targeted and untargeted attacks, comparable to the most popular white-box methods (FGSM, BIM and C&W), and significantly outperforms representative black-box methods (ZOO, AutoZOOM, NES-PGD, Bandits, FD, FD-PSO and GenAttack). Our results suggest that the discretization problem should be treated more seriously, and the discrete optimization algorithms show a promising future in crafting effective black-box attacks.

Generating Adversarial Examples With Shadow Model

The reduction in the number of queries to the object model is a hot topic in the current research of black-box adversarial attack methods. To solve this problem, in this article, we propose generating adversarial examples with shadow model (GASM) that shifts the number of queries to the object model to the shadow model. The method first determines the shadow model based on the robustness and transferability of classifiers and fine-tunes the decision boundary of the shadow model by constructing adversarial datasets. Second, accesses the shadow model and constructs adversarial examples by maximizing the output probability of the targeted class (any class other than the current one) to modify the image gradient information. Finally, the results show that GASM has the strongest transferability and outperforms white-box attacks when AlexNet (MNIST), VGG-19 (CIFAR10), and MobileNet v2 (Tiny ImageNet) are selected as shadow models.

ZQBA: A Zero-Query, Boosted Ambush Adversarial Attack on Image Retrieval

Content Based Image Retrieval (CBIR) Systems have been employed in a wide variety of critical applications such as intellectual property management [47], facial recognition [46], and inappropriate content detection [48]. Most CBIRs are vulnerable to adversarial attacks, where small, imperceptible perturbations to input images cause system failure. In this paper, we propose a zero-query, black-box adversarial attack method that simulates an attack setting where the attacker has no knowledge about the CBIR model architecture and is unable to make multiple queries. The proposed method uses an ensemblebased approach, generating one perturbation for an input image that severely hinders the ability of six different CBIR models. Our approach is successfully able to disrupt the relevance of our target image retrieval models with a 65% decrease in Mean Average Precision (mAP) as compared to state-of-the-art UAP [18]. We hope our method serves as a baseline for the evaluation of robustness for future image retrieval research.