Black-box Adversarial Attacks Research Articles

FE-DaST: Fast and effective data-free substitute training for black-box adversarial attacks

Deep learning models have shown their advantages in computer vision, e.g., image classification, whereas they are well-known to be susceptible to imperceptible perturbations of input images which are called adversarial attacks. Recently proposed data-free substitute training (DaST), an adversarial framework based on a multi-branch generator where each branch generated images of the corresponding class to balance synthetic images, trained surrogate models without the requirement of any real image for transfer-based black-box adversarial attacks. However, this multi-branch framework was too redundant to converge quickly and was limited to datasets of a few categories. In this paper, we propose a simpler adversarial framework based on a single-branch generator to train substitute models fast and effectively, named FE-DaST. More specifically, we adopt a single-branch deep convolutional generator with an information entropy loss to stimulate the generation of balanced images, promote the similarity between substitute models and target models, and further enhance the strength of the transfer-based attack. Despite its simplicity, experimental results demonstrate the superiority of our proposed FE-DaST over DaST in terms of computational loads, similarities between surrogate models and target models, and attack success rates of transferable adversarial examples on MNIST and CIFAR-10 datasets. For CIFAR-100 and Tiny-ImageNet datasets where DaST is not available, our FE-DaST also achieves competitive attack success rates compared with pre-trained models which are trained with realistic training images. Furthermore, the attack performance of FE-DaST outperforms other state-of-the-art substitute training methods on the four datasets.

Multitask adversarial attack with dispersion amplification

Recently, adversarial attacks have drawn the community’s attention as an effective tool to degrade the accuracy of neural networks. However, their actual usage in the world is limited. The main reason is that real-world machine learning systems, such as content filters or face detectors, often consist of multiple neural networks, each performing an individual task. To attack such a system, adversarial example has to pass through many distinct networks at once, which is the major challenge addressed by this paper. In this paper, we investigate multitask adversarial attacks as a threat for real-world machine learning solutions. We provide a novel black-box adversarial attack, which significantly outperforms the current state-of-the-art methods, such as Fast Gradient Sign Attack (FGSM) and Basic Iterative Method (BIM, also known as Iterative-FGSM) in the multitask setting.

An efficient general black-box adversarial attack approach based on multi-objective optimization for high dimensional images

In black-box scenarios, the adversarial attack algorithms based on iterative optimization perform best. But it may give a false sense of model robustness due to the design of inefficient queries. The adversarial attack algorithm based on multi-objective evolution optimization has been proven to be very effective for the low-dimensional images. However, when the attack space dramatically increases for the high-dimensional color images, the evolutionary efficiency is limited, and it needs more inefficient queries to generate adversarial examples. In this paper, we propose an efficient black-box adversarial attack approach for high dimensional images based on multi-objective optimization (MOO-HD), which includes some novel strategies to solve the above problems. We also propose the strategy of “The transformation of the pixel block with a random step size” to reduce the attack space. The experimental results on three image datasets with different dimensions show that our algorithm can achieve a higher success rate with fewer queries.

A black-box adversarial attack for poisoning clustering

Clustering algorithms play a fundamental role as tools in decision-making and sensible automation processes. Due to the widespread use of these applications, a robustness analysis of this family of algorithms against adversarial noise has become imperative. To the best of our knowledge, however, only a few works have currently addressed this problem. In an attempt to fill this gap, in this work, we propose a black-box adversarial attack for crafting adversarial samples to test the robustness of clustering algorithms. We formulate the problem as a constrained minimization program, general in its structure and customizable by the attacker according to her capability constraints. We do not assume any information about the internal structure of the victim clustering algorithm, and we allow the attacker to query it as a service only. In the absence of any derivative information, we perform the optimization with a custom approach inspired by the Abstract Genetic Algorithm (AGA). In the experimental part, we demonstrate the sensibility of different single and ensemble clustering algorithms against our crafted adversarial samples on different scenarios. Furthermore, we perform a comparison of our algorithm with a state-of-the-art approach showing that we are able to reach or even outperform its performance. Finally, to highlight the general nature of the generated noise, we show that our attacks are transferable even against supervised algorithms such as SVMs, random forests and neural networks.

Deep image prior based defense against adversarial examples

Recently, deep neural networks (DNNs) have shown serious vulnerability to adversarial examples with imperceptible perturbation to clean images. To counter this issue, many powerful defensive methods (e.g., ComDefend) focus on rectifying the adversarial examples with well-trained models from a large training dataset (e.g., clean-adversarial image pairs). However, such methods rely heavily on the learned external priors from an external large training dataset, while neglecting the rich image internal priors of the input itself, thus limiting the generalization of the defense models against the adversarial examples with biased image statistics from the external training dataset. Motivated by deep image prior that can capture rich image statistics from a single image, we propose an effective Deep Image Prior Driven Defense (DIPDefend) method against adversarial examples. With a DIP generator to fit the target/adversarial input, we find that our image reconstruction exhibits quite interesting learning preference from a feature learning perspectives, i.e., the early stage primarily learns the robust features resistant to adversarial perturbation, followed by learning non-robust features that are sensitive to adversarial perturbation. Besides, we develop an adaptive stopping strategy that adapts our method to diverse images. In this way, the proposed model obtains a unique defender for each individual adversarial input, thus being robust to various attackers. Experimental results demonstrate the superiority of our method over the state-of-the-art defense methods against white-box and black-box adversarial attacks.

Binary Black-Box Adversarial Attacks with Evolutionary Learning against IoT Malware Detection

5G is about to open Pandora’s box of security threats to the Internet of Things (IoT). Key technologies, such as network function virtualization and edge computing introduced by the 5G network, bring new security threats and risks to the Internet infrastructure. Therefore, higher detection and defense against malware are required. Nowadays, deep learning (DL) is widely used in malware detection. Recently, research has demonstrated that adversarial attacks have posed a hazard to DL-based models. The key issue of enhancing the antiattack performance of malware detection systems that are used to detect adversarial attacks is to generate effective adversarial samples. However, numerous existing methods to generate adversarial samples are manual feature extraction or using white-box models, which makes it not applicable in the actual scenarios. This paper presents an effective binary manipulation-based attack framework, which generates adversarial samples with an evolutionary learning algorithm. The framework chooses some appropriate action sequences to modify malicious samples. Thus, the modified malware can successfully circumvent the detection system. The evolutionary algorithm can adaptively simplify the modification actions and make the adversarial sample more targeted. Our approach can efficiently generate adversarial samples without human intervention. The generated adversarial samples can effectively combat DL-based malware detection models while preserving the consistency of the executable and malicious behavior of the original malware samples. We apply the generated adversarial samples to attack the detection engines of VirusTotal. Experimental results illustrate that the adversarial samples generated by our method reach an evasion success rate of 47.8%, which outperforms other attack methods. By adding adversarial samples in the training process, the MalConv network is retrained. We show that the detection accuracy is improved by 10.3%.

A black-box adversarial attack strategy with adjustable sparsity and generalizability for deep image classifiers

Constructing adversarial perturbations for deep neural networks is an important direction of research. Crafting image-dependent adversarial perturbations using white-box feedback has hitherto been the norm for such adversarial attacks. However, black-box attacks are much more practical for real-world applications. Universal perturbations applicable across multiple images are gaining popularity due to their innate generalizability. There have also been efforts to restrict the perturbations to a few pixels in the image. This helps to retain visual similarity with the original images making such attacks hard to detect. This paper marks an important step that combines all these directions of research. We propose the DEceit algorithm for constructing effective universal pixel-restricted perturbations using only black-box feedback from the target network. We conduct empirical investigations using the ImageNet validation set on the state-of-the-art deep neural classifiers by varying the number of pixels to be perturbed from a meager 10 pixels to as high as all pixels in the image. We find that perturbing only about 10% of the pixels in an image using DEceit achieves a commendable and highly transferable Fooling Rate while retaining the visual quality. We further demonstrate that DEceit can be successfully applied to image-dependent attacks as well. In both sets of experiments, we outperform several state-of-the-art methods.

Derivative-free optimization adversarial attacks for graph convolutional networks.

In recent years, graph convolutional networks (GCNs) have emerged rapidly due to their excellent performance in graph data processing. However, recent researches show that GCNs are vulnerable to adversarial attacks. An attacker can maliciously modify edges or nodes of the graph to mislead the model’s classification of the target nodes, or even cause a degradation of the model’s overall classification performance. In this paper, we first propose a black-box adversarial attack framework based on derivative-free optimization (DFO) to generate graph adversarial examples without using gradient and apply advanced DFO algorithms conveniently. Second, we implement a direct attack algorithm (DFDA) using the Nevergrad library based on the framework. Additionally, we overcome the problem of large search space by redesigning the perturbation vector using constraint size. Finally, we conducted a series of experiments on different datasets and parameters. The results show that DFDA outperforms Nettack in most cases, and it can achieve an average attack success rate of more than 95% on the Cora dataset when perturbing at most eight edges. This demonstrates that our framework can fully exploit the potential of DFO methods in node classification adversarial attacks.

Attention based multi-agent intrusion detection systems using reinforcement learning

Designing an effective network intrusion system (IDS) is a challenging problem because of the emergence of a large number of novel attacks and heterogeneous network applications. The existing IDSs fail to adapt to the changing attack patterns and unseen attacks that lead to inaccurate detection of network vulnerabilities and system performance degradation. Therefore, there is a need to design robust, scalable, efficient, and adaptive IDS for networks. This paper presents a novel deep reinforcement learning-based IDS that employs Deep Q-Network logic in multiple distributed agents and uses attention mechanisms to efficiently detect and classify advanced network attacks. Our proposed multi-agent IDS is designed as a distributed attack detection platform where agents work in a coordinated manner to provide scalable, fault-tolerant, multi-view architecture guided security system. We have tested our model with extensive experimentation on two benchmark datasets: NSL-KDD and CICIDS2017. It shows improved performance in terms of higher accuracy, precision, recall, F1-Score, and low false-positive rate (FPR) in comparison to the state-of-the-art IDS works. On the other hand, many machine learning systems are found vulnerable to adversarial attacks. Thus, we evaluated our model’s robustness against a practical black-box adversarial attack and observed only a little degradation in performance. We integrated the concept of denoising autoencoder (DAE) with our model to further improve its robustness. Finally, we discuss the usability of our system in real-life applications against zero-day attack patterns.

Black-box adversarial attacks based on random coordinate selection

Recent studies show that deep neural networks are vulnerable to adversarial attacks, and the research of adversarial attack has become a hotspot in the field of artificial intelligence security. The decision-based black-box attack is one of the most challenging problems in the field of adversarial attack. Decision-based black-box attacks usually have the problems of high query times and low attack success rate. Several algorithms have been proposed to solve this problem and one of the algorithms presented by Cheng et al. models this attack as an optimization problem. However, this algorithm is based on the method of adding perturbation to the whole pixel, the number of queries is still very high and the convergence perturbation is not necessarily the minimum. In this paper, we present an improved strategy based on stochastic coordinate selection, hoping to improve the query efficiency while reducing the distortion. The convergence of the improved algorithm is analysed, and experiments are carried out on CIFAR-10 and ImageNet datasets. It has been shown that, compared with the original algorithm, our algorithm can find lower distortion using the same number of queries, and the success rate can be improved to different degrees under different query restrictions.

Evaluating and Enhancing the Robustness of Sustainable Neural Relationship Classifiers Using Query-Efficient Black-Box Adversarial Attacks

Neural relation extraction (NRE) models are the backbone of various machine learning tasks, including knowledge base enrichment, information extraction, and document summarization. Despite the vast popularity of these models, their vulnerabilities remain unknown; this is of high concern given their growing use in security-sensitive applications such as question answering and machine translation in the aspects of sustainability. In this study, we demonstrate that NRE models are inherently vulnerable to adversarially crafted text that contains imperceptible modifications of the original but can mislead the target NRE model. Specifically, we propose a novel sustainable term frequency-inverse document frequency (TFIDF) based black-box adversarial attack to evaluate the robustness of state-of-the-art CNN, CGN, LSTM, and BERT-based models on two benchmark RE datasets. Compared with white-box adversarial attacks, black-box attacks impose further constraints on the query budget; thus, efficient black-box attacks remain an open problem. By applying TFIDF to the correctly classified sentences of each class label in the test set, the proposed query-efficient method achieves a reduction of up to 70% in the number of queries to the target model for identifying important text items. Based on these items, we design both character- and word-level perturbations to generate adversarial examples. The proposed attack successfully reduces the accuracy of six representative models from an average F1 score of 80% to below 20%. The generated adversarial examples were evaluated by humans and are considered semantically similar. Moreover, we discuss defense strategies that mitigate such attacks, and the potential countermeasures that could be deployed in order to improve sustainability of the proposed scheme.

Probabilistic Robustness Quantification of Neural Networks

Safety properties of neural networks are critical to their application in safety-critical domains. Quantification of their robustness against uncertainties is an upcoming area of research. In this work, we propose an approach for providing probabilistic guarantees on the performance of a trained neural network. We present two novel metrics for probabilistic verification on training data distribution and test dataset. Given a trained neural network, we quantify the probability of the model to make errors on a random sample drawn from the training data distribution. Second, from the output logits of a sample test point, we measure its p-value on the learned logit distribution to quantify the confidence of the model at this test point. We compare our results with softmax based metric using the black-box adversarial attacks on a simple CNN architecture trained for MNIST digit classification.

A low-query black-box adversarial attack based on transferability

Artificial intelligence systems suffer from black-box adversarial attacks recently. To prevent this kind of attack, a large amount of researches that reveal the nature of this attack has emerged. However, the query count, success rate, and distortion in the existing works cannot fully satisfy the practical purposes. In this paper, we propose a low-query black-box adversarial attack based on transferability by combining the optimization-based method and the transfer-based method. Our approach aims to improve the black-box attack with a lower number of queries, higher success rate, and lower distortion. In addition, we make full use of surrogate models and optimize the objective function to further improve the performance of our algorithm. We verified our method on MNIST (Lecun and Bottou, 1998) [1], CIFAR-10 (Krizhevsky et al., 2009) [2], and ImageNet (Deng et al. 2009) [3], respectively. Experimental results demonstrate that our method can implement a black-box attack with more than 98.5% success rate and achieve specific distortion with less than 5% queries comparing with other state-of-the-art methods.

AdvCapsNet: To defense adversarial attacks based on Capsule networks

Convolutional neural networks have achieved the state-of-the-art results across numerous applications, but recent work finds that these models can be easily fooled by adversarial perturbations. This is partially due to gradient calculation instability, which may be amplified throughout network layers (Liao et al., 2018). To address this issue, we propose a novel AdvCapsNet derived from Capsule (Sabour et al., 2017), which utilizes a significantly more complicated non-linearity, to defend against adversarial attacks. In this paper, we focus on the transfer-based black-box adversarial attacks, which are more practical than their white-box counterparts. Specifically, we investigate vanilla Capsule’s robustness and boost its performance by introducing an adversarial loss function as regularization. The weight updating between capsule layers is implemented via dynamic routing regularized by the additional adversarial term. Extensive experiments demonstrate that the proposed AdvCapsNet can significantly boost Capsule’s robustness and that AdvCapsNet is far more resistance to adversarial attacks than alternative baselines, including both CNN- and Capsule-based defense models.

Adjust-free adversarial example generation in speech recognition using evolutionary multi-objective optimization under black-box condition

This paper proposes a black-box adversarial attack method to automatic speech recognition systems. Some studies have attempted to attack neural networks for speech recognition; however, these methods did not consider the robustness of generated adversarial examples against timing lag with a target speech. The proposed method in this paper adopts Evolutionary Multi-objective Optimization (EMO)that allows it generating robust adversarial examples under black-box scenario. Experimental results showed that the proposed method successfully generated adjust-free adversarial examples, which are sufficiently robust against timing lag so that an attacker does not need to take the timing of playing it against the target speech.

Cyclic Defense GAN Against Speech Adversarial Attacks

This paper proposes a new defense approach for counteracting state-of-the-art white and black-box adversarial attack algorithms. Our approach fits into the implicit reactive defense algorithm category since it does not directly manipulate the potentially malicious input signals. Instead, it reconstructs a similar signal with a synthesized spectrogram using a cyclic generative adversarial network. This cyclic framework helps to yield a stable generative model. Finally, we feed the reconstructed signal into the speech-to-text model for transcription. The conducted experiments on targeted and non-targeted adversarial attacks developed for attacking DeepSpeech, Kaldi, and Lingvo models demonstrate the proposed defense's effectiveness in adverse scenarios.

Graph interpolating activation improves both natural and robust accuracies in data-efficient deep learning

Improving the accuracy and robustness of deep neural nets (DNNs) and adapting them to small training data are primary tasks in deep learning (DL) research. In this paper, we replace the output activation function of DNNs, typically the data-agnostic softmax function, with a graph Laplacian-based high-dimensional interpolating function which, in the continuum limit, converges to the solution of a Laplace–Beltrami equation on a high-dimensional manifold. Furthermore, we propose end-to-end training and testing algorithms for this new architecture. The proposed DNN with graph interpolating activation integrates the advantages of both deep learning and manifold learning. Compared to the conventional DNNs with the softmax function as output activation, the new framework demonstrates the following major advantages: First, it is better applicable to data-efficient learning in which we train high capacity DNNs without using a large number of training data. Second, it remarkably improves both natural accuracy on the clean images and robust accuracy on the adversarial images crafted by both white-box and black-box adversarial attacks. Third, it is a natural choice for semi-supervised learning. This paper is a significant extension of our earlier work published in NeurIPS, 2018. For reproducibility, the code is available athttps://github.com/BaoWangMath/DNN-DataDependentActivation.

A Distributed Black-Box Adversarial Attack Based on Multi-Group Particle Swarm Optimization.

Adversarial attack techniques in deep learning have been studied extensively due to its stealthiness to human eyes and potentially dangerous consequences when applied to real-life applications. However, current attack methods in black-box settings mainly employ a large number of queries for crafting their adversarial examples, hence making them very likely to be detected and responded by the target system (e.g., artificial intelligence (AI) service provider) due to its high traffic volume. A recent proposal able to address the large query problem utilizes a gradient-free approach based on Particle Swarm Optimization (PSO) algorithm. Unfortunately, this original approach tends to have a low attack success rate, possibly due to the model’s difficulty of escaping local optima. This obstacle can be overcome by employing a multi-group approach for PSO algorithm, by which the PSO particles can be redistributed, preventing them from being trapped in local optima. In this paper, we present a black-box adversarial attack which can significantly increase the success rate of PSO-based attack while maintaining a low number of query by launching the attack in a distributed manner. Attacks are executed from multiple nodes, disseminating queries among the nodes, hence reducing the possibility of being recognized by the target system while also increasing scalability. Furthermore, we utilize Multi-Group PSO with Random Redistribution (MGRR-PSO) for perturbation generation, performing better than the original approach against local optima, thus achieving a higher success rate. Additionally, we propose to efficiently remove excessive perturbation (i.e, perturbation pruning) by utilizing again the MGRR-PSO rather than a standard iterative method as used in the original approach. We perform five different experiments: comparing our attack’s performance with existing algorithms, testing in high-dimensional space in ImageNet dataset, examining our hyperparameters (i.e., particle size, number of clients, search boundary), and testing on real digital attack to Google Cloud Vision. Our attack proves to obtain a 100% success rate on MNIST and CIFAR-10 datasets and able to successfully fool Google Cloud Vision as a proof of the real digital attack by maintaining a lower query and wide applicability.

Black-box adversarial attacks by manipulating image attributes

Although there exist various adversarial attacking methods, most of them are performed by generating adversarial noises. Inspired by the fact that people usually set different camera parameters to obtain diverse visual styles when taking a picture, we propose the adversarial attributes, which generate adversarial examples by manipulating the image attributes like brightness, contrast, sharpness, chroma to simulate the imaging process. This task is accomplished under the black-box setting, where only the predicted probabilities are known. We formulate this process into an optimization problem. After efficiently solving this problem, the optimal adversarial attributes are obtained with limited queries. To guarantee the realistic effect of adversarial examples, we bound the attribute changes using Lp norm versus different p values. Besides, we also give a formal explanation for the adversarial attributes based on the linear nature of Deep Neural Networks (DNNs). Extensive experiments are conducted on two public datasets, including CIFAR-10 and ImageNet with respective to four representative DNNs like VGG16, AlexNet, Inception v3 and Resnet50. The results show that at most 97.79% of images in CIFAR-10 test dataset and 98.01% of the ImageNet images can be successfully perturbed to at least one wrong class with only ⩽300 queries per image on average.

Black-box attack against handwritten signature verification with region-restricted adversarial perturbations

Handwritten signature verification is used to verify the identity of individuals through recognizing their signatures. Adversarial examples can induce misclassification, hence posing a severe threat to signature verification. At present, a variety of adversarial example attacks have been developed for image classification, but they are not that useful for attacking signature verification due to two main reasons. First, adversarial perturbations are likely to be imposed on the background of signature images, making them perceptible to human eyes. Second, perfect knowledge about signature verification systems is actually unavailable to attackers. Therefore, how to generate effective and stealthy signature adversarial examples is still an open issue. To shed insights on this challenging problem, we propose the first black-box adversarial example attack against handwritten signature verification in this paper. Our method has two key designs. First, its perturbations are intentionally restricted to the foreground (i.e., strokes) of signature images, which reduces the risk of being recognized by humans. Second, a gradient-free method is developed to achieve the desired perturbations through iteratively updating their positions and optimizing their intensity. Extensive experiments confirm the three advantages of our method. First, the adversarial perturbations generated by our method are almost invisible, while those generated by existing methods are more well-marked. Second, our method defeats the state-of-the-art signature verification method with a surprisingly high success rate of 92.1%. Last, our method breaks through the defense of background cleaning, although this defense can deactivate almost all the existing adversarial example attacks towards signature verification.