Vulnerability Of Deep Neural Networks Research Articles

Perturbation Augmentation for Adversarial Training with Diverse Attacks

Adversarial Training (AT) aims to alleviate the vulnerability of deep neural networks to adversarial perturbations. However, the AT techniques struggle to maintain the performance on natural samples while improving the deep model’s robustness. The absence of perturbation diversity in generated during the adversarial training degrades the generalizability of the robust models, causing overfitting to particular perturbations and a decrease in natural performance. This study proposes an adversarial training framework that augments adversarial directions from a single-step attack to address the trade-off between robustness and generalization. Inspired by feature scattering adversarial training, the proposed framework computes a principal adversarial direction with a single-step attack that finds a perturbation disrupting the inter-sample relationships in the mini-batch during adversarial training. The principal direction obtained at each iteration is augmented by sampling new adversarial directions within a region spanning 45 degrees from the principal adversarial direction. The proposed adversarial training approach does not require extra backpropagation steps in adversarial direction augmentation. Therefore, generalization of the robust model is improved without posing an additional burden on the feature scattering adversarial training. Experiments on CIFAR-10, CIFAR-100, SVHN, Tiny-ImageNet, and The German Traffic Sign Recognition Benchmark consistently improve the accuracy on adversarial with an almost pristine natural performance.

Adversarial Danger Identification on Temporally Dynamic Graphs.

Multivariate time series forecasting plays an increasingly critical role in various applications, such as power management, smart cities, finance, and healthcare. Recent advances in temporal graph neural networks (GNNs) have shown promising results in multivariate time series forecasting due to their ability to characterize high-dimensional nonlinear correlations and temporal patterns. However, the vulnerability of deep neural networks (DNNs) constitutes serious concerns about using these models to make decisions in real-world applications. Currently, how to defend multivariate forecasting models, especially temporal GNNs, is overlooked. The existing adversarial defense studies are mostly in static and single-instance classification domains, which cannot apply to forecasting due to the generalization challenge and the contradiction issue. To bridge this gap, we propose an adversarial danger identification method for temporally dynamic graphs to effectively protect GNN-based forecasting models. Our method consists of three steps: 1) a hybrid GNN-based classifier to identify dangerous times; 2) approximate linear error propagation to identify the dangerous variates based on the high-dimensional linearity of DNNs; and 3) a scatter filter controlled by the two identification processes to reform time series with reduced feature erasure. Our experiments, including four adversarial attack methods and four state-of-the-art forecasting models, demonstrate the effectiveness of the proposed method in defending forecasting models against adversarial attacks.

Adversarial attacks on GAN-based image fusion

Image fusion has achieved significant success, owing to the rapid development of digital computing and Generative Adversarial Networks (GANs). GAN-based fusion techniques fuse latent codes through spatial or arithmetic operations to achieve real image fusion, facilitated by encoders. However, security concerns have arisen due to the vulnerability of deep neural networks to adversarial perturbations. This vulnerability extends to the GAN-based image fusion task. In this paper, we introduce two methods for creating adversarial examples in the context of GAN-based image fusion: adding subtle perturbations to input images or applying the adversarial patch to the input images. The subtle perturbation is meticulously crafted to be imperceptible yet capable of misleading a specific output, regardless of the input images. Conversely, the adversarial patch is a universal perturbation that, when applied to input images as a patch, induces meaningless output images. Our comprehensive experiments, conducted on datasets such as FFHQ (3 × 1024 × 1024) and Stanford Cars (3 × 512 × 512), include both quality and quantity evaluations. According to our experimental results, the subtle perturbation results in almost identical output images, while the adversarial patch induces meaningless fused images and can be transferred to other datasets. By demonstrating that image fusion models are highly vulnerable to adversarial attacks, this study highlights serious concerns regarding the security of these models.

RETRACTED: Integrating confidence calibration and adversarial robustness via adversarial calibration entropy

The vulnerability of deep neural networks to adversarial samples poses significant security concerns. Previous empirical analyses have shown that increasing adversarial robustness through adversarial training leads to models making unconfident decisions, undermining trust in model confidence scores as an accurate indication of their reliability. This raises the question: are adversarial robustness and confidence calibration mutually exclusive? In this work, we find empirically that adversarial examples mislead undefended models to make more confident mistakes during an attack and that adversarial training causes models to become more risk-averse. Further, we investigate the phenomenon of adversarial degradation from an uncertainty perspective and demonstrate that confidence and adversarial robustness can exhibit a uniform trend. To simultaneously improve the model's adversarial robustness and confidence calibration performance, we propose a novel adversarial calibration entropy to regularize the cross-entropy. Extensive experiments show that our approach increases the confidence that the model makes correct decisions and achieves adversarial robustness comparable to current state-of-the-art models.

Image-Level Adaptive Adversarial Ranking for Person Re-Identification.

The potential vulnerability of deep neural networks and the complexity of pedestrian images, greatly limits the application of person re-identification techniques in the field of smart security. Current attack methods often focus on generating carefully crafted adversarial samples or only disrupting the metric distances between targets and similar pedestrians. However, both aspects are crucial for evaluating the security of methods adapted for person re-identification tasks. For this reason, we propose an image-level adaptive adversarial ranking method that comprehensively considers two aspects to adapt to changes in pedestrians in the real world and effectively evaluate the robustness of models in adversarial environments. To generate more refined adversarial samples, our image representation enhancement module leverages channel-wise information entropy, assigning varying weights to different channels to produce images with richer information content, along with a generative adversarial network to create adversarial samples. Subsequently, for adaptive perturbation of ranking, the adaptive weight confusion ranking loss is presented to calculate the weights of distances between positive or negative samples and query samples. It endeavors to push positive samples away from query samples and bring negative samples closer, thereby interfering with the ranking of system. Notably, this method requires no additional hyperparameter tuning or extra data training, making it an adaptive attack strategy. Experimental results on large-scale datasets such as Market1501, CUHK03, and DukeMTMC demonstrate the effectiveness of our method in attacking ReID systems.

Federated Adversarial Domain Hallucination for Privacy-Preserving Domain Generalization

Domain generalization aims to reduce the vulnerability of deep neural networks in the out-of-domain distribution scenario. With the recent and increasing data privacy concerns, federated domain generalization, where multiple domains are distributed on different local clients, has become an important research problem and brings new challenges for learning domain-invariant information from separated domains. In this paper, we address the problem of federated domain generalization from the perspective of domain hallucination. We propose a novel federated domain hallucination learning framework, with no additional data exchange between clients other than model weights, based on the idea that a domain hallucination with enlarged prediction uncertainty for the global model is more likely to transform the samples into an unseen domain. These types of desired domain hallucinations are achieved by generating samples that maximize the entropy of the global model and minimize the cross-entropy of the local model, where the latter loss is further introduced to maintain the sample semantics. By training the local models with the learned domain hallucinations, the final model is expected to be more robust to unseen domain shifts. We perform extensive experiments on three object classification benchmarks and one medical image segmentation benchmark. The proposed method outperforms state-of-the-art methods on all the benchmarks, demonstrating its effectiveness.

A survey on vulnerability of deep neural networks to adversarial examples and defense approaches to deal with them

A survey on vulnerability of deep neural networks to adversarial examples and defense approaches to deal with them

Masking and purifying inputs for blocking textual adversarial attacks

The vulnerability of deep neural networks (DNNs) to adversarial attacks has attracted attention in many fields, and researchers have sought methods to improve the robustness of DNNs. Most existing methods are empirical defenses that can only cope with known attack environments and specific attack conditions and are likely to be broken by more powerful attacks. In contrast, certified defense methods provide theoretical proof of robustness bounds and provide worst-case adversarial robustness that can guarantee reliable robustness for the target model. To ensure the security of DNNs against adversarial attacks, it is crucial to develop certified defense methods. However, the currently available study has paid little attention to certified defenses, and the currently certified defense commonly relies on unrealistic prior knowledge and assumptions, which is limited in practice. In this paper, we propose a model-agnostic and attack-agnostic certified defense method that implements denoising and refactoring of input samples through the Makser and the Purifier. The Makser is rule-based and the Purifier is trained by self-supervised fine-tuning on the improved BERT-MLM, and the whole implementation is independent of any target models or attack methods. We have theoretically demonstrated that the condition for method satisfying certified robustness and obtained its robustness boundary through simulation experiments. Meanwhile, experiments on three datasets demonstrate that our proposed method has superior performance in balancing the prediction accuracy on clean examples and the robustness against adversarial attacks compared with six state-of-the-art defense methods.

QuanDA: GPU Accelerated Quantitative Deep Neural Network Analysis

Over the past years, numerous studies demonstrated the vulnerability of deep neural networks (DNNs) to make correct classifications in the presence of small noise. This motivated the formal analysis of DNNs to ensure they delineate acceptable behavior. However, in case the DNN’s behavior is unacceptable for the desired application, these qualitative approaches are ill-equipped to determine the precise degree to which the DNN behaves unacceptably. Towards this, we propose a novel quantitative DNN analysis framework, QuanDA, which does not only check if the DNN delineates certain behavior, but also provides the estimated probability of the DNN to delineate this particular behavior. Unlike the (few) available quantitative DNN analysis frameworks, QuanDA does not use any implicit assumptions on the probability distribution of the hidden nodes, which enables the framework to propagate close to real probability distributions of the hidden node values to each proceeding DNN layer. Furthermore, our framework leverages CUDA to parallelize the analysis, enabling high-speed GPU implementation for fast analysis. The applicability of the framework is demonstrated using the ACAS Xu benchmark, to provide reachability probability estimates for all network nodes. Moreover, this paper also provides potential applications of QuanDA for the analysis of the DNN safety properties.

Query-Efficient Generation of Adversarial Examples for Defensive DNNs via Multiobjective Optimization

Due to the inherent vulnerability of Deep Neural Networks (DNNs), Adversarial Example (AE) attack has become a serious threat to intelligent systems, e.g., the failure cause of an image classification system. Different to existing works, in this paper we are interested in the generation of AEs for DNNs with defensive mechanisms. To make the attack more practical, we exploit a query-based method to generate image AEs in a black-box attack setting. Considering that the generation of AEs is inherently a constrained optimization problem, this paper first formulates three objectives regarding to defensive DNNs, i.e, attack effectiveness, attack evasiveness and attack coverage. Then, this paper proposes a query-efficient AE attack based on Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) to address the perturbation optimization problem. To improve the efficiency of search and query, AE-specific operators including block-level and pixel-level crossovers, discrete perturbation mutation and direction-driven reproduction are designed within the GA-based search framework. In addition, predication-based adaptation of reproduction-related parameters is implemented to speed up the search convergence. PSO-based jumping process is further devised to avoid stuck in local optimum. Benchmark-based experiments evaluated the efficiency of our method, which can achieve an attack success rate of 100% with averagely 52.95% reduced queries in contrast to existing black-box attacks on non-defensive models. For defensive DNN models, our method can obtain top attack performance with the query reduction up to 70.92% comparing with the candidates.

ZeroGrad: Costless conscious remedies for catastrophic overfitting in the FGSM adversarial training

Vulnerability of deep neural networks to small adversarial examples has recently attracted a lot of attention. As a result, making models robust to small adversarial noises has been sought in many safety critical applications. Adversarial training through iterative projected gradient descent (PGD) has been established as one of the mainstream ideas to achieve this goal. However, PGD is computationally demanding and often prohibitive in the case of large datasets and models. For this reason, the single-step PGD, also known as the Fast Gradient Sign Method (FGSM), has recently gained interest in the field. Unfortunately, FGSM-training leads to a phenomenon called “catastrophic overfitting,” which is a sudden drop in the test adversarial accuracy under the PGD attack. In this paper, we propose new methods to prevent this failure mode of the FGSM-based attacks with almost no extra computational cost. The proposed methods are also backed up with theoretical insights into the causes of the catastrophic overfitting. Our intuition is that small input gradients play a key role in this phenomenon. The signs of such gradients are quite unstable and fragile from an epoch to the next, making the signed gradient method discontinuous along the training process. These instabilities introduce large weight updates by the stochastic gradient descent, and hence potentially cause overfitting. To mitigate this issue, we propose to simply identify such gradients and make them zero prior to taking the sign in the FGSM attack calculation that is used in the training. This remedy makes the training perturbations stable, while almost preserving the adversarial property of such perturbations. The idea while being simple and efficient, achieves competitive adversarial accuracy on various datasets and can be used as an affordable method to train robust deep neural networks2.

Sparse fooling images: Fooling machine perception through unrecognizable images

Fooling images are potential threats to deep neural networks (DNNs). These images cannot be recognized by humans as natural objects, e.g., dogs and cats. However, they are misclassified by DNNs as natural object classes with high confidence scores. Despite their original design concept, existing fooling images, if closely examined, can be seen to retain some features that are characteristic of the target objects. Hence, DNNs can react to these features. In this study, we evaluate whether fooling images with no characteristic pattern of natural objects, either locally or globally, can exist. As a minimal case, we introduce single-color images with a few pixels altered, called sparse fooling images (SFIs). We first prove that SFIs always exist under mild conditions for linear and nonlinear models and reveal that complex models are more likely to be vulnerable to SFI attacks. Using two SFI generation methods, we demonstrate that in deeper layers, SFIs have features similar to those of natural images. Therefore, they fool DNNs successfully. Among the other layers, we discover that the max-pooling layer causes vulnerability to SFIs. The defense against SFIs and transferability are also discussed. This study highlights a new vulnerability of DNNs by introducing a novel class of images that are distributed extremely far from natural images.

DRL: Dynamic rebalance learning for adversarial robustness of UAV with long-tailed distribution

Adversarial robustness has attracted extensive studies in various fields by increasing the interpretability of deep learning and enhancing the understanding of neural network models. In realistic scenarios such as UAV control system, imbalanced datasets are a consensus. Therefore, how to solve the adversarial robustness on imbalanced datasets is a more and more inescapable problem in UAV control system. There have been some works on adversarial robustness on imbalanced datasets, which bring us a deeper understanding of vulnerability of deep neural networks and the generation of adversarial examples. To adjust the classification plane after training, the long-tailed robustness framework is usually designed to be multi-stage, and different classifiers are used in different stages, which can improve the robustness through multi learning. The existing methods are hardly considered to effectively handle the long-tailed robustness problem in UAV control system. To explore the intrinsic features of long-tailed robustness, we propose a one-stage robustness framework. First, we study different classifiers and propose a general cosine classifier. By changing the general cosine classifier adaptively, the model obtains a more robust classification. Then, we analyze the scalability of the focal loss and design a focal-margin loss. Finally, we design a category focus mobile learning strategy, obtained more robust features by changing the learning emphasis with this strategy. From this, we design a simple and efficient one-stage dynamic adversarial robustness method DRL under long-tailed distribution, which consists of an adaptive cosine classifier and a focal-margin loss under long-tailed mobile learning. The extended experiments demonstrate the superiority of our approach over other state-of-the-art methods, and the effectiveness of the designed module. This method can effectively solve the long-tailed robustness problem on UAV control system and other terminals.

Sensitive region-aware black-box adversarial attacks

Recent research on adversarial attacks has highlighted the vulnerability of deep neural networks (DNNs) to perturbations. While existing studies generate adversarial perturbations spread across the entire image, these global perturbations may be visible to human eyes, reducing their effectiveness in real-world scenarios. To alleviate this issue, recent works propose to modify a limited number of input pixels to implement adversarial attacks. However, these approaches still have limitations in terms of both imperceptibility and efficiency.This paper proposes a novel plug-in framework called Sensitive Region-Aware Attack (SRA) to generate soft-label black-box adversarial examples using the sensitivity map and evolution strategies. First, a transferable black-box sensitivity map generation approach is proposed for identifying the sensitive regions of input images. To perform SRA with a limited amount of perturbed pixels, a dynamic l0 and l∞ adjustment strategy is introduced. Furthermore, an adaptive evolution strategy is employed to optimize the selection of generated sensitive regions, allowing for the execution of effective and imperceptible attacks. Experimental results demonstrate that our SRA achieves an imperceptible soft-label black-box attack with a 96.43% success rate using less than 20% of the image pixels on ImageNet and a 100% success rate using 30% of the image pixels on CIFAR-10.

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

Deep cross-modal hashing retrieval models inherit the vulnerability of deep neural networks. They are vulnerable to adversarial attacks, especially for the form of subtle perturbations to the inputs. Although many adversarial attack methods have been proposed to handle the robustness of hashing retrieval models, they still suffer from two problems: (1) Most of them are based on the white-box settings, which is usually unrealistic in practical application. (2) Iterative optimization for the generation of adversarial examples in them results in heavy computation. To address these problems, we propose an Efficient Query-based Black-Box Attack (EQB 2 A) against deep cross-modal hashing retrieval, which can efficiently generate adversarial examples for the black-box attack. Specifically, by sending a few query requests to the attacked retrieval system, the cross-modal retrieval model stealing is performed based on the neighbor relationship between the retrieved results and the query, thus obtaining the knockoffs to substitute the attacked system. A multi-modal knockoffs-driven adversarial generation is proposed to achieve efficient adversarial example generation. While the entire network training converges, EQB 2 A can efficiently generate adversarial examples by forward-propagation with only given benign images. Experiments show that EQB 2 A achieves superior attacking performance under the black-box setting.

Adversarial attack method against image classification based on haze perturbation

The adversarial attack is a cutting-edge technique used to study the vulnerability of deep neural networks (DNNs). However, most existing studies focus on the additive perturbation-based attack, which cannot represent real-world corruption and limit their applications. In particular, haze is a common natural phenomenon that significantly corrupts an image, which inevitably poses a potential threat to deep models. In this work, for the first attempt, we study the effects of haze on DNNs from the perspective of adversarial attacks and propose two adversarial haze attack methods. We first propose the optimization-based adversarial haze attack (OAdvHaze) that optimizes the parameters of the atmospheric scattering model with the guidance of a DNN to synthesize a hazy image, which leads to a high attack success rate. To achieve a more efficient attack, we further propose a predictive adversarial haze attack (PAdvHaze) employing a DNN to predict the hazing parameters through a one-step way. To validate the effectiveness of both methods, we conducted extensive experiments on two publicly available datasets, i.e., ILSVRC 2012 and NIPS 2017. OAdvHaze and PAdvHaze achieve comparable attack success rates and transferability to state-of-the-art attacks. This work would contribute to the evaluation and enhancement of the robustness of DNNs against haze perturbation that may happen in the real world.

Adversarial Examples Protect Your Privacy on Speech Enhancement System

Speech is easily leaked imperceptibly. When people use their phones, the personal voice assistant is constantly listening and waiting to be activated. Private content in speech may be maliciously extracted through automatic speech recognition (ASR) technology by some applications on phone devices. To guarantee that the recognized speech content is accurate, speech enhancement technology is used to denoise the input speech. Speech enhancement technology has developed rapidly along with deep neural networks (DNNs), but adversarial examples can cause DNNs to fail. Considering that the vulnerability of DNN can be used to protect the privacy in speech. In this work, we propose an adversarial method to degrade speech enhancement systems, which can prevent the malicious extraction of private information in speech. Experimental results show that the generated enhanced adversarial examples can be removed most content of the target speech or replaced with target speech content by speech enhancement. The word error rate (WER) between the enhanced original example and enhanced adversarial example recognition result can reach 89.0%. WER of target attack between enhanced adversarial example and target example is low at 33.75%. The adversarial perturbation in the adversarial example can bring much more change than itself. The rate of difference between two enhanced examples and adversarial perturbation can reach more than 1.4430. Meanwhile, the transferability between different speech enhancement models is also investigated. The low transferability of the method can be used to ensure the content in the adversarial example is not damaged, the useful information can be extracted by the friendly ASR. This work can prevent the malicious extraction of speech.

Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image

Adversarial examples easily mislead vision systems based on deep neural networks (DNNs) trained with softmax cross entropy (SCE) loss. The vulnerability of DNN comes from the fact that SCE drives DNNs to fit on the training examples, whereas the resultant feature distributions between the training and adversarial examples are unfortunately misaligned. Several state-of-the-art methods start from improving the inter-class separability of training examples by modifying loss functions, where we argue that the adversarial examples are ignored, thus resulting in a limited robustness to adversarial attacks. In this paper, we exploited the inference region, which inspired us to apply margin-like inference information to SCE, resulting in a novel inference-softmax cross entropy (I-SCE) loss, which is intuitively appealing and interpretable. The inference information guarantees that it is difficult for neural networks to cross the decision boundary under an adversarial attack, and guarantees both the inter-class separability and the improved generalization to adversarial examples, which was further demonstrated and proved under the min-max framework. Extensive experiments show that the DNN models trained with the proposed I-SCE loss achieve a superior performance and robustness over the state-of-the-arts under different prevalent adversarial attacks; for example, the accuracy of I-SCE is 63% higher than SCE under the PGD50un attack on the MNIST dataset. These experiments also show that the inference region can effectively solve the misaligned distribution.

Visually imperceptible adversarial patch attacks

The vulnerability of deep neural networks (DNNs) to adversarial examples has attracted more attention. Many algorithms have been proposed to craft powerful adversarial examples. However, most of these algorithms modified the global or local region of pixels without taking network explanations into account. Hence, the perturbations are redundant, which are easily detected by human eyes. In this paper, we propose a novel method to generate local region perturbations. The main idea is to find a contributing feature region (CFR) of an image by simulating the human attention mechanism and then add perturbations to CFR. Furthermore, a soft mask matrix is designed on the basis of an activation map to finely represent the contributions of each pixel in CFR. With this soft mask, we develop a new loss function with inverse temperature to search for optimal perturbations in CFR. Due to the network explanations, the perturbations added to CFR are more effective than those added to other regions. Extensive experiments conducted on CIFAR-10 and ILSVRC2012 demonstrate the effectiveness of the proposed method, including attack success rate, imperceptibility, and transferability.

Textual adversarial attacks by exchanging text‐self words

Adversarial attacks expose the vulnerability of deep neural networks. Compared to image adversarial attacks, textual adversarial attacks are more challenging due to the discrete nature of texts. Recent synonym-based methods achieve the current state-of-the-art results. However, these methods introduce new words against the original text, leading to that humans easily perceive the difference between the adversarial example and the original text. Motivated by the fact that humans are usually unaware of chaotic word order in some cases, we propose exchange-attack (EA), a concise and effective word-level textual adversarial attack model. Specifically, the EA model generates adversarial examples by exchanging words of the original text itself according to the contributions that these words make regarding classification results. Intuitively, the smaller the distance between the two exchanged words, the more difficult the chaotic word order to be perceived by humans. We thus take the word distance into consideration when generating the chaotic word orders. Extensive experiments on several text classification data sets show that the EA model consistently outperforms the selected baselines in terms of averaged after-attack accuracy, modification rate, query number, and semantic similarity. And human evaluation results reveal that humans difficultly perceive the adversarial examples generated by the EA model. In addition, quantitative and qualitative analyses further validate the effectiveness of the EA model, including that the generated adversarial examples are grammatically correct and semantically preserved.