Backdoor Attacks Research Articles

From Toxic to Trustworthy: Using Self-Distillation and Semi-supervised Methods to Refine Neural Networks

Despite the tremendous success of deep neural networks (DNNs) across various fields, their susceptibility to potential backdoor attacks seriously threatens their application security, particularly in safety-critical or security-sensitive ones. Given this growing threat, there is a pressing need for research into purging backdoors from DNNs. However, prior efforts on erasing backdoor triggers not only failed to withstand increasingly powerful attacks but also resulted in reduced model performance. In this paper, we propose From Toxic to Trustworthy (FTT), an innovative approach to eliminate backdoor triggers while simultaneously enhancing model accuracy. Following the stringent and practical assumption of limited availability of clean data, we introduce a self-attention distillation (SAD) method to remove the backdoor by aligning the shallow and deep parts of the network. Furthermore, we first devise a semi-supervised learning (SSL) method that leverages ubiquitous and available poisoned data to further purify backdoors and improve accuracy. Extensive experiments on various attacks and models have shown that our FTT can reduce the attack success rate from 97% to 1% and improve the accuracy of 4% on average, demonstrating its effectiveness in mitigating backdoor attacks and improving model performance. Compared to state-of-the-art (SOTA) methods, our FTT can reduce the attack success rate by 2 times and improve the accuracy by 5%, shedding light on backdoor cleansing.

BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning

Backdoor attacks in reinforcement learning (RL) have previously employed intense attack strategies to ensure attack success. However, these methods suffer from high attack costs and increased detectability. In this work, we propose a novel approach, BadRL, which focuses on conducting highly sparse backdoor poisoning efforts during training and testing while maintaining successful attacks. Our algorithm, BadRL, strategically chooses state observations with high attack values to inject triggers during training and testing, thereby reducing the chances of detection. In contrast to the previous methods that utilize sample-agnostic trigger patterns, BadRL dynamically generates distinct trigger patterns based on targeted state observations, thereby enhancing its effectiveness. Theoretical analysis shows that the targeted backdoor attack is always viable and remains stealthy under specific assumptions. Empirical results on various classic RL tasks illustrate that BadRL can substantially degrade the performance of a victim agent with minimal poisoning efforts (0.003% of total training steps) during training and infrequent attacks during testing. Code is available at: https://github.com/7777777cc/code.

Progressive Poisoned Data Isolation for Training-Time Backdoor Defense

Deep Neural Networks (DNN) are susceptible to backdoor attacks where malicious attackers manipulate the model's predictions via data poisoning. It is hence imperative to develop a strategy for training a clean model using a potentially poisoned dataset. Previous training-time defense mechanisms typically employ an one-time isolation process, often leading to suboptimal isolation outcomes. In this study, we present a novel and efficacious defense method, termed Progressive Isolation of Poisoned Data (PIPD), that progressively isolates poisoned data to enhance the isolation accuracy and mitigate the risk of benign samples being misclassified as poisoned ones. Once the poisoned portion of the dataset has been identified, we introduce a selective training process to train a clean model. Through the implementation of these techniques, we ensure that the trained model manifests a significantly diminished attack success rate against the poisoned data. Extensive experiments on multiple benchmark datasets and DNN models, assessed against nine state-of-the-art backdoor attacks, demonstrate the superior performance of our PIPD method for backdoor defense. For instance, our PIPD achieves an average True Positive Rate (TPR) of 99.95% and an average False Positive Rate (FPR) of 0.06% for diverse attacks over CIFAR-10 dataset, markedly surpassing the performance of state-of-the-art methods. The code is available at https://github.com/RorschachChen/PIPD.git.

Backdoor Attacks via Machine Unlearning

As a new paradigm to erase data from a model and protect user privacy, machine unlearning has drawn significant attention. However, existing studies on machine unlearning mainly focus on its effectiveness and efficiency, neglecting the security challenges introduced by this technique. In this paper, we aim to bridge this gap and study the possibility of conducting malicious attacks leveraging machine unlearning. Specifically, we consider the backdoor attack via machine unlearning, where an attacker seeks to inject a backdoor in the unlearned model by submitting malicious unlearning requests, so that the prediction made by the unlearned model can be changed when a particular trigger presents. In our study, we propose two attack approaches. The first attack approach does not require the attacker to poison any training data of the model. The attacker can achieve the attack goal only by requesting to unlearn a small subset of his contributed training data. The second approach allows the attacker to poison a few training instances with a pre-defined trigger upfront, and then activate the attack via submitting a malicious unlearning request. Both attack approaches are proposed with the goal of maximizing the attack utility while ensuring attack stealthiness. The effectiveness of the proposed attacks is demonstrated with different machine unlearning algorithms as well as different models on different datasets.

Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift

Diffusion models (DM) have become state-of-the-art generative models because of their capability of generating high-quality images from noises without adversarial training. However, they are vulnerable to backdoor attacks as reported by recent studies. When a data input (e.g., some Gaussian noise) is stamped with a trigger (e.g., a white patch), the backdoored model always generates the target image (e.g., an improper photo). However, effective defense strategies to mitigate backdoors from DMs are underexplored. To bridge this gap, we propose the first backdoor detection and removal framework for DMs. We evaluate our framework Elijah on over hundreds of DMs of 3 types including DDPM, NCSN and LDM, with 13 samplers against 3 existing backdoor attacks. Extensive experiments show that our approach can have close to 100% detection accuracy and reduce the backdoor effects to close to zero without significantly sacrificing the model utility.

DataElixir: Purifying Poisoned Dataset to Mitigate Backdoor Attacks via Diffusion Models

Dataset sanitization is a widely adopted proactive defense against poisoning-based backdoor attacks, aimed at filtering out and removing poisoned samples from training datasets. However, existing methods have shown limited efficacy in countering the ever-evolving trigger functions, and often leading to considerable degradation of benign accuracy. In this paper, we propose DataElixir, a novel sanitization approach tailored to purify poisoned datasets. We leverage diffusion models to eliminate trigger features and restore benign features, thereby turning the poisoned samples into benign ones. Specifically, with multiple iterations of the forward and reverse process, we extract intermediary images and their predicted labels for each sample in the original dataset. Then, we identify anomalous samples in terms of the presence of label transition of the intermediary images, detect the target label by quantifying distribution discrepancy, select their purified images considering pixel and feature distance, and determine their ground-truth labels by training a benign model. Experiments conducted on 9 popular attacks demonstrates that DataElixir effectively mitigates various complex attacks while exerting minimal impact on benign accuracy, surpassing the performance of baseline defense methods.

BadSAM: Exploring Security Vulnerabilities of SAM via Backdoor Attacks (Student Abstract)

Image segmentation is foundational to computer vision applications, and the Segment Anything Model (SAM) has become a leading base model for these tasks. However, SAM falters in specialized downstream challenges, leading to various customized SAM models. We introduce BadSAM, a backdoor attack tailored for SAM, revealing that customized models can harbor malicious behaviors. Using the CAMO dataset, we confirm BadSAM's efficacy and identify SAM vulnerabilities. This study paves the way for the development of more secure and customizable vision foundation models.

PointCVaR: Risk-Optimized Outlier Removal for Robust 3D Point Cloud Classification

With the growth of 3D sensing technology, the deep learning system for 3D point clouds has become increasingly important, especially in applications such as autonomous vehicles where safety is a primary concern. However, there are growing concerns about the reliability of these systems when they encounter noisy point clouds, either occurring naturally or introduced with malicious intent. This paper highlights the challenges of point cloud classification posed by various forms of noise, from simple background noise to malicious adversarial/backdoor attacks that can intentionally skew model predictions. While there's an urgent need for optimized point cloud denoising, current point outlier removal approaches, an essential step for denoising, rely heavily on handcrafted strategies and are not adapted for higher-level tasks, such as classification. To address this issue, we introduce an innovative point outlier cleansing method that harnesses the power of downstream classification models. Using gradient-based attribution analysis, we define a novel concept: point risk. Drawing inspiration from tail risk minimization in finance, we recast the outlier removal process as an optimization problem, named PointCVaR. Extensive experiments show that our proposed technique not only robustly filters diverse point cloud outliers but also consistently and significantly enhances existing robust methods for point cloud classification. A notable feature of our approach is its effectiveness in defending against the latest threat of backdoor attacks in point clouds.

Bidirectional Contrastive Split Learning for Visual Question Answering

Visual Question Answering (VQA) based on multi-modal data facilitates real-life applications such as home robots and medical diagnoses. One significant challenge is to devise a robust decentralized learning framework for various client models where centralized data collection is refrained due to confidentiality concerns. This work aims to tackle privacy-preserving VQA by decoupling a multi-modal model into representation modules and a contrastive module, leveraging inter-module gradients sharing and inter-client weight sharing. To this end, we propose Bidirectional Contrastive Split Learning (BiCSL) to train a global multi-modal model on the entire data distribution of decentralized clients. We employ the contrastive loss that enables a more efficient self-supervised learning of decentralized modules. Comprehensive experiments are conducted on the VQA-v2 dataset based on five SOTA VQA models, demonstrating the effectiveness of the proposed method. Furthermore, we inspect BiCSL's robustness against a dual-key backdoor attack on VQA. Consequently, BiCSL shows significantly enhanced resilience when exposed to the multi-modal adversarial attack compared to the centralized learning method, which provides a promising approach to decentralized multi-modal learning.

Enhanced Coalescence Backdoor Attack Against DNN Based on Pixel Gradient

Deep learning has been widely used in many applications such as face recognition, autonomous driving, etc. However, deep learning models are vulnerable to various adversarial attacks, among which backdoor attack is emerging recently. Most of the existing backdoor attacks use the same trigger or the same trigger generation approach to generate the poisoned samples in the training and testing sets, which is also commonly adopted by many backdoor defense strategies. In this paper, we develop an enhanced backdoor attack (EBA) that aims to reveal the potential flaws of existing backdoor defense methods. We use a low-intensity trigger to embed the backdoor, while a high-intensity trigger to activate it. Furthermore, we propose an enhanced coalescence backdoor attack (ECBA) where multiple low-intensity incipient triggers are designed to train the backdoor model, and then, all incipient triggers are gathered on one sample and enhanced to launch the attack. Experiment results on three popular datasets show that our proposed attacks can achieve high attack success rates while maintaining the model classification accuracy of benign samples. Meanwhile, by hiding the incipient poisoned samples and preventing them from activating the backdoor, the proposed attack exhibits significant stealth and the ability to evade mainstream defense methods during the model training phase.

RoPE: Defending against backdoor attacks in federated learning systems

Federated learning (FL) is vulnerable to backdoor attacks, which aim to cause the misclassification on samples with a specific backdoor. Most existing algorithms are restricted by some conditions, such as the data distribution across the joint clients, the number of attackers and some auxiliary information, thereby being limited in the practical FL. In this paper, we propose RoPE containing three parts: using principal component analysis to extract the important features of model gradients; leveraging expectation–maximization to separate malicious clients from benign ones in accordance to the important features; removing the potential malicious gradients within the selected cluster with Isolated Forest. RoPE requires no restricted assumptions during the training process. We evaluate the performance of RoPE on three image classification tasks under non-independent and identically distributed scenario (non-iid) against centralized backdoor attacks with various ratios of attackers and distributed backdoor attacks, respectively. We also evaluate the performance of RoPE against other backdoor attack scenarios, including independent and identically distributed (iid) scheme, elaborately designed attack schemes. The results show that RoPE can defend against these backdoor attacks and outperform the existing algorithms. In addition, we also explore the impact of different numbers of features on RoPE’s performance and conduct ablation experiments.

MITDBA: Mitigating Dynamic Backdoor Attacks in Federated Learning for IoT Applications

MITDBA: Mitigating Dynamic Backdoor Attacks in Federated Learning for IoT Applications

FDNet: Imperceptible backdoor attacks via frequency domain steganography and negative sampling

Backdoor attacks against Deep Neural Networks (DNNs) have surfaced as a substantial and concerning security challenge. These backdoor vulnerabilities in DNNs can be introduced by third-party sources through maliciously manipulated training data. Existing backdoor attacks are primarily built on perturbation trigger patterns in the spatial domain, which makes practical deployment arduous due to the ease of detection by inspectors. Moreover, shortcut learning renders the backdoor network less robust against defense methods. This work advances an effective and adaptable approach to backdoor attacks, situated in the frequency domain. This methodology involves the incorporation of specific natural perturbations within the frequency domain of images. Remarkably, these introduced triggers yield minimal alterations in the image’s semantic content, rendering them nearly imperceptible to human observers. To evade detection by machine-based defenders, we introduce a new training paradigm that incorporates negative sampling techniques. This approach compels the neural network to learn richer differences as trigger patterns. We evaluate our attacks on popular convolutional neural networks, visual transformers, and MLP-Mixer models as well as four standard datasets including MNIST, CIFAR-10, GTSRB, and ImageNet. Experimental results demonstrate that the trained networks can be successfully injected with backdoors. Our attack methods exhibit remarkable efficacy, achieving high attack success rates in both All-to-one (near 100% on all datasets) and All-to-all (over 90% except on ImageNet) scenarios and also demonstrate robustness against contemporary state-of-the-art defense mechanisms. Furthermore, our work reveals that DNNs can capture discrepancies in the frequency components of images that are barely perceptible to humans.

Backdoor Two-Stream Video Models on Federated Learning

Video models on federated learning (FL) enable continual learning of the involved models for video tasks on end-user devices while protecting the privacy of end-user data. As a result, the security issues on FL, e.g., the backdoor attacks on FL and their defense have increasingly becoming the domains of extensive research in recent years. The backdoor attacks on FL are a class of poisoning attacks, in which an attacker, as one of the training participants, submits poisoned parameters and thus injects the backdoor into the global model after aggregation. Existing backdoor attacks against videos based on FL only poison RGB frames, which makes that the attack could be easily mitigated by two-stream model neutralization. Therefore, it is a big challenge to manipulate the most advanced two-stream video model with a high success rate by poisoning only a small proportion of training data in the framework of FL. In this paper, a new backdoor attack scheme incorporating the rich spatial and temporal structures of video data is proposed, which injects the backdoor triggers into both the optical flow and RGB frames of video data through multiple rounds of model aggregations. In addition, the adversarial attack is utilized on the RGB frames to further boost the robustness of the attacks. Extensive experiments on real-world datasets verify that our methods outperform the state-of-the-art backdoor attacks and show better performance in terms of stealthiness and persistence.

Federated learning algorithm based on matrix mapping for data privacy over edge computing

PurposeThis paper aims to provide the security and privacy for Byzantine clients from different types of attacks.Design/methodology/approachIn this paper, the authors use Federated Learning Algorithm Based On Matrix Mapping For Data Privacy over Edge Computing.FindingsBy using Softmax layer probability distribution for model byzantine tolerance can be increased from 40% to 45% in the blocking-convergence attack, and the edge backdoor attack can be stopped.Originality/valueBy using Softmax layer probability distribution for model the results of the tests, the aggregation method can protect at least 30% of Byzantine clients.

Incremental Learning, Incremental Backdoor Threats

Class incremental learning from a pre-trained DNN model is gaining lots of popularity. Unfortunately, the pre-trained model also introduces a new attack vector, which enables an adversary to inject a backdoor into it and further compromise the downstream models learned from it. Prior works proposed backdoor attacks against the pre-trained models in the transfer learning scenario. However, they become less effective when the adversary does not have the knowledge of the downstream tasks or new data, which is more practical and considered in this paper. To this end, we design the first latent backdoor attacks against incremental learning. We propose two novel techniques, which can effectively and stealthily embed a backdoor into the pre-trained model. Such backdoor can only be activated when the pre-trained model is extended to a downstream model with incremental learning. It has a very high attack success rate, and is able to bypass existing backdoor detection approaches. Extensive experiments confirm the effectiveness of our attacks over different datasets and incremental learning methods, as well as strong robustness against state-of-the-art backdoor defense mechanisms including <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Neural Cleanse</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fine-Pruning</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">STRIP</i> .

Critical Path-Based Backdoor Detection for Deep Neural Networks.

Backdoor attack to deep neural networks (DNNs) is among the predominant approaches to bring great threats into artificial intelligence. The existing methods to detect backdoor attacks focus on the perspective of distributions in DNNs, however, limited by its ability of generalization across DNN models. In this article, a critical-path-based backdoor detector (CPBD) is proposed, which approaches to detect backdoor attacks via DNN's interpretability. CPBD is designed to efficiently discover the characteristics of backdoors, which distinguish the critical paths in the attacked DNNs. To deal with the intractably large number of neurons, we propose to simplify the neurons, and the preserved key nodes are integrated into a set of critical paths. Thus, a DNN model can be formulated as a combination of several critical paths. Afterward, the detection of backdoors is performed based on the analysis of critical paths corresponding to different classes. Then, combining all the above steps, the CPBD algorithm is integrated to present the results in a standard and systematic manner. In addition, CPBD is able to locate neurons associated with malicious triggers, the combination of which is named as trigger propagation path. Extensive experiments are conducted, which testify the efficiency of the proposed method on multiple DNNs and different trigger sizes.

Backdoor Attacks on Graph Neural Networks Trained with Data Augmentation

Backdoor Attacks on Graph Neural Networks Trained with Data Augmentation

PerVK: A Robust Personalized Federated Framework to Defend Against Backdoor Attacks for IoT Applications

PerVK: A Robust Personalized Federated Framework to Defend Against Backdoor Attacks for IoT Applications

Untargeted Backdoor Attack Against Deep Neural Networks With Imperceptible Trigger

Untargeted Backdoor Attack Against Deep Neural Networks With Imperceptible Trigger