Towards transferable adversarial attacks on vision transformers for image classification

Abstract

The deployment of high-performance Vision Transformer (ViT) models has garnered attention from both industry and academia. However, their vulnerability to adversarial examples highlights security risks for scenarios such as intelligent surveillance, autonomous driving, and fintech regulation. As a black-box attack technique, transfer attacks leverage a surrogate model to generate transferable adversarial examples to attack a target victim model, which mainly focuses on a forward (input diversification) and a backward (gradient modification) approach. However, both approaches are currently implemented straightforwardly and limit the transferability of surrogate models. In this paper, we propose a Forward-Backward Transferable Adversarial Attack framework (FBTA) that can generate highly transferable adversarial examples against different models by fully leveraging ViT’s distinctive intermediate layer structures. In the forward inference process of FBTA, we propose a Dropout-based Transferable Attack (DTA) approach to diversify the intermediate states of ViT models, simulating an ensemble learning effect; in the backward process, a Backpropagation Gradient Clipping (BGC) method is designed to refine the gradients within intermediate layers of ViT models intricately. Extensive experiments on state-of-the-art ViTs and robust CNNs demonstrate that our FBTA framework achieves an average performance improvement of 2.79% compared to state-of-the-art transfer-based attacks, offering insights for the comprehension and defense against transfer attacks.