Learning a robust foundation model against clean-label data poisoning attacks at downstream tasks

Abstract

In the transfer learning paradigm, models that are pre-trained on large datasets are used as the foundation models for various downstream tasks. However, this paradigm exposes downstream practitioners to data poisoning threats, as attackers can inject malicious samples into the re-training datasets to manipulate the behavior of models in downstream tasks. In this work, we propose a defense strategy that significantly reduces the success rate of various data poisoning attacks in downstream tasks. Our defense aims to pre-train a robust foundation model by reducing adversarial feature distance and increasing inter-class feature distance. Experiments demonstrate the excellent defense performance of the proposed strategy towards state-of-the-art clean-label poisoning attacks in the transfer learning scenario.