Towards trustworthy dataset distillation

Abstract

Efficiency and trustworthiness are two eternal pursuits when applying deep learning in practical scenarios. Considering efficiency, dataset distillation (DD) endeavors to reduce training costs by distilling large datasets into tiny ones. However, existing methods concentrate on in-distribution (InD) classification, disregarding out-of-distribution (OOD) samples. On the other hand, OOD detection aims to enhance models’ trustworthiness, which is inefficiently achieved in full-data settings. For the first time, we consider both issues and propose a novel paradigm called Trustworthy Dataset Distillation (TrustDD). By distilling both InD samples and outliers, the condensed datasets are capable of training models competent in both InD classification and OOD detection. To alleviate the requirement of real outlier data, we further propose to corrupt InD samples to generate pseudo-outliers for TrustDD, namely Pseudo-Outlier Exposure. Comprehensive experiments demonstrate the effectiveness of TrustDD. TrustDD is more trustworthy and applicable to real open-world scenarios. Our code is available at https://github.com/mashijie1028/TrustDD.