Boosting fine-tuning via Conditional Online Knowledge Transfer

Abstract

Fine-tuning is an effective technique to enhance network performance in scenarios with limited labeled data. To achieve this, recent methods exploit the knowledge mined in the source model (e.g., feature maps) to construct an extra regularization signal (RS), collaboratively supervising the target model along with target labels. However, these RSs are generated independently from the target information or are generated from the rough assistance of the target information, resulting in biased supervision different from the target task. In this paper, we propose a Conditional Online Knowledge Transfer (COKT) framework that finely utilizes the target information to construct robust and target-related RS. Specifically, we train a target-dominant RS branch that online supervises the target model in a knowledge distillation manner. The target information dominates the RS branch from three aspects: sample-wise conditional attention, residual feature fusion, and target task loss. With such a target-oriented framework, we can effectively exploit target-related prior knowledge of the source model. Extensive experiments demonstrate that COKT significantly outperforms the fine-tuning baselines, especially for dissimilar target tasks and small datasets. Moreover, different from most of the fine-tuning methods that are restricted to the vanilla fine-tuning scenario, COKT can be easily extended to cross-model and multi-model fine-tuning scenarios.