Adaprompt: Prompt Tuning with Adaptive Neighbours for Generalized Category Discovery

Abstract

Translate Article

In this work, we address the challenging task of generalized category discovery, where the goal is to correctly classify unlabelled objects from previously seen classes, while also categorizing instances of completely new, unseen classes. Inspired by the remarkable success of prompt tuning of Vision Transformer models for this task, we propose two novel modifications to further improve their effectiveness. First, we propose to simultaneously perform parametric classification and representation learning with prompt tuning in an end-to-end learning framework, instead of a two-stage process of representation learning followed by clustering. Second, we introduce an adaptive neighborhood strategy for positive mining and contrastive learning, eliminating the need for large memory banks for affinity learning. We demonstrate the effectiveness and efficiency of our approach in comparison with the state-of-the-art methods on four benchmark datasets.