Few-shot image classification with composite rotation based self-supervised auxiliary task

Abstract

Many real-life problem settings have classes of data with very few examples for training. Deep learning networks do not perform well for such few-shot classes. In order to perform well in this setting, the networks should learn to extract highly discriminative and generic features. In this paper, we propose to use a composite rotation based self-supervised auxiliary task to improve the representation learning of the network so that it can extract such discriminative features. Our proposed composite rotation based auxiliary task rotates the image at two levels, i.e., it rotates patches inside the image (inner rotation) and also rotates the whole image (outer rotation), and assigns one out of 16 rotation classes to the transformed image. We jointly train the network on the main image classification task and the composite rotation based auxiliary task. This helps the network to learn to extract more generic and discriminative features, which in turn helps to improve its few-shot classification performance. Additionally, during the few-shot testing phase, we consolidate the predictions for images obtained by applying different composite rotation transformations to the same query image to improve the final prediction further. We perform experiments on several few-shot benchmark datasets and empirically show the efficacy of our method for various problem settings. We experimentally show that our method improves the performance of deep learning models on the few-shot classification, fine-grained few-shot classification, cross-domain few-shot classification, and transductive few-shot classification settings. We also experimentally show that models trained using our approach perform better than the baseline even when the query examples in the episode are not aligned with the support examples in the episode. We perform extensive ablation experiments to validate the different components of our approach. We also analyze the effect of our approach on the ability of the network to focus on the discriminative regions of the image.