Energy-based learning for open-set classification in remote sensing imagery

Abstract

ABSTRACT Scene classification in remote sensing imagery is generally addressed from a closed set-setting perspective where the training and testing domains share the same land-cover classes. In practice, we may face situations where test images can belong to new land-cover classes unseen during the training phase. Yet, the classifier will wrongly assign them to one of these known training classes. This calls for the development of specific open-set methods with unknown image detection ability. In this paper, we propose, an end-to-end learning approach based on vision transformers. We use energy-based learning to jointly model the class labels and data distribution by reinterpreting the logits of the token classification head of the transformer to learn the density of the training data. This trick allows the network to act as a generative model while retaining its discriminative power. In the test phase, we identify images with low log-likelihood scores as unknown and discard them from classification. Experiments on three remote sensing scene datasets confirm the promising capability of the proposed open-set classification model.