Feature fusion network for long-tailed visual recognition

Abstract

Deep learning has achieved remarkable success in recent years; however, deep learning methods face significant challenges on long-tailed datasets, which are prevalent in real-world scenarios. In a long-tailed dataset, there are many more samples in the head classes than in the tail classes, and this class imbalance makes it difficult to learn a good feature representation for both head and tail classes simultaneously, particularly when using a single-stage method. Although the existing two-stage methods can alleviate the problem of single-stage methods not performing well on the tail classes by classifier retraining in the second stage, this does not resolve the problem of insufficient learning of head and tail features. Thus, in this paper, we propose a two-stage feature fusion network (FFN). The proposed FFN addresses this issue using one network for the head classes and another network for the tail classes, each of which is trained with a different loss function. This allows the feature learning module to effectively distinguish between the head and tail classes in the embedding space. The classifier learning module fuses the features obtained from the feature learning module, and the classifier is fine-tuned to classify the input images. Different from traditional two-stage methods, the proposed utilizes different loss functions for the head and tail classes; thus, the classifier can achieve balanced results between the head and tail classes. We conduct extensive experiments on three benchmark datasets comparing the proposed FFN with six state-of-the-art methods including two baseline methods, the experimental results demonstrate that the FFN achieves significant improvement on all three benchmark datasets. The code is publicly available at https://github.com/zxsong999/Feature-Fusion-Network.pytorch.