Dual aggregated feature pyramid network for multi label classification

Abstract

• Feature level aggregation for multi-label classification. • Classifier level aggregation with the one-vs-all strategy. • End-to-end learning for combined feature and classifier aggregation network. • Evaluation on various experiments in comparison with state-of-the-art methods. • Ablation study to validate both aggregation methods. While many deep convolutional neural networks show promising performance in various classification tasks, multiple objects appearing in very different sizes, shapes, and appearances cause difficulty in multi-label classification using conventional neural networks. In this paper, we introduce a dual aggregated network on pyramidal convolutional features for multi-label classification. The proposed method includes both feature- and classifier-level aggregation to learn discriminant multi-scale information of various target objects in the image. First, the feature-level aggregation collects the convolutional activation maps from the multi-scale pyramid network, and then it densely pools them to take localized features of each object. We elaborately design the feature aggregation method so that the responses from the objects with different sizes, aspect ratios, and shapes are properly reflected the aggregated activation map. Unlike conventional methods, this process does not require the region proposal step, which reduces the computational burden significantly. Second, we introduce the classifier level aggregation algorithm for integrating the multi-object classifier modules. To maximize the discrimination power of each class, we train one-vs-all classifiers for individual classes using the class-wise loss function. For each test image, the scores from the class-wise classifiers are aggregated to get the final multi-label classification result. By combining the above feature- and classifier-level aggregation methods, our network can be trained in an end-to-end fashion, which is not possible for the conventional multi-label classification algorithms using region proposals. Extensive evaluations on PASCAL VOC 2007 and PASCAL VOC 2012 demonstrate that the proposed algorithm outperforms the state-of-the-art methods.