Image-level classification by hierarchical structure learning with visual and semantic similarities

Abstract

Image classification methods often use class-level information without considering the distinctive character of each image. Images of the same class may have varied appearances. Besides, visually similar images may not be semantically correlated. To solve these problems, in this paper, we propose a novel image classification method by automatically learning the image-level hierarchical structure (ILHS) using both visual and semantic similarities. We try to generate new representations by exploring both visual and semantic similarities of images. Images are clustered hierarchically to explore their correlations. We then use them for image representations. The diversity of image classes within each cluster is used to re-weight visual similarities. The re-weighted similarities are aggregated to generate new image representations. We conduct image classification experiments on the Caltech-256 dataset, the PASCAL VOC 2007 dataset and the PASCAL VOC 2012 dataset. Experimental results demonstrate the effectiveness of the proposed method.