Rare Class Oriented Scene Labeling Using CNN Incorporated Label Transfer

Abstract

In natural scene images, rare class objects have low occurrence frequencies and limited spatial coverage, and they may be easily ignored during scene labeling. However, rare class objects are often more important to semantic labeling and image understanding compared to background areas. In this work, we present a rare class-oriented scene labeling framework (RCSL) that involves two new techniques pertaining to rare classes. First, scene assisted rare class retrieval is introduced in label transfer that is intended to enrich the retrieval set with scene-relevant rare classes. Second, a complementary rare class balanced CNN is incorporated to address the unbalanced training data issues, where rare classes are usually dominated by common ones in natural scene images. Furthermore, a superpixels-based re-segmentation was implemented to produce perceptually meaningful object boundaries. Experimental results demonstrate promising scene labeling performance of the proposed framework on the SIFTflow dataset both qualitatively and quantitatively, especially for rare class objects.