Zero shot learning based on class visual prototypes and semantic consistency

Abstract

Zero shot classification is to recognize unseen images which are not present in training set, which is a quite difficult task. Traditional zero shot classification methods ignore the semantic inconsistencies between visual and semantic spaces which make those methods less effective. Projecting semantic representations to visual space can alleviate hubness problem. However, directly utilizing the semantic to visual mapping function learnt by seen classes to unseen classes will lead to domain shift problem. We propose a zero shot learning method which learns visual prototypes and preserves semantic consistency across visual and semantic spaces simultaneously, to handle semantic inconsistency problem and domain shift problem. The semantic consistency is represented by a shared sparse graph of visual space and semantic space. Our key insight is that the visual prototypes learning and the sparse graph learning are unified into a single process. Extensive experiments demonstrate that the results by the proposed method could be boosted significantly.