Visual-semantic consistency matching network for generalized zero-shot learning

Abstract

Generalized zero-shot learning aims to classify samples of seen and unseen classes by providing only the labels of seen classes. Most GZSL methods directly associate seen classes’ visual features with semantic information or leverage semantic information to synthesize samples of unseen classes to transfer the knowledge of seen classes to unseen classes. However, existing generative methods simply employ visual features extracted from the per-trained CNN backbone, which overlooks that the visual features of similar categories have significant similarities and lack enough discriminating information, leading to poor classification performance when generalizing the knowledge of seen classes to unseen classes. To mitigate this issue, we present a Visual-Semantic Consistency Matching Network (VSCM) for generalized zero-shot learning. Our proposed method employs a conditional VAE to generate the visual features of unseen classes and utilizes a visual-semantic consistency matching network that aligns visual space with semantic space to obtain visual-semantic consistency features. Specifically, we propose a semantic chunking network that teams up with semantic attention and a semantic encoder to guide the visual-semantic consistency matching network to get synthetic semantic information aligned with factual semantic information. The semantic relation network guarantees the consistency between the factual semantic information and the block features, while the semantic independence network measures the independence of the blocks. Finally, we concatenate visual features and synthetic semantic information as visual-semantic consistency features to improve the separability between categories. Extensive experiments on five GZSL benchmark datasets demonstrate the significant generalization performance of our proposed method over the state-of-the-art methods. Our codes have been available at:https://github.com/zzq158/VSCM-GZSL.