Generalized Zero-shot Learning Task Research Articles

Contrastive Prototype-Guided Generation for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes, while only samples from seen classes are available for training. The mainstream methods mitigate the lack of unseen training data by simulating the visual unseen samples. However, the sample generator is actually learned with just seen-class samples, and semantic descriptions of unseen classes are just provided to the pre-trained sample generator for unseen data generation, therefore, the generator would have bias towards seen categories, and the unseen generation quality, including both precision and diversity, is still the main learning challenge. To this end, we propose a Prototype-Guided Generation for Generalized Zero-Shot Learning (PGZSL), in order to guide the sample generation with unseen knowledge. First, unseen data generation is guided and rectified in PGZSL by contrastive prototypical anchors with both class semantic consistency and feature discriminability. Second, PGZSL introduces Certainty-Driven Mixup for generator to enrich the diversity of generated unseen samples, while suppress the generation of uncertain boundary samples as well. Empirical results over five benchmark datasets show that PGZSL significantly outperforms the SOTA methods in both ZSL and GZSL tasks.

Co-GZSL: Feature Contrastive Optimization for Generalized Zero-Shot Learning

Generalized Zero-Shot Learning (GZSL) learns from only labeled seen classes during training but discriminates both seen and unseen classes during testing. In GZSL tasks, most of the existing methods commonly utilize visual and semantic features for training. Due to the lack of visual features for unseen classes, recent works generate real-like visual features by using semantic features. However, the synthesized features in the original feature space lack discriminative information. It is important that the synthesized visual features should be similar to the ones in the same class, but different from the other classes. One way to solve this problem is to introduce the embedding space after generating visual features. Following this situation, the embedded features from the embedding space can be inconsistent with the original semantic features. For another way, some recent methods constrain the representation by reconstructing the semantic features using the original visual features and the synthesized visual features. In this paper, we propose a hybrid GZSL model, named feature Contrastive optimization for GZSL (Co-GZSL), to reconstruct the semantic features from the embedded features, which ensures that the embedded features are close to the original semantic features indirectly by comparing reconstructed semantic features with original semantic features. In addition, to settle the problem that the synthesized features lack discrimination and semantic consistency, we introduce a Feature Contrastive Optimization Module (FCOM) and jointly utilize contrastive and semantic cycle-consistency losses in the FCOM to strengthen the intra-class compactness and the inter-class separability and to encourage the model to generate semantically consistent and discriminative visual features. By combining the generative module, the embedding module, and the FCOM, we achieve Co-GZSL. We evaluate the proposed Co-GZSL model on four benchmarks, and the experimental results indicate that our model is superior over current methods. Code is available at: https://github.com/zhanzhuxi/Co-GZSL.

Explanatory Object Part Aggregation for Zero-Shot Learning.

Zero-shot learning (ZSL) aims to recognize objects from unseen classes only based on labeled images from seen classes. Most existing ZSL methods focus on optimizing feature spaces or generating visual features of unseen classes, both in conventional ZSL and generalized zero-shot learning (GZSL). However, since the learned feature spaces are suboptimal, there exists many virtual connections where visual features and semantic attributes are not corresponding to each other. To reduce virtual connections, in this paper, we propose to discover comprehensive and fine-grained object parts by building explanatory graphs based on convolutional feature maps, then aggregate object parts to train a part-net to obtain prediction results. Since the aggregated object parts contain comprehensive visual features for activating semantic attributes, the virtual connections can be reduced by a large extent. Since part-net aims to extract local fine-grained visual features, some attributes related to global structures are ignored. To take advantage of both local and global visual features, we design a feature distiller to distill local features into a master-net which aims to extract global features. The experimental results on AWA2, CUB, FLO, and SUN dataset demonstrate that our proposed method obviously outperforms the state-of-the-arts in both conventional ZSL and GZSL tasks.

Generative Mixup Networks for Zero-Shot Learning.

Zero-shot learning casts light on lacking unseen class data by transferring knowledge from seen classes via a joint semantic space. However, the distributions of samples from seen and unseen classes are usually imbalanced. Many zero-shot learning methods fail to obtain satisfactory results in the generalized zero-shot learning task, where seen and unseen classes are all used for the test. Also, irregular structures of some classes may result in inappropriate mapping from visual features space to semantic attribute space. A novel generative mixup networks with semantic graph alignment is proposed in this article to mitigate such problems. To be specific, our model first attempts to synthesize samples conditioned with class-level semantic information as the prototype to recover the class-based feature distribution from the given semantic description. Second, the proposed model explores a mixup mechanism to augment training samples and improve the generalization ability of the model. Third, triplet gradient matching loss is developed to guarantee the class invariance to be more continuous in the latent space, and it can help the discriminator distinguish the real and fake samples. Finally, a similarity graph is constructed from semantic attributes to capture the intrinsic correlations and guides the feature generation process. Extensive experiments conducted on several zero-shot learning benchmarks from different tasks prove that the proposed model can achieve superior performance over the state-of-the-art generalized zero-shot learning.

Multiscale Visual-Attribute Co-Attention for Zero-Shot Image Recognition.

Zero-shot image recognition aims to classify data from unseen classes, by exploring the association between visual features and the semantic representations of each class. Most existing approaches focus on learning a shared single-scale embedding space (often at the output layer of the network) for both visual and semantic features, ignoring a fact that different-scale visual features exhibit different semantics. In this article, we propose a multi-scale visual-attribute co-attention (mVACA) model, considering both visual-semantic alignment and visual discrimination at multiple scales. At each scale, a hybrid visual attention is realized by attribute-related attention and visual self-attention. The attribute-related attention is guided by a pseudo attribute vector inferred via a mutual information regularization (MIR). The visual self-attentive features further influence the attribute attention to emphasize visual-associated attributes. Leveraging multiscale visual discrimination, mVACA unifies standard zero-shot learning (ZSL) and generalized ZSL tasks in one framework, achieving state-of-the-art or competitive performance on several commonly used benchmarks of both setups. To better understand the interaction between images and attributes in mVACA, we also provide visualized analysis.

Knowledge Distillation Classifier Generation Network for Zero-Shot Learning.

In this article, we present a conceptually simple but effective framework called knowledge distillation classifier generation network (KDCGN) for zero-shot learning (ZSL), where the learning agent requires recognizing unseen classes that have no visual data for training. Different from the existing generative approaches that synthesize visual features for unseen classifiers' learning, the proposed framework directly generates classifiers for unseen classes conditioned on the corresponding class-level semantics. To ensure the generated classifiers to be discriminative to the visual features, we borrow the knowledge distillation idea to both supervise the classifier generation and distill the knowledge with, respectively, the visual classifiers and soft targets trained from a traditional classification network. Under this framework, we develop two, respectively, strategies, i.e., class augmentation and semantics guidance, to facilitate the supervision process from the perspectives of improving visual classifiers. Specifically, the class augmentation strategy incorporates some additional categories to train the visual classifiers, which regularizes the visual classifier weights to be compact, under supervision of which the generated classifiers will be more discriminative. The semantics-guidance strategy encodes the class semantics into the visual classifiers, which would facilitate the supervision process by minimizing the differences between the generated and the real-visual classifiers. To evaluate the effectiveness of the proposed framework, we have conducted extensive experiments on five datasets in image classification, i.e., AwA1, AwA2, CUB, FLO, and APY. Experimental results show that the proposed approach performs best in the traditional ZSL task and achieves a significant performance improvement on four out of the five datasets in the generalized ZSL task.

A Discriminative Cross-Aligned Variational Autoencoder for Zero-Shot Learning.

Zero-shot learning (ZSL) aims to classify unseen samples based on the relationship between the learned visual features and semantic features. Traditional ZSL methods typically capture the underlying multimodal data structures by learning an embedding function between the visual space and the semantic space with the Euclidean metric. However, these models suffer from the hubness problem and domain bias problem, which leads to unsatisfactory performance, especially in the generalized ZSL (GZSL) task. To tackle such a problem, we formulate a discriminative cross-aligned variational autoencoder (DCA-VAE) for ZSL. The proposed model effectively utilizes a modified cross-modal-alignment variational autoencoder (VAE) to transform both visual features and semantic features obtained by the discriminative cosine metric into latent features. The key to our method is that we collect principal discriminative information from visual and semantic features to construct latent features which contain the discriminative multimodal information associated with unseen samples. Finally, the proposed model DCA-VAE is validated on six benchmarks including the large dataset ImageNet, and several experimental results demonstrate the superiority of DCA-VAE over most existing embedding or generative ZSL models on the standard ZSL and the more realistic GZSL tasks.

An end-to-end deep generative approach with meta-learning optimization for zero-shot object classification

Zero-shot object classification aims to recognize the object of unseen classes whose supervised data are unavailable in the training stage. Recent zero-shot learning (ZSL) methods usually propose to generate new supervised data for unseen classes by designing various deep generative networks. In this paper, we propose an end-to-end deep generative ZSL approach that trains the data generation module and object classification module jointly, rather than separately as in the majority of existing generation-based ZSL methods. Due to the ZSL assumption that unseen data are unavailable in the training stage, the distribution of generated unseen data will shift to the distribution of seen data, and subsequently causes the projection domain shift problem. Therefore, we further design a novel meta-learning optimization model to improve the proposed generation-based ZSL approach, where the parameters initialization and the parameters update algorithm are meta-learned to assist model convergence. We evaluate the proposed approach on five standard ZSL datasets. The average accuracy increased by the proposed jointly training strategy is 2.7% and 23.0% for the standard ZSL task and generalized ZSL task respectively, and the meta-learning optimization further improves the accuracy by 5.0% and 2.1% on two ZSL tasks respectively. Experimental results demonstrate that the proposed approach has significant superiority in various ZSL tasks.

Auto‐encode the synthesis pseudo features for generalized zero‐shot learning

Zero-shot learning (ZSL) is to identify target categories without labeled data, in which semantic information is used to transfer knowledge from some seen categories. In the existing Generalized Zero-Shot Learning (GZSL) methods, domains shift problem always appeared during generating feature stage. In order to solve this problem, a new method to Auto-Encode the Synthesis Pseudo Features for the GZSL task (AESPF-GZSL) is proposed in this manuscript. Specifically, the AESPF-GZSL method trains the generated features under the semantic auto-encoder framework and exploits attention mechanism to train the generated features again. Then, the generated features are input to the classifier. The proposed method is performed on three benchmark data sets referred as to AWA, CUB and SUN. The experimental results show that the proposed method achieves the state-of-the art classifier accuracy both in ZSL and GZSL settings. In ZSL setting, the classification accuracy of our method is superior to the compared algorithms, improved by 0.40% in AWA and 0.30% in SUN, respectively. And in GZSL setting, the classification accuracy of the method is superior to the comparison algorithm 0.41% in Harmonic mean on AWA, and 1.01%, 0.62%, and 1.05% in training data set, testing data set, and harmonic average on SUN.

Attributes learning network for generalized zero-shot learning

In the absence of unseen training data, zero-shot learning algorithms utilize the semantic knowledge shared by the seen and unseen classes to establish the connection between the visual space and the semantic space, so as to realize the recognition of the unseen classes. However, in real applications, the original semantic representation cannot well characterize both the class-specificity structure and discriminative information in dimension space, which leads to unseen classes being easily misclassified into seen classes. To tackle this problem, we propose a Salient Attributes Learning Network (SALN) to generate discriminative and expressive semantic representation under the supervision of the visual features. Meanwhile, ℓ1,2-norm constraint is employed to make the learned semantic representation well characterize the class-specificity structure and discriminative information in dimension space. Then feature alignment network projects the learned semantic representation into visual space and a relation network is adopted for classification. The performance of the proposed approach has made progress on the five benchmark datasets in generalized zero-shot learning task, and in-depth experiments indicate the effectiveness and excellence of our method.

Cross-modal distribution alignment embedding network for generalized zero-shot learning

Many approaches in generalized zero-shot learning (GZSL) rely on cross-modal mapping between the image feature space and the class embedding space, which achieves knowledge transfer from seen to unseen classes. However, these two spaces are completely different space and their manifolds are inconsistent, the existing methods suffer from highly overlapped semantic description of different classes, as in GZSL tasks unseen classes can be easily misclassified into seen classes. To handle these problems, we adopt a novel semantic embedding network which helps to encode more discriminative information from initial semantic attributes to semantic embeddings in visual space. Meanwhile, a distribution alignment constraint is adopted to help keep the distribution of the learned semantic embeddings consistent with the distribution of real image features. Moreover, an auxiliary classifier is adopted to strengthen the quality of the learned semantic embeddings. Finally, a relation network is used to classify the unseen images by computing the relation scores between the semantic embeddings and image features, which is much more flexible than the fixed distance metric functions. Experimental results demonstrate that our proposed method is superior to other state-of-the-arts.

Dual VAEGAN: A generative model for generalized zero-shot learning

Generalized zero-shot learning (GZSL) aims to recognize samples from all classes based on training samples of seen classes by bridging the gap between the seen and unseen classes through the semantic descriptions (attributes). Recently, generative-based methods have been used to convert the GZSL task into a supervised learning problem by generating visual features for unseen classes. In this paper, we propose a dual framework based on variational auto-encoder (VAE) and generative adversarial network (GAN), known as dual VAEGAN, to produce more clear visual features than VAE and alleviate the model collapse problem of GAN. To avoid generating unconstraint visual features, the generated visual features are forced to map back into their respective common space using a cycle consistency loss. Meanwhile, the cycle-consistency loss promotes the diversity and preserves the semantic consistency of the generated visual features. The experimental results on the six standard datasets indicate that dual VAEGAN can produce promising results as compared with other methods reported in the literature.

A plug-in attribute correction module for generalized zero-shot learning

While Zero Shot Learning models can recognize new classes without training examples, they often fails to incorporate both seen and unseen classes together at the test time, which is known as the Generalized Zero-shot Learning (GZSL) problem. This paper identifies a bottleneck issue when attributes are not well-defined, reliable, inaccurate in quantitative representations, or suffering from the visual-semantic discrepancy. We propose a Generic Plug-in Attribute Correction (GPAC) module which can effectively accommodate conventional ZSL in GZSL tasks. Different from existing embedding-based approaches which often lose the favor of transparency in attributes, our key challenge is to fully preserve the original meaning of the attributes and make it complementary and interpretable to upgrade existing ZSL models. To this end, we propose a novel nonnegative constraint with iterative Stochastic Gradient Descent toolbox to effectively fit our GPAC module into previous ZSL models. Extensive experiments on five popular datasets show that our method can effectively correct attributes and make conventional ZSL can achieve state-of-the-art performance on GZSL tasks. It is also a good practice for future models when incorporating prior human knowledge.

Transfer Increment for Generalized Zero-Shot Learning

Zero-shot learning (ZSL) is a successful paradigm for categorizing objects from the previously unseen classes. However, it suffers from severe performance degradation in the generalized ZSL (GZSL) setting, i.e., to recognize the test images that are from both seen and unseen classes. In this article, we present a simple but effective mechanism for GZSL and more open scenarios based on a transfer-increment strategy. On the one hand, a dual-knowledge-source-based generative model is constructed to tackle the missing data problem. Specifically, the local relational knowledge extracted from the label-embedding space and the global relational knowledge, which is the estimated data center in the feature-embedding space, are concurrently considered to synthesize the virtual exemplars. On the other hand, we further explore the training issue for the generative models under the GZSL setting. Two incremental training modes are designed to learn directly the unseen classes from the synthesized exemplars instead of the training classifiers with the seen and synthesized unseen exemplars together. It not only presents an effective unseen class learning but also requires less computing and storage resources in practical application. Comprehensive experiments are conducted based on five benchmark data sets. In comparison with the state-of-the-art methods, both the generating and training processes are considered for virtual exemplars by the proposed transfer-increment strategy, which results in a significant improvement in the conventional and GZSL tasks.

Multi-modal generative adversarial network for zero-shot learning

In this paper, we propose a novel approach for Zero-Shot Learning (ZSL), where the test instances are from the novel categories that no visual data are available during training. The existing approaches typically address ZSL by embedding the visual features into a category-shared semantic space. However, these embedding-based approaches easily suffer from the “heterogeneity gap” issue since a single type of class semantic prototype cannot characterize the categories well. To alleviate this issue, we assume that different class semantics reflect different views of the corresponding class, and thus fuse various types of class semantic prototypes resided in different semantic spaces with a feature fusion network to generate pseudo visual features. Through the adversarial mechanism of the real visual features and the fused pseudo visual features, the complementary semantics in various spaces are effectively captured. Experimental results on three benchmark datasets demonstrate that the proposed approach achieves impressive performances on both traditional ZSL and generalized ZSL tasks.

Zero-VAE-GAN: Generating Unseen Features for Generalized and Transductive Zero-Shot Learning.

Zero-shot learning (ZSL) is a challenging task due to the lack of unseen class data during training. Existing works attempt to establish a mapping between the visual and class spaces through a common intermediate semantic space. The main limitation of existing methods is the strong bias towards seen class, known as the domain shift problem, which leads to unsatisfactory performance in both conventional and generalized ZSL tasks. To tackle this challenge, we propose to convert ZSL to the conventional supervised learning by generating features for unseen classes. To this end, a joint generative model that couples variational autoencoder (VAE) and generative adversarial network (GAN), called Zero-VAE-GAN, is proposed to generate high-quality unseen features. To enhance the class-level discriminability, an adversarial categorization network is incorporated into the joint framework. Besides, we propose two self-training strategies to augment unlabeled unseen features for the transductive extension of our model, addressing the domain shift problem to a large extent. Experimental results on five standard benchmarks and a large-scale dataset demonstrate the superiority of our generative model over the state-of-the-art methods for conventional, especially generalized ZSL tasks. Moreover, the further improvement of the transductive setting demonstrates the effectiveness of the proposed self-training strategies.

Multi-Modality Adversarial Auto-Encoder for Zero-Shot Learning

The existing generative Zero-Shot Learning (ZSL) methods only consider the unidirectional alignment from the class semantics to the visual features while ignoring the alignment from the visual features to the class semantics, which fails to construct the visual-semantic interactions well. In this paper, we propose to generate visual features based on an auto-encoder framework paired with multi-modality adversarial networks respectively for visual and semantic modalities to reinforce the visual-semantic interactions with a bidirectional alignment, which ensures the generated visual features to fit the real visual distribution and to be highly related to the semantics. The encoder aims at generating real-like visual features while the decoder forces both the real and the generated visual features to be more related to the class semantics. To further capture the discriminative information of the generated visual features, both the real and generated visual features are forced to be classified into the correct classes via a classification network. Experimental results on four benchmark datasets show that the proposed approach is particularly competitive on both the traditional ZSL and the generalized ZSL tasks.