Generalized Zero-shot Learning Research Articles

Generalized Zero-Shot Learning via Disentangled Representation

Zero-Shot Learning (ZSL) aims to recognize images belonging to unseen classes that are unavailable in the training process, while Generalized Zero-Shot Learning (GZSL) is a more realistic variant that both seen and unseen classes appear during testing. Most GZSL approaches achieve knowledge transfer based on the features of samples that inevitably contain information irrelevant to recognition, bringing negative influence for the performance. In this work, we propose a novel method, dubbed Disentangled-VAE, which aims to disentangle category-distilling factors and category-dispersing factors from visual as well as semantic features, respectively. In addition, a batch re-combining strategy on latent features is introduced to guide the disentanglement, encouraging the distilling latent features to be more discriminative for recognition. Extensive experiments demonstrate that our method outperforms the state-of-the-art approaches on four challenging benchmark datasets

Task Aligned Generative Meta-learning for Zero-shot Learning

Zero-shot learning (ZSL) refers to the problem of learning to classify instances from novel classes (unseen) that are absent in the training set (seen). Most ZSL methods infer the correlation between visual features and attributes to train the classifier for unseen classes. They may have a strong bias towards seen classes during training. Meta-learning has been introduced to mitigate the basis, but meta-ZSL methods are inapplicable when tasks used for training are sampled from diverse distributions. In this regard, we propose a novel Task-aligned Generative Meta-learning model for Zero-shot learning (TGMZ), aiming to mitigate the potentially biased training and to enable meta-ZSL to accommodate real-world datasets that contain diverse distributions. Specifically, TGMZ incorporates an attribute-conditioned task-wise distribution alignment network that projects tasks into a unified distribution to deliver an unbiased model. Our experiments show TGMZ achieves a relative improvement of 2.1%, 3.0%, 2.5%, and 7.6% over state-of-the-art algorithms on AWA1, AWA2, CUB, and aPY datasets, respectively. Overall, TGMZ outperforms competitors by 3.6% in the generalized zero-shot learning (GZSL) setting and 7.9% in our proposed fusion-ZSL setting.

Task-Independent Knowledge Makes for Transferable Representations for Generalized Zero-Shot Learning

Generalized Zero-Shot Learning (GZSL) targets recognizing new categories by learning transferable image representations. Existing methods find that, by aligning image representations with corresponding semantic labels, the semantic-aligned representations can be transferred to unseen categories. However, supervised by only seen category labels, the learned semantic knowledge is highly task-specific, which makes image representations biased towards seen categories. In this paper, we propose a novel Dual-Contrastive Embedding Network (DCEN) that simultaneously learns task-specific and task-independent knowledge via semantic alignment and instance discrimination. First, DCEN leverages task labels to cluster representations of the same semantic category by cross-modal contrastive learning and exploring semantic-visual complementarity. Besides task-specific knowledge, DCEN then introduces task-independent knowledge by attracting representations of different views of the same image and repelling representations of different images. Compared to high-level seen category supervision, this instance discrimination supervision encourages DCEN to capture low-level visual knowledge, which is less biased toward seen categories and alleviates the representation bias. Consequently, the task-specific and task-independent knowledge jointly make for transferable representations of DCEN, which obtains averaged 4.1% improvement on four public benchmarks.

Semantic-guided Reinforced Region Embedding for Generalized Zero-Shot Learning

Generalized zero-shot Learning (GZSL) aims to recognize images from either seen or unseen domain, mainly by learning a joint embedding space to associate image features with the corresponding category descriptions. Recent methods have proved that localizing important object regions can effectively bridge the semantic-visual gap. However, these are all based on one-off visual localizers, lacking of interpretability and flexibility. In this paper, we propose a novel Semantic-guided Reinforced Region Embedding (SR2E) network that can localize important objects in the long-term interests to construct semantic-visual embedding space. SR2E consists of Reinforced Region Module (R2M) and Semantic Alignment Module (SAM). First, without the annotated bounding box as supervision, R2M encodes the semantic category guidance into the reward and punishment criteria to teach the localizer serialized region searching. Besides, R2M explores different action spaces during the serialized searching path to avoid local optimal localization, which thereby generates discriminative visual features with less redundancy. Second, SAM preserves the semantic relationship into visual features via semantic-visual alignment and designs a domain detector to alleviate the domain confusion. Experiments on four public benchmarks demonstrate that the proposed SR2E is an effective GZSL method with reinforced embedding space, which obtains averaged 6.1% improvements.

Augmented semantic feature based generative network for generalized zero-shot learning

Zero-shot learning (ZSL) aims to recognize objects in images when no training data is available for the object classes. Under generalized zero-shot learning (GZSL) setting, the test objects belong to seen or unseen categories. In many recent studies, zero-shot learning is performed by leveraging generative networks to synthesize visual features for unseen class from class-specific semantic features. The user-defined semantic information is incomplete and lack of discriminability. However, most generative methods use user-defined semantic information directly as constraints of the generative model, which makes the visual features synthesized by the models lack of diversity and separability. In this paper, we propose a novel method to improve the semantic feature by utilizing discriminative visual features. Furthermore, a novel Augmented Semantic Feature Based Generative Network (ASFGN) is built to synthesize the separable visual representations for unseen classes. Since GAN-based generative model may suffer from mode collapse, we propose a novel collapse-alleviate loss to improve the training stability and generalization performance of our generative network. Extensive experiments on four benchmark datasets prove that our method outperforms the state-of-art approaches in both ZSL and GZSL settings.

Dual VAEGAN: A generative model for generalized zero-shot learning

Generalized zero-shot learning (GZSL) aims to recognize samples from all classes based on training samples of seen classes by bridging the gap between the seen and unseen classes through the semantic descriptions (attributes). Recently, generative-based methods have been used to convert the GZSL task into a supervised learning problem by generating visual features for unseen classes. In this paper, we propose a dual framework based on variational auto-encoder (VAE) and generative adversarial network (GAN), known as dual VAEGAN, to produce more clear visual features than VAE and alleviate the model collapse problem of GAN. To avoid generating unconstraint visual features, the generated visual features are forced to map back into their respective common space using a cycle consistency loss. Meanwhile, the cycle-consistency loss promotes the diversity and preserves the semantic consistency of the generated visual features. The experimental results on the six standard datasets indicate that dual VAEGAN can produce promising results as compared with other methods reported in the literature.

Sequence feature generation with temporal unrolling network for zero-shot action recognition

Zero-Shot Action Recognition (ZSAR) aims to recognize unseen action classes not included in the training dataset. Existing generative methods for ZSAR synthesize a feature of unseen action from a class embedding to overcome the absence of training data. Specifically, previous methods synthesize a feature which is averaged along a time axis, even though a video is extracted as a sequence of feature vectors. They suffer from the ambiguity of temporal information, which leads to confusion among actions sharing similar subactions. To tackle the problem, we first propose to synthesize not an averaged feature but a sequence consisting of feature vectors along the time axis. Hence, we design Sequence Feature Generative Adversarial Network (SFGAN) with Temporal Unrolling NEtwork (TUNE), which unrolls a class embedding into a set of condition vectors for generating sequences of features. Also, we employ a sequence discriminator as the second teacher. Through extensive experiments on the three benchmarks, HMDB51, UCF101, and Olympic, we validate the efficacy of sequence generation for ZSAR, and our method achieves the state-of-the-art generalized zero-shot learning performances.

Alleviating Domain Shift via Discriminative Learning for Generalized Zero-Shot Learning

In zero-shot learning (ZSL) tasks, especially in generalized zero-shot learning (GZSL), the model tends to classify unseen test samples into seen categories, which is well known as the domain shift problem, because the model is trained from seen samples without unseen samples. Recently, generative adversarial network (GAN) based methods have achieved good performance in GZSL, which replace real unseen features by synthesizing fake ones to mitigate the domain shift. However, the domain shift problem is still not well solved, due to the lacking of unseen samples in the training progress of the GAN generator. In this paper, we propose a generative model named discriminative learning GAN (DL-GAN) to alleviate the domain shift in GZSL. Specifically, the DL-GAN is designed with three novel components: a dual-stream embedding model that aligns features to the ground-truth attributes to extract discriminative latent attributes from features, an attribute-based generative model that generates high-quality unseen features from semantic attributes to guarantee inter-class discriminability and semantic consistency, and a seen/unseen classifier that leverages validation samples to distinguish seen samples from unseen ones. Experimental results on four widely used datasets verify that our proposed approach significantly outperforms the state-of-the-art methods under the GZSL protocol.

Dual Generative Network with Discriminative Information for Generalized Zero-Shot Learning

Zero-shot learning is dedicated to solving the classification problem of unseen categories, while generalized zero-shot learning aims to classify the samples selected from both seen classes and unseen classes, in which “seen” and “unseen” classes indicate whether they can be used in the training process, and if so, they indicate seen classes, and vice versa. Nowadays, with the promotion of deep learning technology, the performance of zero-shot learning has been greatly improved. Generalized zero-shot learning is a challenging topic that has promising prospects in many realistic scenarios. Although the zero-shot learning task has made gratifying progress, there is still a strong deviation between seen classes and unseen classes in the existing methods. Recent methods focus on learning a unified semantic-aligned visual representation to transfer knowledge between two domains, while ignoring the intrinsic characteristics of visual features which are discriminative enough to be classified by itself. To solve the above problems, we propose a novel model that uses the discriminative information of visual features to optimize the generative module, in which the generative module is a dual generation network framework composed of conditional VAE and improved WGAN. Specifically, the model uses the discrimination information of visual features, according to the relevant semantic embedding, synthesizes the visual features of unseen categories by using the learned generator, and then trains the final softmax classifier by using the generated visual features, thus realizing the recognition of unseen categories. In addition, this paper also analyzes the effect of the additional classifiers with different structures on the transmission of discriminative information. We have conducted a lot of experiments on six commonly used benchmark datasets (AWA1, AWA2, APY, FLO, SUN, and CUB). The experimental results show that our model outperforms several state-of-the-art methods for both traditional as well as generalized zero-shot learning.

Wheel Hub Defects Image Recognition Base on Zero-Shot Learning

In the wheel hub industry, the quality control of the product surface determines the subsequent processing, which can be realized through the hub defect image recognition based on deep learning. Although the existing methods based on deep learning have reached the level of human beings, they rely on large-scale training sets, however, these models are completely unable to cope with the situation without samples. Therefore, in this paper, a generalized zero-shot learning framework for hub defect image recognition was built. First, a reverse mapping strategy was adopted to reduce the hubness problem, then a domain adaptation measure was employed to alleviate the projection domain shift problem, and finally, a scaling calibration strategy was used to avoid the recognition preference of seen defects. The proposed model was validated using two data sets, VOC2007 and the self-built hub defect data set, and the results showed that the method performed better than the current popular methods.

Generalized Zero-Shot Learning With Multiple Graph Adaptive Generative Networks.

Generative adversarial networks (GANs) for (generalized) zero-shot learning (ZSL) aim to generate unseen image features when conditioned on unseen class embeddings, each of which corresponds to one unique category. Most existing works on GANs for ZSL generate features by merely feeding the seen image feature/class embedding (combined with random Gaussian noise) pairs into the generator/discriminator for a two-player minimax game. However, the structure consistency of the distributions among the real/fake image features, which may shift the generated features away from their real distribution to some extent, is seldom considered. In this paper, to align the weights of the generator for better structure consistency between real/fake features, we propose a novel multigraph adaptive GAN (MGA-GAN). Specifically, a Wasserstein GAN equipped with a classification loss is trained to generate discriminative features with structure consistency. MGA-GAN leverages the multigraph similarity structures between sliced seen real/fake feature samples to assist in updating the generator weights in the local feature manifold. Moreover, correlation graphs for the whole real/fake features are adopted to guarantee structure correlation in the global feature manifold. Extensive evaluations on four benchmarks demonstrate well the superiority of MGA-GAN over its state-of-the-art counterparts.

A Semantic Encoding Out-of-Distribution Classifier for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) poses a challenging problem in that it aims to recognize both seen classes that have appeared in the training stage and unseen classes that have not appeared during training. By utilizing a gating mechanism as the binary classifier, gating methods can decompose GZSL into a conventional ZSL problem and a supervision learning task, thereby leading to outstanding performance by GZSL. However, unseen classes contain many confusing visual samples that distribute too close to the seen class boundaries and are prone to misclassification. To solve this problem, we propose a novel semantic encoding out-of-distribution classifier (SE-OOD) for GZSL. Our method first utilizes semantically consistent mapping to project all the visual samples to their corresponding semantic attributes. Then, both the projected visual samples and original semantic attributes are encoded to their latent representations for distribution alignment. After separating the unseen samples from seen samples in the learned latent space, two domain classifiers are adopted to perform ZSL and supervised classification tasks. Extensive experiments are conducted on four benchmarks, and the results show that our proposed SE-OOD can outperform the state-of-the-arts by a large margin.

An Iterative Co-Training Transductive Framework for Zero Shot Learning

In zero-shot learning (ZSL) community, it is generally recognized that transductive learning performs better than inductive one as the unseen-class samples are also used in its training stage. How to generate pseudo labels for unseen-class samples and how to use such usually noisy pseudo labels are two critical issues in transductive learning. In this work, we introduce an iterative co-training framework which contains two different base ZSL models and an exchanging module. At each iteration, the two different ZSL models are co-trained to separately predict pseudo labels for the unseen-class samples, and the exchanging module exchanges the predicted pseudo labels, then the exchanged pseudo-labeled samples are added into the training sets for the next iteration. By such, our framework can gradually boost the ZSL performance by fully exploiting the potential complementarity of the two models' classification capabilities. In addition, our co-training framework is also applied to the generalized ZSL (GZSL), in which a semantic-guided OOD detector is proposed to pick out the most likely unseen-class samples before class-level classification to alleviate the bias problem in GZSL. Extensive experiments on three benchmarks show that our proposed methods could significantly outperform about 31 state-of-the-art ones.

On Implicit Attribute Localization for Generalized Zero-Shot Learning

Zero-shot learning (ZSL) aims to discriminate images from unseen classes by exploiting relations to seen classes via their attribute-based descriptions. Since attributes are often related to specific parts of objects, many recent works focus on discovering discriminative regions. However, these methods usually require additional complex part detection modules or attention mechanisms. In this paper, 1) we show that common ZSL backbones (without explicit attention nor part detection) can implicitly localize attributes, yet this property is not exploited. 2) Exploiting it, we then propose SELAR, a simple method that further encourages attribute localization, surprisingly achieving very competitive generalized ZSL (GZSL) performance when compared with more complex state-of-the-art methods. Our findings provide useful insight for designing future GZSL methods, and SELAR provides an easy to implement yet strong baseline.

Generalized Zero-Shot Learning Via Multi-Modal Aggregated Posterior Aligning Neural Network

The visual-semantic gap between the visual space (visual features) and semantic space (semantic attributes) is one of the main problems in the Generalized Zero-Shot Learning (GZSL) task. The essence of this problem is that the structure of manifolds in these two spaces is inconsistent, which makes it difficult to learn embeddings that unify visual features and semantic attributes for similarity measurement. In this work, we tackle this problem by proposing a multi-modal aggregated posterior aligning neural network based on Wasserstein Auto-encoders (WAE) which learns a shared latent space for visual features and semantic attributes. The key to our approach is that the aggregated posterior distribution of the latent representations encoded from visual features of each class is encouraged to be aligned with a Gaussian distribution predicted by the corresponding semantic attribute in the latent space. On one hand, requiring the latent manifolds of visual features and semantic attributes to be consistent preserves the inter-class association between seen and unseen classes. On the other hand, the aggregated posterior of each class is directly defined as a Gaussian in the latent space, which provides a reliable way to synthesize latent features for training classification models. Using the AWA1, AWA2, CUB, aPY, FLO, and SUN benchmark datasets, we extensively conducted comparative evaluations to demonstrate the advantages of our method over state-of-the-art approaches.

A plug-in attribute correction module for generalized zero-shot learning

While Zero Shot Learning models can recognize new classes without training examples, they often fails to incorporate both seen and unseen classes together at the test time, which is known as the Generalized Zero-shot Learning (GZSL) problem. This paper identifies a bottleneck issue when attributes are not well-defined, reliable, inaccurate in quantitative representations, or suffering from the visual-semantic discrepancy. We propose a Generic Plug-in Attribute Correction (GPAC) module which can effectively accommodate conventional ZSL in GZSL tasks. Different from existing embedding-based approaches which often lose the favor of transparency in attributes, our key challenge is to fully preserve the original meaning of the attributes and make it complementary and interpretable to upgrade existing ZSL models. To this end, we propose a novel nonnegative constraint with iterative Stochastic Gradient Descent toolbox to effectively fit our GPAC module into previous ZSL models. Extensive experiments on five popular datasets show that our method can effectively correct attributes and make conventional ZSL can achieve state-of-the-art performance on GZSL tasks. It is also a good practice for future models when incorporating prior human knowledge.

Modality independent adversarial network for generalized zero shot image classification

Zero Shot Learning (ZSL) aims to classify images of unseen target classes by transferring knowledge from source classes through semantic embeddings. The core of ZSL research is to embed both visual representation of object instance and semantic description of object class into a joint latent space and learn cross-modal (visual and semantic) latent representations. However, the learned representations by existing efforts often fail to fully capture the underlying cross-modal semantic consistency, and some of the representations are very similar and less discriminative. To circumvent these issues, in this paper, we propose a novel deep framework, called Modality Independent Adversarial Network (MIANet) for Generalized Zero Shot Learning (GZSL), which is an end-to-end deep architecture with three submodules. First, both visual feature and semantic description are embedded into a latent hyper-spherical space, where two orthogonal constraints are employed to ensure the learned latent representations discriminative. Second, a modality adversarial submodule is employed to make the latent representations independent of modalities to make the shared representations grab more cross-modal high-level semantic information during training. Third, a cross reconstruction submodule is proposed to reconstruct latent representations into the counterparts instead of themselves to make them capture more modality irrelevant information. Comprehensive experiments on five widely used benchmark datasets are conducted on both GZSL and standard ZSL settings, and the results show the effectiveness of our proposed method.

Inference guided feature generation for generalized zero-shot learning

Generalized zero-shot learning suffers from an extreme data imbalance problem, that is, the training data only come from seen classes while no unseen class data are available. Recently, a number of feature generation methods based on generative adversarial networks (GAN) have been proposed to address this problem. Existing feature generation methods, however, have never considered the under-constrained problem, and thus could generate an unrestricted visual feature corresponding to no meaningful object class. In this paper, we propose to equip the feature generation framework with a parallel inference network that projects visual feature to the semantic descriptor space, constraining to avoid the generation of unrestricted visual features. The two-parallel-stream framework (1) enables our method, termed inference guided feature generation (Inf-FG), to mitigate the under-constrained problem and (2) makes our Inf-FG applicable to transductive ZSL. Our Inf-FG learns the feature generator and the inference network simultaneously by aligning the joint distribution of visual features and semantic descriptors from the feature generator and the joint distribution from the inference network. We evaluate our approach on four benchmark ZSL datasets, including AWA, CUB, SUN, and FLO, on which our method improves our baselines on generalized zero-shot learning.

Discriminative comparison classifier for generalized zero-shot learning

Comparison classifier based generalized zero-shot learning (GZSL) helps to achieve knowledge transfer from seen to unseen classes. However, it only utilizes the original semantic features, which are highly related and indistinguishable, to learn an embedding with no consideration of the relationship between them. Moreover, it cannot well encode discriminative information embedded in semantic features. To handle these problems, we present a discriminative comparison classifier for GZSL, which consists of semantic embedding network and relation network. Semantic embedding network takes as input original semantic features and relationship features which can be obtained by clustering, it ensures that the embedding network from semantic space to visual space can learn more discriminative features. Relation network is used to learn relationship between the embedded features and visual features, the validation information will guide embedding network to learn more discriminate features. Moreover, we adopt a novel semantic pivot regularization to keep inter-class discrimination in the visual space. Extensive experiments on several real-world datasets demonstrate the effectiveness of our method over the other state-of-the-arts.

Transfer Increment for Generalized Zero-Shot Learning

Zero-shot learning (ZSL) is a successful paradigm for categorizing objects from the previously unseen classes. However, it suffers from severe performance degradation in the generalized ZSL (GZSL) setting, i.e., to recognize the test images that are from both seen and unseen classes. In this article, we present a simple but effective mechanism for GZSL and more open scenarios based on a transfer-increment strategy. On the one hand, a dual-knowledge-source-based generative model is constructed to tackle the missing data problem. Specifically, the local relational knowledge extracted from the label-embedding space and the global relational knowledge, which is the estimated data center in the feature-embedding space, are concurrently considered to synthesize the virtual exemplars. On the other hand, we further explore the training issue for the generative models under the GZSL setting. Two incremental training modes are designed to learn directly the unseen classes from the synthesized exemplars instead of the training classifiers with the seen and synthesized unseen exemplars together. It not only presents an effective unseen class learning but also requires less computing and storage resources in practical application. Comprehensive experiments are conducted based on five benchmark data sets. In comparison with the state-of-the-art methods, both the generating and training processes are considered for virtual exemplars by the proposed transfer-increment strategy, which results in a significant improvement in the conventional and GZSL tasks.