Generalized Zero-shot Learning Research Articles

Isolation and distillation network for generalized zero-shot learning

Isolation and distillation network for generalized zero-shot learning

Domain-Aware Prototype Network for Generalized Zero-Shot Learning

Domain-Aware Prototype Network for Generalized Zero-Shot Learning

Ontoloji-Based Generalized Zero-Shot Learning with Generative Networks

Ontoloji-Based Generalized Zero-Shot Learning with Generative Networks

Generalized zero-shot learning for classifying unseen wafer map patterns

A data-driven approach for classifying defect patterns in wafer maps is crucial for automating the quality assurance process during semiconductor fabrication. Existing works have proposed machine learning-based methods to accomplish this important problem. However, many of these approaches assume model training from a supervised learning perspective, in which a substantial number of annotated wafer maps, each with its actual defect pattern, need to be available. Due to the cost and time required to generate such an annotated dataset, recently a few works started to explore the problem of identifying previously unseen defect patterns. However, the methods proposed in those works are able to only detect the presence of unseen defect patterns without identifying what those defect patterns are. To address this issue, this paper introduces a generalized zero-shot learning (GZSL)-based framework capable of classifying and identifying both seen and unseen defect patterns. It consists of a visual feature extraction phase based on semi-supervised contrastive learning (CL), a clustering phase using distance-based initialization, and an alignment phase based on the Wasserstein-Distance-based Visual Structure Constraint method. Our framework further reduces the need for expensive annotated datasets previously required to train supervised learning-based models, thereby improving the practicality of the data-driven approach for wafer map pattern classification. Furthermore, our framework introduces an effective way of incorporating additional semantic information in wafer map pattern classification that traditionally relied on training models solely from wafer maps. To the best of the authors’ knowledge, our work is the first work to adopt GZSL in wafer map pattern classification.

Instance-Specific Model Perturbation Improves Generalized Zero-Shot Learning.

Zero-shot learning (ZSL) refers to the design of predictive functions on new classes (unseen classes) of data that have never been seen during training. In a more practical scenario, generalized zero-shot learning (GZSL) requires predicting both seen and unseen classes accurately. In the absence of target samples, many GZSL models may overfit training data and are inclined to predict individuals as categories that have been seen in training. To alleviate this problem, we develop a parameter-wise adversarial training process that promotes robust recognition of seen classes while designing during the test a novel model perturbation mechanism to ensure sufficient sensitivity to unseen classes. Concretely, adversarial perturbation is conducted on the model to obtain instance-specific parameters so that predictions can be biased to unseen classes in the test. Meanwhile, the robust training encourages the model robustness, leading to nearly unaffected prediction for seen classes. Moreover, perturbations in the parameter space, computed from multiple individuals simultaneously, can be used to avoid the effect of perturbations that are too extreme and ruin the predictions. Comparison results on four benchmark ZSL data sets show the effective improvement that the proposed framework made on zero-shot methods with learned metrics.

Contrastive Prototype-Guided Generation for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes, while only samples from seen classes are available for training. The mainstream methods mitigate the lack of unseen training data by simulating the visual unseen samples. However, the sample generator is actually learned with just seen-class samples, and semantic descriptions of unseen classes are just provided to the pre-trained sample generator for unseen data generation, therefore, the generator would have bias towards seen categories, and the unseen generation quality, including both precision and diversity, is still the main learning challenge. To this end, we propose a Prototype-Guided Generation for Generalized Zero-Shot Learning (PGZSL), in order to guide the sample generation with unseen knowledge. First, unseen data generation is guided and rectified in PGZSL by contrastive prototypical anchors with both class semantic consistency and feature discriminability. Second, PGZSL introduces Certainty-Driven Mixup for generator to enrich the diversity of generated unseen samples, while suppress the generation of uncertain boundary samples as well. Empirical results over five benchmark datasets show that PGZSL significantly outperforms the SOTA methods in both ZSL and GZSL tasks.

Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition

In the realm of Zero-Shot Learning (ZSL), we address biases in Generalized Zero-Shot Learning (GZSL) models, which favor seen data. To counter this, we introduce an end-to-end generative GZSL framework called D3GZSL. This framework respects seen and synthesized unseen data as in-distribution and out-of-distribution data, respectively, for a more balanced model. D3GZSL comprises two core modules: in-distribution dual space distillation (ID2SD) and out-of-distribution batch distillation (O2DBD). ID2SD aligns teacher-student outcomes in embedding and label spaces, enhancing learning coherence. O2DBD introduces low-dimensional out-of-distribution representations per batch sample, capturing shared structures between seen and un seen categories. Our approach demonstrates its effectiveness across established GZSL benchmarks, seamlessly integrating into mainstream generative frameworks. Extensive experiments consistently showcase that D3GZSL elevates the performance of existing generative GZSL methods, under scoring its potential to refine zero-shot learning practices. The code is available at: https://github.com/PJBQ/D3GZSL.git

Co-GZSL: Feature Contrastive Optimization for Generalized Zero-Shot Learning

Generalized Zero-Shot Learning (GZSL) learns from only labeled seen classes during training but discriminates both seen and unseen classes during testing. In GZSL tasks, most of the existing methods commonly utilize visual and semantic features for training. Due to the lack of visual features for unseen classes, recent works generate real-like visual features by using semantic features. However, the synthesized features in the original feature space lack discriminative information. It is important that the synthesized visual features should be similar to the ones in the same class, but different from the other classes. One way to solve this problem is to introduce the embedding space after generating visual features. Following this situation, the embedded features from the embedding space can be inconsistent with the original semantic features. For another way, some recent methods constrain the representation by reconstructing the semantic features using the original visual features and the synthesized visual features. In this paper, we propose a hybrid GZSL model, named feature Contrastive optimization for GZSL (Co-GZSL), to reconstruct the semantic features from the embedded features, which ensures that the embedded features are close to the original semantic features indirectly by comparing reconstructed semantic features with original semantic features. In addition, to settle the problem that the synthesized features lack discrimination and semantic consistency, we introduce a Feature Contrastive Optimization Module (FCOM) and jointly utilize contrastive and semantic cycle-consistency losses in the FCOM to strengthen the intra-class compactness and the inter-class separability and to encourage the model to generate semantically consistent and discriminative visual features. By combining the generative module, the embedding module, and the FCOM, we achieve Co-GZSL. We evaluate the proposed Co-GZSL model on four benchmarks, and the experimental results indicate that our model is superior over current methods. Code is available at: https://github.com/zhanzhuxi/Co-GZSL.

Rectification-Based Knowledge Retention for Task Incremental Learning.

In the task incremental learning problem, deep learning models suffer from catastrophic forgetting of previously seen classes/tasks as they are trained on new classes/tasks. This problem becomes even harder when some of the test classes do not belong to the training class set, i.e., the task incremental generalized zero-shot learning problem. We propose a novel approach to address the task incremental learning problem for both the non zero-shot and zero-shot settings. Our proposed approach, called Rectification-based Knowledge Retention (RKR), applies weight rectifications and affine transformations for adapting the model to any task. During testing, our approach can use the task label information (task-aware) to quickly adapt the network to that task. We also extend our approach to make it task-agnostic so that it can work even when the task label information is not available during testing. Specifically, given a continuum of test data, our approach predicts the task and quickly adapts the network to the predicted task. We experimentally show that our proposed approach achieves state-of-the-art results on several benchmark datasets for both non zero-shot and zero-shot task incremental learning.

Explanatory Object Part Aggregation for Zero-Shot Learning.

Zero-shot learning (ZSL) aims to recognize objects from unseen classes only based on labeled images from seen classes. Most existing ZSL methods focus on optimizing feature spaces or generating visual features of unseen classes, both in conventional ZSL and generalized zero-shot learning (GZSL). However, since the learned feature spaces are suboptimal, there exists many virtual connections where visual features and semantic attributes are not corresponding to each other. To reduce virtual connections, in this paper, we propose to discover comprehensive and fine-grained object parts by building explanatory graphs based on convolutional feature maps, then aggregate object parts to train a part-net to obtain prediction results. Since the aggregated object parts contain comprehensive visual features for activating semantic attributes, the virtual connections can be reduced by a large extent. Since part-net aims to extract local fine-grained visual features, some attributes related to global structures are ignored. To take advantage of both local and global visual features, we design a feature distiller to distill local features into a master-net which aims to extract global features. The experimental results on AWA2, CUB, FLO, and SUN dataset demonstrate that our proposed method obviously outperforms the state-of-the-arts in both conventional ZSL and GZSL tasks.

Prototype-Augmented Self-Supervised Generative Network for Generalized Zero-Shot Learning.

Generalized Zero-Shot Learning (GZSL) aims at recognizing images from both seen and unseen classes by constructing correspondences between visual images and semantic embedding. However, existing methods suffer from a strong bias problem, where unseen images in the target domain tend to be recognized as seen classes in the source domain. To address this issue, we propose a Prototype-augmented Self-supervised Generative Network by integrating self-supervised learning and prototype learning into a feature generating model for GZSL. The proposed model enjoys several advantages. First, we propose a Self-supervised Learning Module to exploit inter-domain relationships, where we introduce anchors as a bridge between seen and unseen categories. In the shared space, we pull the distribution of the target domain away from the source domain and obtain domain-aware features. To our best knowledge, this is the first work to introduce self-supervised learning into GZSL as learning guidance. Second, a Prototype Enhancing Module is proposed to utilize class prototypes to model reliable target domain distribution in finer granularity. In this module, a Prototype Alignment mechanism and a Prototype Dispersion mechanism are combined to guide the generation of better target class features with intra-class compactness and inter-class separability. Extensive experimental results on five standard benchmarks demonstrate that our model performs favorably against state-of-the-art GZSL methods.

A Gating Model for Bias Calibration in Generalized Zero-shot Learning.

Generalized zero-shot learning (GZSL) aims at training a model that can generalize to unseen class data by only using auxiliary information. One of the main challenges in GZSL is a biased model prediction toward seen classes caused by overfitting on only available seen class data during training. To overcome this issue, we propose a two-stream autoencoder-based gating model for GZSL. Our gating model predicts whether the query data is from seen classes or unseen classes, and utilizes separate seen and unseen experts to predict the class independently from each other. This framework avoids comparing the biased prediction scores for seen classes with the prediction scores for unseen classes. In particular, we measure the distance between visual and attribute representations in the latent space and the cross-reconstruction space of the autoencoder. These distances are utilized as complementary features to characterize unseen classes at different levels of data abstraction. Also, the two-stream autoencoder works as a unified framework for the gating model and the unseen expert, which makes the proposed method computationally efficient. We validate our proposed method in four benchmark image recognition datasets. In comparison with other state-of-the-art methods, we achieve the best harmonic mean accuracy in SUN and AWA2, and the second best in CUB and AWA1. Furthermore, our base model requires at least 20% less number of model parameters than state-of-the-art methods relying on generative models.

Generative Mixup Networks for Zero-Shot Learning.

Zero-shot learning casts light on lacking unseen class data by transferring knowledge from seen classes via a joint semantic space. However, the distributions of samples from seen and unseen classes are usually imbalanced. Many zero-shot learning methods fail to obtain satisfactory results in the generalized zero-shot learning task, where seen and unseen classes are all used for the test. Also, irregular structures of some classes may result in inappropriate mapping from visual features space to semantic attribute space. A novel generative mixup networks with semantic graph alignment is proposed in this article to mitigate such problems. To be specific, our model first attempts to synthesize samples conditioned with class-level semantic information as the prototype to recover the class-based feature distribution from the given semantic description. Second, the proposed model explores a mixup mechanism to augment training samples and improve the generalization ability of the model. Third, triplet gradient matching loss is developed to guarantee the class invariance to be more continuous in the latent space, and it can help the discriminator distinguish the real and fake samples. Finally, a similarity graph is constructed from semantic attributes to capture the intrinsic correlations and guides the feature generation process. Extensive experiments conducted on several zero-shot learning benchmarks from different tasks prove that the proposed model can achieve superior performance over the state-of-the-art generalized zero-shot learning.

ZS-VAT: Learning Unbiased Attribute Knowledge for Zero-Shot Recognition Through Visual Attribute Transformer.

In zero-shot learning (ZSL), attribute knowledge plays a vital role in transferring knowledge from seen classes to unseen classes. However, most existing ZSL methods learn biased attribute knowledge, which usually results in biased attribute prediction and a decline in zero-shot recognition performance. To solve this problem and learn unbiased attribute knowledge, we propose a visual attribute Transformer for zero-shot recognition (ZS-VAT), which is an effective and interpretable Transformer designed specifically for ZSL. In ZS-VAT, we design an attribute-head self-attention (AHSA) that is capable of learning unbiased attribute knowledge. Specifically, each attribute head in AHSA first transforms the local features into attribute-reinforced features and then accumulates the attribute knowledge from all corresponding reinforced features, reducing the mutual influence between attributes and avoiding information loss. AHSA finally preserves unbiased attribute knowledge through attribute embeddings. We also propose an attribute fusion model (AFM) that learns to recover the correct category knowledge from the attribute knowledge. In particular, AFM takes all features from AHSA as input and generates global embeddings. We carried out experiments to demonstrate that the attribute knowledge from AHSA and the category knowledge from AFM are able to assist each other. During the final semantic prediction, we combine the attribute embedding prediction (AEP) and global embedding prediction (GEP). We evaluated the proposed scheme on three benchmark datasets. ZS-VAT outperformed the state-of-the-art generalized ZSL (GZSL) methods on two datasets and achieved competitive results on the other dataset.

A Dual Discriminator Method for Generalized Zero-Shot Learning

A Dual Discriminator Method for Generalized Zero-Shot Learning

Adaptive Conditional Denoising Diffusion Model with Hybrid Affinity Regularizer for Generalized Zero-shot Learning

Adaptive Conditional Denoising Diffusion Model with Hybrid Affinity Regularizer for Generalized Zero-shot Learning

Generalized Zero-Shot Learning for Action Recognition Fusing Text and Image GANs

Generalized Zero-Shot Learning for Action Recognition Fusing Text and Image GANs

CGASNet: A Generalized Zero-Shot Learning Compound Fault Diagnosis Approach for Bearings

CGASNet: A Generalized Zero-Shot Learning Compound Fault Diagnosis Approach for Bearings

Incorporating attribute-level aligned comparative network for generalized zero-shot learning

The key challenge of zero-shot learning (ZSL) is to sufficiently disentangle each latent attribute from the class-level semantic annotations of images, thereby achieving a desirable semantic transfer to unseen classes with the disentangled attributes. However, most existing studies tackle the ZSL task with a strict class-level alignment strategy that may yield insufficient disentanglement: (1) this strategy simply aligns holistic visual feature with its associated class-level semantic vector for each image; (2) the class-level semantic vectors have limited diversity and complex compositions of attributes. To address these issues, we propose an incorporating attribute-level aligned comparative network, i.e., IAAC-net, that develops the alignment strategy of ZSL to the attribute level. IAAC-net aims to establish diversified attribute-level and refined class-level alignments to facilitate attribute disentanglement and simultaneously improve zero-shot generalization. By further proposing a confusion-aware loss, the model is forced to rectify the disentanglement of indistinguishable attributes, which leads to a more accurate attribute disentanglement. The proposed IAAC-net yields significant improvements over the strong baselines, leading to new state-of-the-art performances on three popular challenging benchmarks, i.e., CUB, SUN, and AWA2.

Leveraging Balanced Semantic Embedding for Generative Zero-Shot Learning.

Generative (generalized) zero-shot learning [(G)ZSL] models aim to synthesize unseen class features by using only seen class feature and attribute pairs as training data. However, the generated fake unseen features tend to be dominated by the seen class features and thus classified as seen classes, which can lead to inferior performances under zero-shot learning (ZSL), and unbalanced results under generalized ZSL (GZSL). To address this challenge, we tailor a novel balanced semantic embedding generative network (BSeGN), which incorporates balanced semantic embedding learning into generative learning scenarios in the pursuit of unbiased GZSL. Specifically, we first design a feature-to-semantic embedding module (FEM) to distinguish real seen and fake unseen features collaboratively with the generator in an online manner. We introduce the bidirectional contrastive and balance losses for the FEM learning, which can guarantee a balanced prediction for the interdomain features. In turn, the updated FEM can boost the learning of the generator. Next, we propose a multilevel feature integration module (mFIM) from the cycle-consistency branch of BSeGN, which can mitigate the domain bias through feature enhancement. To the best of our knowledge, this is the first work to explore embedding and generative learning jointly within the field of ZSL. Extensive evaluations on four benchmarks demonstrate the superiority of BSeGN over its state-of-the-art counterparts.