Unseen Categories Research Articles

From Pixel to Patch: Synthesize Context-Aware Features for Zero-Shot Semantic Segmentation.

Zero-shot learning (ZSL) has been actively studied for image classification tasks to relieve the burden of annotating image labels. Interestingly, the semantic segmentation task requires more labor-intensive pixel-wise annotation, but zero-shot semantic segmentation has not attracted extensive research interest. Thus, we focus on zero-shot semantic segmentation that aims to segment unseen objects with only category-level semantic representations provided for unseen categories. In this article, we propose a novel context-aware feature generation network (CaGNet) that can synthesize context-aware pixel-wise visual features for unseen categories based on category-level semantic representations and pixel-wise contextual information. The synthesized features are used to fine-tune the classifier to enable segmenting of unseen objects. Furthermore, we extend pixel-wise feature generation and fine-tuning to patch-wise feature generation and fine-tuning, which additionally considers the interpixel relationship. Experimental results on Pascal-VOC, Pascal-context, and COCO-stuff show that our method significantly outperforms the existing zero-shot semantic segmentation methods.

No Adversaries to Zero-Shot Learning: Distilling an Ensemble of Gaussian Feature Generators.

In zero-shot learning (ZSL), the task of recognizing unseen categories when no data for training is available, state-of-the-art methods generate visual features from semantic auxiliary information (e.g., attributes). In this work, we propose a valid alternative (simpler, yet better scoring) to fulfill the very same task. We observe that, if first- and second-order statistics of the classes to be recognized were known, sampling from Gaussian distributions would synthesize visual features that are almost identical to the real ones as per classification purposes. We propose a novel mathematical framework to estimate first- and second-order statistics, even for unseen classes: our framework builds upon prior compatibility functions for ZSL and does not require additional training. Endowed with such statistics, we take advantage of a pool of class-specific Gaussian distributions to solve the feature generation stage through sampling. We exploit an ensemble mechanism to aggregate a pool of softmax classifiers, each trained in a one-seen-class-out fashion to better balance the performance over seen and unseen classes. Neural distillation is finally applied to fuse the ensemble into a single architecture which can perform inference through one forward pass only. Our method, termed Distilled Ensemble of Gaussian Generators, scores favorably with respect to state-of-the-art works.

DILF: Differentiable rendering-based multi-view Image–Language Fusion for zero-shot 3D shape understanding

Zero-shot 3D shape understanding aims to recognize “unseen” 3D categories that are not present in training data. Recently, Contrastive Language–Image Pre-training (CLIP) has shown promising open-world performance in zero-shot 3D shape understanding tasks by information fusion among language and 3D modality. It first renders 3D objects into multiple 2D image views and then learns to understand the semantic relationships between the textual descriptions and images, enabling the model to generalize to new and unseen categories. However, existing studies in zero-shot 3D shape understanding rely on predefined rendering parameters, resulting in repetitive, redundant, and low-quality views. This limitation hinders the model’s ability to fully comprehend 3D shapes and adversely impacts the text–image fusion in a shared latent space. To this end, we propose a novel approach called Differentiable rendering-based multi-view Image–Language Fusion (DILF) for zero-shot 3D shape understanding. Specifically, DILF leverages large-scale language models (LLMs) to generate textual prompts enriched with 3D semantics and designs a differentiable renderer with learnable rendering parameters to produce representative multi-view images. These rendering parameters can be iteratively updated using a text–image fusion loss, which aids in parameters’ regression, allowing the model to determine the optimal viewpoint positions for each 3D object. Then a group-view mechanism is introduced to model interdependencies across views, enabling efficient information fusion to achieve a more comprehensive 3D shape understanding. Experimental results can demonstrate that DILF outperforms state-of-the-art methods for zero-shot 3D classification while maintaining competitive performance for standard 3D classification. The code is available at https://github.com/yuzaiyang123/DILP.

Cross-domain mechanism for few-shot object detection on Urine Sediment Image

Deep learning object detection networks require a large amount of box annotation data for training, which is difficult to obtain in the medical image field. The few-shot object detection algorithm is significant for an unseen category, which can be identified and localized with a few labeled data. For medical image datasets, the image style and target features are incredibly different from the knowledge obtained from training on the original dataset. We propose a background suppression attention(BSA) and feature space fine-tuning module (FSF) for this cross-domain situation where there is a large gap between the source and target domains. The background suppression attention reduces the influence of background information in the training process. The feature space fine-tuning module adjusts the feature distribution of the interest features, which helps to make better predictions. Our approach improves detection performance by using only the information extracted from the model without maintaining additional information, which is convenient and can be easily plugged into other networks. We evaluate the detection performance in the in-domain situation and cross-domain situation. In-domain experiments on the VOC and COCO datasets and the cross-domain experiments on the VOC to medical image dataset UriSed2K show that our proposed method effectively improves the few-shot detection performance.

ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval

The huge domain gap between sketches and photos and the highly abstract sketch representations pose challenges for sketch-based image retrieval (\underline{SBIR}). The zero-shot sketch-based image retrieval (\underline{ZS-SBIR}) is more generic and practical but poses an even greater challenge because of the additional knowledge gap between the seen and unseen categories. To simultaneously mitigate both gaps, we propose an \textbf{A}pproaching-and-\textbf{C}entralizing \textbf{Net}work (termed "\textbf{ACNet}") to jointly optimize sketch-to-photo synthesis and the image retrieval. The retrieval module guides the synthesis module to generate large amounts of diverse photo-like images which gradually approach the photo domain, and thus better serve the retrieval module than ever to learn domain-agnostic representations and category-agnostic common knowledge for generalizing to unseen categories. These diverse images generated with retrieval guidance can effectively alleviate the overfitting problem troubling concrete category-specific training samples with high gradients. We also discover the use of proxy-based NormSoftmax loss is effective in the zero-shot setting because its centralizing effect can stabilize our joint training and promote the generalization ability to unseen categories. Our approach is simple yet effective, which achieves state-of-the-art performance on two widely used ZS-SBIR datasets and surpasses previous methods by a large margin.

Attribute subspaces for zero-shot learning

Zero-shot learning (ZSL) aims to recognize unseen categories without corresponding training samples, which is a practical yet challenging task in computer vision and pattern recognition community. Current state-of-the-art locality-based ZSL methods aim to learn the explicit locality of discriminative attributes, which may suffer from insufficient class-level attribute supervision. In this paper, we introduce an Attribute Subspace learning method for ZSL (AS-ZSL) to learn implicit attribute composition, which is more general than attribute localization with only class-level attribute supervision. AS-ZSL exploits subspace representations that can effectively capture the intrinsic composition of high-dimensional image features and the diversity within attribute appearance. Furthermore, we develop a subspace distance based triplet loss to improve the distinguishability of the attribute subspace representation. Attribute subspace learning module is only needed for the training phase to jointly learn discriminative global features. This leads to a compact inference phase. Furthermore, the proposed AS-ZSL can be naturally extended to adapt to the transductive ZSL setting using a novel self-supervised training strategy. Extensive experimental results on several widely used ZSL datasets, i.e., CUB, AwA2, and SUN, demonstrate the advantage of AS-ZSL compared with the state-of-the-art under different ZSL settings.

Discriminative and Robust Attribute Alignment for Zero-Shot Learning

Zero-shot learning (ZSL) aims to learn models that can recognize images of semantically related unseen categories, through transferring attribute-based knowledge learned from training data of seen classes to unseen testing data. As visual attributes play a vital role in ZSL, recent embedding-based methods usually focus on learning a compatibility function between the visual representation and the class semantic attributes. While in this work, in addition to simply learning the region embedding of different semantic attributes to maintain the generalization capability of the learned model, we further consider to improve the discrimination power of the learned visual features themselves by contrastive embedding. It exploits both the class-wise and instance-wise supervision for GZSL, under the attribute guided weakly supervised representation learning framework. To further improve the robustness of the ZSL model, we also propose to train the model under the consistency regularization constraint, through taking full advantages of self-supervised signals of the image under various perturbed augmentation situations, which could make the model robust to some occluded or un-related attribute regions. Extensive experimental results demonstrate the effectiveness of the proposed ZSL method, achieving superior performances to state-of-the-art methods on three widely-used benchmark datasets, namely CUB, SUN, and AWA2. Our source code is released at https://github.com/KORIYN/CC-ZSL.

Dual-Aligned Feature Confusion Alleviation for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen samples by leveraging the connections between semantic and visual representations. Recently, a majority of GZSL methods focus on generating visual features for unseen categories conditioned on category-level semantic attributes. However, there is a considerable gap between generated features and real unseen features since the generator is trained with only seen samples. The final classifier may get confused by the unfaithful generated features and make misclassification. To alleviate this issue, we propose a dual-aligned feature confusion alleviation (DFCA) framework that simultaneously generates faithful and discriminative features for unseen categories. Specifically, our DFCA attains the faithfulness via a conditional invertible neural network (cINN) and aligns the generated visual features and reconstructed semantic conditions with their real counterparts, respectively. To further encourage distinguishable synthetic features, we learn discriminative category-level semantic conditions for cINN with an attributes mapping layer. To verify the proposed method, we conduct extensive experiments on five widely used benchmarks. Experimental results show that our method outperforms previous state-of-the-arts and successfully alleviates features confusion problem in GZSL. For instance, our method achieves the best performance in terms of seen accuracy, unseen accuracy and harmonic mean accuracy on FLO.

Contrastive Generative Network with Recursive-Loop for 3D point cloud generalized zero-shot classification

Generalized Zero-Shot Learning (GZSL) aims to recognize objects from both seen and unseen categories by transferring semantic knowledge and merely utilizing seen class data for training. Recent feature generation methods in the 2D image domain have made great progress. However, very little is known about its usefulness in 3D point cloud zero-shot learning. This work aims to facilitate research on 3D point cloud generalized zero-shot learning. Different from previous works, we focus on synthesizing the more high-level discriminative point cloud features. To this end, we design a representation enhancement strategy to generate the features. Specifically, we propose a Contrastive Generative Network with Recursive-Loop, termed as CGRL, which can be leveraged to enlarge the inter-class distances and narrow the intra-class gaps. By applying the contrastive representations to the generative model in a recursive-loop form, it can provide the self-guidance for the generator recurrently, which can help yield more discriminative features and train a better classifier. To validate the effectiveness of the proposed method, extensive experiments are conducted on three benchmarks, including ModelNet40, McGill, and ScanObjectNN. Experimental evaluations demonstrate the superiority of our approach and it can outperform the state-of-the-arts by a large margin. Code is available at https://github.com/photon-git/CGRL

Boosting generalized zero-shot learning with category-specific filters

Zero-Shot Learning (ZSL) has made significant progress driven by deep learning and is being promoted further with the advent of generative models. Despite the success of these methods, the type and number of unseen categories are nailed in the generative models, which makes it challenging to recognize unseen categories in an incremental manner, and the profits of some superior performance algorithms largely arise from their advanced capability of feature extraction, such as Transformers. This paper rigidly follows the assumptions introduced in conventional ZSL and proposes a visual feature filtering method based on a semantic mapping model, namely, filtering visual features through class-specific filters to effectively remove class-agnostic information. Extensive experiments are conducted on four benchmark datasets and have achieved very competitive performance.

Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval

Zero-shot (ZS) sketch-based three-dimensional (3D) shape retrieval (SBSR) is challenging due to the abstraction of sketches, cross-domain discrepancies between two-dimensional sketches and 3D shapes, and ZS-driven semantic knowledge transference from seen to unseen categories. Extant SBSR datasets suffer from lack of data, and no current SBSR methods consider ZS scenarios. In this paper, we contribute a new Doodle2Object (D2O) dataset consisting of 8,992 3D shapes and over 7M sketches spanning 50 categories. Then, we propose a novel prototype contrastive learning (PCL) method that effectively extracts features from different domains and adapts them to unseen categories. Specifically, our PCL method combines the ideas of contrastive and cluster-based prototype learning, and several randomly selected prototypes of different classes are assigned to each sample. By comparing these prototypes, a given sample can be moved closer to the same semantic class of samples while moving away from negative ones. Extensive experiments on two common SBSR benchmarks and our D2O dataset demonstrate the efficacy of the proposed PCL method for ZS-SBSR. Resource is available at https://github.com/yigohw/doodle2object.

Integrating topology beyond descriptions for zero-shot learning

Zero-shot learning (ZSL) aims to discriminate object categories through the identification of their attributes and has received much attention for its capability to predict unseen categories without collecting training data. Recently, excellent works have been devoted to optimizing the model inference by mining the topology among categories/attributes, which proves that the topology learning is beneficial and important for ZSL. However, existing works focus almost exclusively on the construction of semantic topological knowledge with textual descriptions, which, though effective, still suffer from two deficiencies: first, the semantic gap between modalities makes it difficult for the category attributes to accurately describe the corresponding visual characters, resulting in the topology constructed in the semantic modality being distorted in the visual modality; second, it is difficult for one to enumerate all the attributes hidden in images, resulting in an incomplete topology mined only from the defined attributes. Therefore, we propose a Cross-Modality Topology Propagation Matcher (CTPM) to construct a more complete topology system by collaborative mining of topological knowledge in both the visual and semantic modalities. We stand at the dataset level to construct sample-based visual topological knowledge based on the global image features to preserve the integrity of visual information. Meanwhile, we exploit the matching relationship between visual and semantic modalities to make topological knowledge propagate effectively across modalities, and fully enjoy the benefits of multi-modality topological knowledge in category/attribute reasoning. We validate the effectiveness of our CTPM through extensive experiments and achieve state-of-the-art performance on four ZSL datasets.

Adaptive adversarial prototyping network for few-shot prototypical translation

Translating multiple real-world source images to a single prototypical image is a challenging problem. Notably, these source images belong to unseen categories that did not exist during model training. We address this problem by proposing an adaptive adversarial prototype network (AAPN) and enhancing existing one-shot classification techniques. To overcome the limitations that traditional works cannot extract samples from novel categories, our method tends to solve the image translation task of unseen categories through a meta-learner. We train the model in an adversarial learning manner and introduce a style encoder to guide the model with an initial target style. The encoded style latent code enhances the performance of the network with conditional target style images. The AAPN outperforms the state-of-the-art methods in one-shot classification of brand logo dataset and achieves the competitive accuracy in the traffic sign dataset. Additionally, our model improves the visual quality of the reconstructed prototypes in unseen categories. Based on the qualitative and quantitative analysis, the effectiveness of our model for few-shot classification and generation is demonstrated.

GroupPlate: Toward Multi-Category License Plate Recognition

License Plate Detection and Recognition (LPDR) is widely used in Intelligent Transportation Systems (ITS). Although there are typically multiple categories of license plates, the majority of existing research cannot be applied to multi-category plates due to that existing methods are not optimised for multi-category plate scenarios and the scarcity of large-scale multi-category plate datasets. In this paper, we propose a multi-category license plate recognition framework called GroupPlate, which consists of Group Module and Indirect Supervision Module, making full use of the implicit and explicit grouping information of license plate. In addition, the Category Decouple Module is intended to decouple the grouping information from the original features, allowing the decoder to concentrate on character features. Simultaneously, we propose a large-scale All-Category license Plate detection and recognition Dataset (ACPD) for vehicles on the Chinese mainland, which also includes annotations of plates’ categories. Considering the domain gap between synthetic data and real data, we propose a simple but effective strategy called Feature Shift to mitigate the performance degradation caused by this gap. Experiments demonstrate that GroupPlate achieves the comparable performance to the existing methods on single-category license plate dataset and outperforms our baseline on the multi-category license plates dataset. Ablation experiments demonstrate the effectiveness of the modules in GroupPlate. Extensive results demonstrate that the dataset we proposed can mitigate the problem of models trained on a single-category license plate dataset failing to recognize multi-category license plates, and that our model can generalizes well to unseen categories. The work will be available at https://github.com/YilinGao-SHU/ACPD.

A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections

Abstract Until recently, most research in the Digital Humanities (DH) was monomodal, meaning that the object of analysis was either textual or visual. Seeking to integrate multimodality theory into the DH, this article demonstrates that recently developed multimodal deep learning models, such as Contrastive Language Image Pre-training (CLIP), offer new possibilities to explore and analyze image–text combinations at scale. These models, which are trained on image and text pairs, can be applied to a wide range of text-to-image, image-to-image, and image-to-text prediction tasks. Moreover, multimodal models show high accuracy in zero-shot classification, i.e. predicting unseen categories across heterogeneous datasets. Based on three exploratory case studies, we argue that this zero-shot capability opens up the way for a multimodal turn in DH research. Moreover, multimodal models allow scholars to move past the artificial separation of text and images that was dominant in the field and analyze multimodal meaning at scale. However, we also need to be aware of the specific (historical) bias of multimodal deep learning that stems from biases in the training data used to train these models.

Towards Zero-Shot Learning: A Brief Review and an Attention-Based Embedding Network

Zero-shot learning (ZSL), an emerging topic in recent years, targets at distinguishing unseen class images by taking images from seen classes for training the classifier. Existing works often build embeddings between global feature space and attribute space, which, however, neglect the treasure in image parts. Discrimination information is usually contained in the image parts, e.g., black and white striped area of a zebra is the key difference from a horse. As such, image parts can facilitate the transfer of knowledge among the seen and unseen categories. In this paper, we first conduct a brief review on ZSL with detailed descriptions of these methods. Next, to discover meaningful parts, we propose an end-to-end attention-based embedding network for ZSL, which contains two sub-streams: the attention part embedding (APE) stream, and the attention second-order embedding (ASE) stream. APE is used to discover multiple image parts based on attention. ASE is introduced for ensuring knowledge transfer stably by second-order collaboration. Furthermore, an adaptive thresholding strategy is proposed to suppress noise and redundant parts. Finally, a global branch is incorporated for the full use of global information. Experiments on four benchmarks demonstrate that our models achieve superior results under both ZSL and GZSL settings.

Multi-Dimensional Information Alignment in Different Modalities for Generalized Zero-Shot and Few-Shot Learning

Generalized zero-shot learning (GZSL) aims to solve the category recognition tasks for unseen categories under the setting that training samples only contain seen classes while unseen classes are not available. This research is vital as there are always existing new categories and large amounts of unlabeled data in realistic scenarios. Previous work for GZSL usually maps the visual information of the visible classes and the semantic description of the invisible classes into the identical embedding space to bridge the gap between the disjointed visible and invisible classes, while ignoring the intrinsic features of visual images, which are sufficiently discriminative to classify themselves. To better use discriminative information from visual classes for GZSL, we propose the n-CADA-VAE. In our approach, we map the visual feature of seen classes to a high-dimensional distribution while mapping the semantic description of unseen classes to a low-dimensional distribution under the same latent embedding space, thus projecting information of different modalities to corresponding space positions more accurately. We conducted extensive experiments on four benchmark datasets (CUB, SUN, AWA1, and AWA2). The results show our model’s superior performance in generalized zero-shot as well as few-shot learning.

ICCL: Independent and Correlative Correspondence Learning for few-shot image classification

Few-shot learning, which aims to transfer knowledge from past experiences to recognize novel categories with limited samples, is a challenging task in computer vision. However, existing few-shot works tend to focus on determining the baseline model independently and ignoring the correlation learning among instances. In light of this, in this paper, we propose a novel approach, termed Independent and Correlative Correspondence Learning (ICCL), to deal with the few-shot image classification problem. To build independent learning, we supervise the different geometry directions of each instance under data augmentation and the category of the instance in the source domain space, which enhance the discrimination of features. For constructing the correlation learning, we utilize the transformer unit to yield the long-range relationship between the global and local representation and the N-pair metric to learn the intra-class and inter-class variations, which generate credible decision boundaries. By performing the operations of independent learning and correlation learning, the generalization ability of the model can be substantially enhanced for the unseen category. Comprehensive experiments on public few-shot image classification benchmarks demonstrate that our proposed ICCL gains state-of-the-art results, and different ingredients in ICCL reflect their effectiveness.

SAR Target Recognition with Limited Training Samples in Open Set Conditions.

It is difficult to collect training samples for all types of synthetic aperture radar (SAR) targets. A realistic problem comes when unseen categories exist that are not included in training and benchmark data at the time of recognition, which is defined as open set recognition (OSR). Without the aid of side-information, generalized OSR methods used on ordinary optical images are usually not suitable for SAR images. In addition, OSR methods that require a large number of samples to participate in training are also not suitable for SAR images with the realistic situation of collection difficulty. In this regard, a task-oriented OSR method for SAR is proposed by distribution construction and relation measures to recognize targets of seen and unseen categories with limited training samples, and without any other simulation information. The method can judge category similarity to explain the unseen category. Distribution construction is realized by the graph convolutional network. The experimental results on the MSTAR dataset show that this method has a good recognition effect for the targets of both seen and unseen categories and excellent interpretation ability for unseen targets. Specifically, while recognition accuracy for seen targets remains above 95%, the recognition accuracy for unseen targets reaches 67% for the three-type classification problem, and 53% for the five-type classification problem.

Hierarchical context-agnostic network with contrastive feature diversity for one-shot semantic segmentation

One-shot semantic segmentation aims at distinguishing pixels of an unseen category from the background, using merely one annotated image from the same category. However, most previous works neglect the feature diversity of foreground and the context information of background by using the masked average pooling. To solve these issues, we propose the Hierarchical Context-Agnostic Network (HCNet). It mainly includes two modules: (1) a Hierarchical Pyramid Supportive (HPS) module that generate the hierarchical supportive prototypes from coarse to fine to ensure feature diversity, and (2) a Background Exclusion Supportive (BES) module that explicitly introduces the contrastive information from the background for more precise category features. We conduct extensive experiments on Pascal-5i and COCO-20i to evaluate the performance of HCNet. HCNet achieves the mIoU score of 62.1% on Pascal-5i and 40.7% on COCO-20i and outperforms other works for the challenging one-shot segmentation, which has proved the efficiency of the whole network. Code is available at https://github.com/fangzy97/hcnet.