Zero-shot Learning Research Articles

Predicting pairwise interaction affinities with ℓ0-penalized least squares–a nonsmooth bi-objective optimization based approach*

In this paper, we introduce a novel nonsmooth optimization-based method LMBM-Kron ℓ 0 LS for solving large-scale pairwise interaction affinity prediction problems. The aim of LMBM-Kron ℓ 0 LS is to produce accurate predictions using as sparse a model as possible. We apply the least squares approach with Kronecker product kernels for a loss function and a continuous formulation of ℓ 0 pseudonorm for regularization. Thus, we end up solving a nonsmooth optimization problem. In addition, we apply a specific bi-objective criterion to strike a balance between the prediction accuracy of the learned model and the sparsity of the obtained solution. We compare LMBM-Kron ℓ 0 LS with some state-of-the-art methods using three benchmark and two simulated data sets under four distinct experimental settings, including zero-shot learning. Moreover, both binary and continuous interaction affinity labels are considered with LMBM-Kron ℓ 0 LS. The results show that LMBM-Kron ℓ 0 LS finds sparse solutions without sacrificing too much in the prediction performance.

How to apply zero-shot learning to text data in substance use research: An overview and tutorial with media data.

A vast amount of media-related text data is generated daily in the form of social media posts, news stories or academic articles. These text data provide opportunities for researchers to analyse and understand how substance-related issues are being discussed. The main methods to analyse large text data (content analyses or specifically trained deep-learning models) require substantial manual annotation and resources. A machine-learning approach called 'zero-shot learning' may be quicker, more flexible and require fewer resources. Zero-shot learning uses models trained on large, unlabelled (or weakly labelled) data sets to classify previously unseen data into categories on which the model has not been specifically trained. This means that a pre-existing zero-shot learning model can be used to analyse media-related text data without the need for task-specific annotation or model training. This approach may be particularly important for analysing data that is time critical. This article describes the relatively new concept of zero-shot learning and how it can be applied to text data in substance use research, including a brief practical tutorial.

Learning adversarial semantic embeddings for zero-shot recognition in open worlds

Zero-Shot Learning (ZSL) focuses on classifying samples of unseen classes with only their side semantic information presented during training. It cannot handle real-life, open-world scenarios where there are test samples of unknown classes for which neither samples (e.g., images) nor their side semantic information is known during training. Open-Set Recognition (OSR) is dedicated to addressing the unknown class issue, but existing OSR methods are not designed to model the semantic information of the unseen classes. To tackle this combined ZSL and OSR problem, we consider the case of “Zero-Shot Open-Set Recognition” (ZS-OSR), where a model is trained under the ZSL setting but it is required to accurately classify samples from the unseen classes while being able to reject samples from the unknown classes during inference. We perform large experiments on combining existing state-of-the-art ZSL and OSR models for the ZS-OSR task on four widely used datasets adapted from the ZSL task, and reveal that ZS-OSR is a non-trivial task as the simply combined solutions perform badly in distinguishing the unseen-class and unknown-class samples. We further introduce a novel approach specifically designed for ZS-OSR, in which our model learns to generate adversarial semantic embeddings of the unknown classes to train an unknowns-informed ZS-OSR classifier. Extensive empirical results show that our method 1) substantially outperforms the combined solutions in detecting the unknown classes while retaining the classification accuracy on the unseen classes and 2) achieves similar superiority under generalized ZS-OSR settings. Our code is available at https://github.com/lhrst/ASE.

Radiation-Variation Insensitive Coarse-to-Fine Image Registration for Infrared and Visible Remote Sensing Based on Zero-Shot Learning

Infrared and visible remote sensing image registration is significant for utilizing remote sensing images to obtain scene information. However, it is difficult to establish a large number of correct matches due to the difficulty in obtaining similarity metrics due to the presence of radiation variation between heterogeneous sensors, which is caused by different imaging principles. In addition, the existence of sparse textures in infrared images as well as in some scenes and the small number of relevant trainable datasets also hinder the development of this field. Therefore, we combined data-driven and knowledge-driven methods to propose a Radiation-variation Insensitive, Zero-shot learning-based Registration (RIZER). First, RIZER, as a whole, adopts a detector-free coarse-to-fine registration framework, and the data-driven methods use a Transformer based on zero-shot learning. Next, the knowledge-driven methods are embodied in the coarse-level matches, where we adopt the strategy of seeking reliability by introducing the HNSW algorithm and employing a priori knowledge of local geometric soft constraints. Then, we simulate the matching strategy of the human eye to transform the matching problem into a model-fitting problem and employ a multi-constrained incremental matching approach. Finally, after fine-level coordinate fine tuning, we propose an outlier culling algorithm that only requires very few iterations. Meanwhile, we propose a multi-scene infrared and visible remote sensing image registration dataset. After testing, RIZER achieved a correct matching rate of 99.55% with an RMSE of 1.36 and had an advantage in the number of correct matches, as well as a good generalization ability for other multimodal images, achieving the best results when compared to some traditional and state-of-the-art multimodal registration algorithms.

A novel mechanical fault diagnosis for high-voltage circuit breakers with zero-shot learning

In recent years, data-driven methods have been widely used in the field of high-voltage circuit breakers (HVCBs) fault diagnosis. However, due to the complex mechanical structure of HVCBs and the special operating environment, it is difficult to obtain a large amount of fault samples and exhaust all fault types. The lack of fault samples and fault types typically results in significant degradation of diagnostic performance. To address this problem, we design a novel method named R-MLL for zero-shot HVCB fault diagnosis. R-MLL tries to identify unseen fault types only by training seen fault types. First, to focus on all the details of the HVCB mechanical vibration signal, the wavelet transform is used to multi-scale refine the fault data. Second, a new network (RDSCNN) is designed to extract multidimensional features based on convolutional neural network incorporating residual block and depthwise separable convolution. Third, a multi-label attribute learning network is designed, enabling the fusion of fault features and attributes and allowing attribute labels to assist fault diagnosis tasks. Extensive experiments show that R-MLL gets average accuracy of 86.2% for compound fault diagnosis without the need of using target fault samples for building the diagnostic model. Comparisons with a number of state-of-the-art techniques show the superiority of the proposed method for zero-shot HVCB diagnosis.

Multi-agent cooperative operation based on cross-domain zero-shot learning

Multi-agent systems (MAS) based on reinforcement learning have shown impressive performance in complex tasks. However, they are still weak when it comes to addressing tasks without any samples. In this paper, we design a 2D ground and a 3D underwater cooperative task, which serve as the source and target tasks, respectively. The MAS requires learning prior knowledge that can be generalized to the target task from the source task. To solve this problem, we propose cross-domain zero-shot learning (CDZSL) for MAS cooperative operation. Specifically, we first construct a state transition graph by decomposing the source tasks into multiple state phases. Subsequently, we describe the task description document for MAS and construct the task-related semantic embedding space, which effectively combats the semantic gap. Then, we propose zero-shot learning based on feature synthesis mapping (FSM), which synthesizes the feature centers of the unseen classes by the features of seen classes. During knowledge transfer, FSM is employed to assist MAS's decisions-making in the target task. Finally, extensive experiments show that CDZSL, compared with MAX-Q, improves the adaptability of MAS, where the cost of physical time for the first time to complete the new task is reduced by eight times.

Low-shot learning and class imbalance: a survey

AbstractThe tasks of few-shot, one-shot, and zero-shot learning—or collectively “low-shot learning” (LSL)—at first glance are quite similar to the long-standing task of class imbalanced learning; specifically, they aim to learn classes for which there is little labeled data available. Motivated by this similarity, we conduct a survey to review the recent literature for works which combine these fields in one of two ways, either addressing the obstacle of class imbalance within a LSL setting, or utilizing LSL techniques or frameworks in order to combat class imbalance within other settings. In our survey of over 60 papers in a wide range of applications from January 2020 to July 2023 (inclusive), we examine and report methodologies and experimental results, find that most works report performance at or above their respective state-of-the-art, and highlight current research gaps which hold potential for future work, especially those involving the use of LSL techniques in imbalanced tasks. To this end, we emphasize the lack of works utilizing LSL approaches based on large language models or semantic data, and works using LSL for big-data imbalanced tasks.

Generative Mixup Networks for Zero-Shot Learning.

Zero-shot learning casts light on lacking unseen class data by transferring knowledge from seen classes via a joint semantic space. However, the distributions of samples from seen and unseen classes are usually imbalanced. Many zero-shot learning methods fail to obtain satisfactory results in the generalized zero-shot learning task, where seen and unseen classes are all used for the test. Also, irregular structures of some classes may result in inappropriate mapping from visual features space to semantic attribute space. A novel generative mixup networks with semantic graph alignment is proposed in this article to mitigate such problems. To be specific, our model first attempts to synthesize samples conditioned with class-level semantic information as the prototype to recover the class-based feature distribution from the given semantic description. Second, the proposed model explores a mixup mechanism to augment training samples and improve the generalization ability of the model. Third, triplet gradient matching loss is developed to guarantee the class invariance to be more continuous in the latent space, and it can help the discriminator distinguish the real and fake samples. Finally, a similarity graph is constructed from semantic attributes to capture the intrinsic correlations and guides the feature generation process. Extensive experiments conducted on several zero-shot learning benchmarks from different tasks prove that the proposed model can achieve superior performance over the state-of-the-art generalized zero-shot learning.

ATZSL: Defensive Zero-Shot Recognition in the Presence of Adversaries

Zero-shot learning (ZSL) has received extensive attention recently especially in areas of fine-grained object recognition, retrieval, and image captioning. Due to the complete lack of training samples and high requirement of defense transferability, the ZSL model learned is particularly vulnerable against adversarial attacks. Recent work also showed adversarially robust generalization requires more data. This may significantly affect the robustness of ZSL. However, very few efforts have been devoted towards this direction. In this paper, we take an initial attempt, and propose a generic formulation to provide a systematical solution (named ATZSL) for learning a defensive ZSL model. It is capable of achieving better generalization on various adversarial objects recognition while only losing a negligible performance on clean images for unseen classes, by casting ZSL into a min-max optimization problem. To address it, we design a defensive relation prediction network, which can bridge the seen and unseen class domains via attributes to generalize prediction and defense strategy. Additionally, our framework can be extended to deal with the poisoned scenario of unseen class attributes. An extensive group of experiments are then presented, demonstrating that ATZSL obtains remarkably more favorable trade-off between model transferability and robustness, over currently available alternatives under various settings.

ZS-VAT: Learning Unbiased Attribute Knowledge for Zero-Shot Recognition Through Visual Attribute Transformer.

In zero-shot learning (ZSL), attribute knowledge plays a vital role in transferring knowledge from seen classes to unseen classes. However, most existing ZSL methods learn biased attribute knowledge, which usually results in biased attribute prediction and a decline in zero-shot recognition performance. To solve this problem and learn unbiased attribute knowledge, we propose a visual attribute Transformer for zero-shot recognition (ZS-VAT), which is an effective and interpretable Transformer designed specifically for ZSL. In ZS-VAT, we design an attribute-head self-attention (AHSA) that is capable of learning unbiased attribute knowledge. Specifically, each attribute head in AHSA first transforms the local features into attribute-reinforced features and then accumulates the attribute knowledge from all corresponding reinforced features, reducing the mutual influence between attributes and avoiding information loss. AHSA finally preserves unbiased attribute knowledge through attribute embeddings. We also propose an attribute fusion model (AFM) that learns to recover the correct category knowledge from the attribute knowledge. In particular, AFM takes all features from AHSA as input and generates global embeddings. We carried out experiments to demonstrate that the attribute knowledge from AHSA and the category knowledge from AFM are able to assist each other. During the final semantic prediction, we combine the attribute embedding prediction (AEP) and global embedding prediction (GEP). We evaluated the proposed scheme on three benchmark datasets. ZS-VAT outperformed the state-of-the-art generalized ZSL (GZSL) methods on two datasets and achieved competitive results on the other dataset.

Part-Object Progressive Refinement Network for Zero-Shot Learning.

Zero-shot learning (ZSL) recognizes unseen images by sharing semantic knowledge transferred from seen images, encouraging the investigation of associations between semantic and visual information. Prior works have been devoted to the alignment of global visual features with semantic information, i.e., attribute vectors, or further mining the local part regions related to each attribute and then simply concatenating them for category decisions. Although effective, these works ignore intrinsic interactions between local parts and the whole object, which enables a more discriminative and representative knowledge transfer for ZSL. In this paper, we propose a Part-Object Progressive Refinement Network (POPRNet), where discriminative and transferable semantics are progressively refined by the cooperation between parts and the whole object. Specifically, POPRNet incorporates discriminative part semantics and object-centric semantics guided by semantic intensity to improve cross-domain transferability. To achieve part-object learning, a semantic-augment transformer (SaT) is proposed to model the part-object relation at the part-level via an encoder and at the object-level via a decoder, generating a comprehensive semantic representation to boost discriminability and transferability. By introducing the prototype updating module embedded with the prototype selection layers, the discriminative ability of the updated category prototype is enhanced to further improve the recognition performance of ZSL. Extensive experiments are conducted to demonstrate the superiority and competitiveness of our proposed POPRNet method on three public benchmark datasets. The code is available at https://github.com/ManLiuCoder/POPRNet.

A novel approach to intrusion detection using zero-shot learning hybrid partial labels

Computer networks have become the backbone of our interconnected world in today's technologically driven landscape. Unauthorized access or malicious activity carried out by threat actors to acquire control of network resources, exploit vulnerabilities, or undermine system integrity are examples of network intrusion. ZSL(Zero-Shot Learning) is a machine learning paradigm that addresses the problem of detecting and categorizing objects or concepts that were not present in the training data. . Traditional supervised learning algorithms for intrusion detection frequently struggle with insufficient labeled data and may struggle to adapt to unexpected assault patterns. In this article We have proposed a unique zero-shot learning hybrid partial label model suited to a large image-based network intrusion dataset to overcome these difficulties. The core contribution of this study is the creation and successful implementation of a novel zero-shot learning hybrid partial label model for network intrusion detection, which has a remarkable accuracy of 99.12%. The suggested system lays the groundwork for future study into other feature selection techniques and the performance of other machine learning classifiers on larger datasets. Such research can advance the state-of-the-art in intrusion detection and improve our ability to detect and prevent the network attacks. We hope that our research will spur additional research and innovation in this critical area of cybersecurity.

Brain-Inspired Fast-and Slow-Update Prompt Tuning for Few-Shot Class-Incremental Learning.

Few-shot class-incremental learning (FSCIL) aims to learn new classes incrementally with a limited number of samples per class. Foundation models combined with prompt tuning showcase robust generalization and zero-shot learning (ZSL) capabilities, endowing them with potential advantages in transfer capabilities for FSCIL. However, existing prompt tuning methods excel in optimizing for stationary datasets, diverging from the inherent sequential nature in the FSCIL paradigm. To address this issue, taking inspiration from the "fast and slow mechanism" of the complementary learning systems (CLSs) in the brain, we present fast-and slow-update prompt tuning FSCIL (FSPT-FSCIL), a brain-inspired prompt tuning method for transferring foundation models to the FSCIL task. We categorize the prompts into two groups: fast-update prompts and slow-update prompts, which are interactively trained through meta-learning. Fast-update prompts aim to learn new knowledge within a limited number of iterations, while slow-update prompts serve as meta-knowledge and aim to strike a balance between rapid learning and avoiding catastrophic forgetting. Through experiments on multiple benchmark tests, we demonstrate the effectiveness and superiority of FSPT-FSCIL. The code is available at https://github.com/qihangran/FSPT-FSCIL.

Boosting Zero-Shot Learning via Contrastive Optimization of Attribute Representations.

Zero-shot learning (ZSL) aims to recognize classes that do not have samples in the training set. One representative solution is to directly learn an embedding function associating visual features with corresponding class semantics for recognizing new classes. Many methods extend upon this solution, and recent ones are especially keen on extracting rich features from images, e.g., attribute features. These attribute features are normally extracted within each individual image; however, the common traits for features across images yet belonging to the same attribute are not emphasized. In this article, we propose a new framework to boost ZSL by explicitly learning attribute prototypes beyond images and contrastively optimizing them with attribute-level features within images. Besides the novel architecture, two elements are highlighted for attribute representations: a new prototype generation module (PM) is designed to generate attribute prototypes from attribute semantics; a hard-example-based contrastive optimization scheme is introduced to reinforce attribute-level features in the embedding space. We explore two alternative backbones, CNN-based and transformer-based, to build our framework and conduct experiments on three standard benchmarks, Caltech-UCSD Birds-200-2011 (CUB), SUN attribute database (SUN), and animals with attributes 2 (AwA2). Results on these benchmarks demonstrate that our method improves the state of the art by a considerable margin. Our codes will be available at https://github.com/dyabel/CoAR-ZSL.git.

Can we spot fake public comments generated by ChatGPT(-3.5, -4)?: Japanese stylometric analysis expose emulation created by one-shot learning.

Public comments are an important opinion for civic when the government establishes rules. However, recent AI can easily generate large quantities of disinformation, including fake public comments. We attempted to distinguish between human public comments and ChatGPT-generated public comments (including ChatGPT emulated that of humans) using Japanese stylometric analysis. Study 1 conducted multidimensional scaling (MDS) to compare 500 texts of five classes: Human public comments, GPT-3.5 and GPT-4 generated public comments only by presenting the titles of human public comments (i.e., zero-shot learning, GPTzero), GPT-3.5 and GPT-4 emulated by presenting sentences of human public comments and instructing to emulate that (i.e., one-shot learning, GPTone). The MDS results showed that the Japanese stylometric features of the public comments were completely different from those of the GPTzero-generated texts. Moreover, GPTone-generated public comments were closer to those of humans than those generated by GPTzero. In Study 2, the performance levels of the random forest (RF) classifier for distinguishing three classes (human, GPTzero, and GPTone texts). RF classifiers showed the best precision for the human public comments of approximately 90%, and the best precision for the fake public comments generated by GPT (GPTzero and GPTone) was 99.5% by focusing on integrated next writing style features: phrase patterns, parts-of-speech (POS) bigram and trigram, and function words. Therefore, the current study concluded that we could discriminate between GPT-generated fake public comments and those written by humans at the present time.

Mutual Information Guided Diffusion for Zero-Shot Cross-Modality Medical Image Translation.

Cross-modality data translation has attracted great interest in medical image computing. Deep generative models show performance improvement in addressing related challenges. Nevertheless, as a fundamental challenge in image translation, the problem of zero-shot learning cross-modality image translation with fidelity remains unanswered. To bridge this gap, we propose a novel unsupervised zero-shot learning method called Mutual Information guided Diffusion Model, which learns to translate an unseen source image to the target modality by leveraging the inherent statistical consistency of Mutual Information between different modalities. To overcome the prohibitive high dimensional Mutual Information calculation, we propose a differentiable local-wise mutual information layer for conditioning the iterative denoising process. The Local-wise-Mutual-Information-Layer captures identical cross-modality features in the statistical domain, offering diffusion guidance without relying on direct mappings between the source and target domains. This advantage allows our method to adapt to changing source domains without the need for retraining, making it highly practical when sufficient labeled source domain data is not available. We demonstrate the superior performance of MIDiffusion in zero-shot cross-modality translation tasks through empirical comparisons with other generative models, including adversarial-based and diffusion-based models. Finally, we showcase the real-world application of MIDiffusion in 3D zero-shot learning-based cross-modality image segmentation tasks.

Rethink, Revisit, Revise: A Spiral Reinforced Self-Revised Network for Zero-Shot Learning.

Current approaches to zero-shot learning (ZSL) struggle to learn generalizable semantic knowledge capable of capturing complex correlations. Inspired by Spiral Curriculum, which enhances learning processes by revisiting knowledge, we propose a form of spiral learning that revisits visual representations based on a sequence of attribute groups (e.g., a combined group of color and shape). Spiral learning aims to learn generalized local correlations, enabling models to gradually enhance global learning and, thus, understand complex correlations. Our implementation is based on a two-stage reinforced self-revised (RSR) framework: preview and review. RSR first previews visual information to construct diverse attribute groups in a weakly supervised manner. Then, it spirally learns refined localities based on attribute groups and uses localities to revise global semantic correlations. Our framework outperforms state-of-the-art algorithms on four benchmark datasets in both zero-shot and generalized zero-shot settings, which demonstrates the effectiveness of spiral learning in learning generalizable and complex correlations. We also conduct extensive analysis to show that attribute groups and reinforced decision processes can capture complementary semantic information to improve predictions and aid explainability.

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning.

The spiking neural networks (SNNs) that efficiently encode temporal sequences have shown great potential in extracting audio-visual joint feature representations. However, coupling SNNs (binary spike sequences) with transformers (float-point sequences) to jointly explore the temporal-semantic information still facing challenges. In this paper, we introduce a novel Spiking Tucker Fusion Transformer (STFT) for audio-visual zero-shot learning (ZSL). The STFT leverage the temporal and semantic information from different time steps to generate robust representations. The time-step factor (TSF) is introduced to dynamically synthesis the subsequent inference information. To guide the formation of input membrane potentials and reduce the spike noise, we propose a global-local pooling (GLP) which combines the max and average pooling operations. Furthermore, the thresholds of the spiking neurons are dynamically adjusted based on semantic and temporal cues. Integrating the temporal and semantic information extracted by SNNs and Transformers are difficult due to the increased number of parameters in a straightforward bilinear model. To address this, we introduce a temporal-semantic Tucker fusion module, which achieves multi-scale fusion of SNN and Transformer outputs while maintaining full second-order interactions. Our experimental results demonstrate the effectiveness of the proposed approach in achieving state-of-the-art performance in three benchmark datasets. The harmonic mean (HM) improvement of VGGSound, UCF101 and ActivityNet are around 15.4%, 3.9%, and 14.9%, respectively.

Defending against Misinformation: Evaluating Transformer Architectures for Quick Misinformation Detection on Social Media

Prior to the technological revolution, reliance on traditional sources such as newspapers and other mass communication channels was the predominant method of consuming news. With the pervasive influence of social media, traditional news sources have undergone significant displacement, yielding predominance to platforms rooted in social media networks. Informal news propagation through social media has been observed to be fast in nature. With the information propagating fast, the misinformation also propagates rapidly on social media. If the misinformation is not detected early, it can cause serious problems. Thus the problem of identifying misinformation at the onset needs much attention. This paper explores various architectures for detecting misinformation, employing deep transfer learning techniques. The study utilizes transformer architectures, namely MPNet, SentenceT5, and Generalizable T5-based dense Retrievers, to generate embeddings. These architectures have undergone pre-training and fine-tuning on diverse tasks. The system’s performance is assessed using a benchmark dataset of misinformation from Reddit, referred to as ‘Fakeddit’. When tested on unseen data, the models perform well, and outperform previously reported baselines and other models on the given dataset with textual modality. The results indicate that transfer learning with zero-shot or few-shot learning can prove very cost-effective and efficient for the problem of misinformation detection.

Low-Light Phase Retrieval With Implicit Generative Priors.

Phase retrieval (PR) is fundamentally important in scientific imaging and is crucial for nanoscale techniques like coherent diffractive imaging (CDI). Low radiation dose imaging is essential for applications involving radiation-sensitive samples. However, most PR methods struggle in low-dose scenarios due to high shot noise. Recent advancements in optical data acquisition setups, such as in-situ CDI, have shown promise for low-dose imaging, but they rely on a time series of measurements, making them unsuitable for single-image applications. Similarly, data-driven phase retrieval techniques are not easily adaptable to data-scarce situations. Zero-shot deep learning methods based on pre-trained and implicit generative priors have been effective in various imaging tasks but have shown limited success in PR. In this work, we propose low-dose deep image prior (LoDIP), which combines in-situ CDI with the power of implicit generative priors to address single-image low-dose phase retrieval. Quantitative evaluations demonstrate LoDIP's superior performance in this task and its applicability to real experimental scenarios.