Few-shot Setting Research Articles

Learning Scene-Aware Spatio-Temporal GNNs for Few-Shot Early Action Prediction

We aim to address a new task named few-shot early action prediction (FS-EAP) that learns classifiers for novel actions from only a few partially observed videos. We argue that the task is extremely challenging since the partially observed videos do not contain enough action information in a few-shot environment. To tackle this task, in this paper, we propose a scene-aware spatio-temporal graph neural network (SA-STGNN) by leveraging the fine-grained spatio-temporal interactions in the video scenes. Specifically, we first generate a spatio-temporal graph corresponding to the partially observed video to capture comprehensive spatio-temporal correlations. Then we utilize the spatio-temporal graph as the input of our SA-STGNN and predict the augmented video features corresponding to the complete video. The architecture uses several scene-aware learning blocks, which are a combination of edge fusion graph neural layers and temporal gated convolutional layers to jointly model spatial and temporal dependencies. Finally, we employ an early action predictor to exploit the learned video features for predicting actions in the few-shot setting. Extensive experimental results on two widely adopted video datasets demonstrate the effectiveness of our approach and its superior performance over the state-of-the-art approaches.

Two-Stream Prototype Learning Network for Few-Shot Face Recognition Under Occlusions

Few-shot face recognition under occlusion (FSFRO) aims to recognize novel subjects given only a few, probably occluded face images, and it is challenging and common in real-world scenarios. Unknown occlusions may deteriorate the class prototypes, while an occluded image in the support set may be critical for recognition if the query image is occluded. This motivates us to propose a novel Two-stream Prototype Learning Network (TSPLN) for FSFR under occlusions by simultaneously considering the quality of support images and their relevance to the query i mage. Specifically, we design a two-stream architecture, which mainly consists of a support-centered stream and query-centered stream, to learn the optimal class prototypes. The former stream is to reduce the negative impact of occluded images on the prototype. This is achieved by exploring the similarities between different images in the support set. In the query-centered stream, we exploit the relevance between the query and support set based on feature alignment (FA). We conduct extensive experiments on two popular datasets: CASIA-WebFace and RMFRD. The experimental results show that our proposed method achieves the state-of-the-art performance for occluded face recognition in the few-shot setting.

Filtering Specialized Change in a Few-Shot Setting

The aim of change detection in remote sensing usually is not to find all differences between the observations, but rather only specific types of change, such as urban development, deforestation, or even more specialized categories like roadwork. However, often there are no large public datasets available for very fine-grained tasks, and to collect the amount of training data needed for most supervised learning methods is very costly and often prohibitive. For this reason, we formulate the problem of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">few-shot filtering</i> , where we are provided with a relatively large change detection dataset and, at test time, a few instances of one particular change type that we try to “filter out” of the learned changes. For example, we might train on data of general urban change, and, given some samples of building construction, aim to only predict instances of these on the test set, all without any explicit labels for buildings in the training data. We further investigate a fine-tuning approach to this problem and assess its performance on a public dataset that we adapt to be used in this novel setting.

Progressive Parsing and Commonality Distillation for Few-Shot Remote Sensing Segmentation

In recent years, few-shot segmentation (FSS) has received widespread attention from scholars by virtue of its superiority in low-data regimes. Most existing research focuses on natural image processing, and very few studies are dedicated to the practical but challenging topic of remote sensing image understanding. Related experimental results show that directly transferring the previously proposed framework to the current domain is prone to produce unsatisfactory results with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">incomplete objects</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">irrelevant distractors</i> . Such phenomena can be attributed to the lack of modules specifically designed for the complex characteristics of remote sensing images, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., great intra-class diversity and low target-background contrast. In this paper, we propose a conceptually simple and easy-to-implement framework to tackle the aforementioned problems. Specifically, our innovative design embodies two main aspects: i) the support mask is progressively parsed into multiple valuable sub-regions that can be further exploited to compute local descriptors with segmentation cues about intractable parts; ii) the base-class memories stored in the meta-training phase are replayed and leveraged for the distillation of novel-class prototypes, where the commonalities between classes are adequately explored, more in line with the concept of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">learning to learn</i> . These two components, i.e., the progressive parsing module and commonality distillation module, contribute to each other and together constitute the proposed PCNet. We conduct extensive experiments on the standard benchmark to evaluate segmentation performance in few-shot settings. Quantitative and qualitative results illustrate that our PCNet distinctly outperforms previous FSS approaches and sets a new state-of-the-art.

A Global Model Approach to Robust Few-Shot SAR Automatic Target Recognition

In real-world scenarios, it may not always be possible to collect hundreds of labeled samples per class for training deep learning-based SAR Automatic Target Recognition (ATR) models. This work specifically tackles the few-shot SAR ATR problem, where only a handful of labeled samples may be available to support the task of interest. Our approach is composed of two stages. In the first, a global representation model is trained via self-supervised learning on a large pool of diverse and unlabeled SAR data. In the second stage, the global model is used as a fixed feature extractor and a classifier is trained to partition the feature space given the few-shot support samples, while simultaneously being calibrated to detect anomalous inputs. Unlike competing approaches which require a pristine labeled dataset for pretraining via meta-learning, our approach learns highly transferable features from unlabeled data that have little-to-no relation to the downstream task. We evaluate our method in standard and extended MSTAR operating conditions and find it to achieve high accuracy and robust out-of-distribution detection in many different few-shot settings. Our results are particularly significant because they show the merit of a global model approach to SAR ATR, which makes minimal assumptions, and provides many axes for extendability.

Revisiting Relation Extraction in the era of Large Language Models.

Relation extraction (RE) is the core NLP task of inferring semantic relationships between entities from text. Standard supervised RE techniques entail training modules to tag tokens comprising entity spans and then predict the relationship between them. Recent work has instead treated the problem as a sequence-to-sequence task, linearizing relations between entities as target strings to be generated conditioned on the input. Here we push the limits of this approach, using larger language models (GPT-3 and Flan-T5 large) than considered in prior work and evaluating their performance on standard RE tasks under varying levels of supervision. We address issues inherent to evaluating generative approaches to RE by doing human evaluations, in lieu of relying on exact matching. Under this refined evaluation, we find that: (1) Few-shot prompting with GPT-3 achieves near SOTA performance, i.e., roughly equivalent to existing fully supervised models; (2) Flan-T5 is not as capable in the few-shot setting, but supervising and fine-tuning it with Chain-of-Thought (CoT) style explanations (generated via GPT-3) yields SOTA results. We release this model as a new baseline for RE tasks.

APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation

Few-shot semantic segmentation aims to segment novel-class objects in a given query image with only a few labeled support images. Most advanced solutions exploit a metric learning framework that performs segmentation through matching each query feature to a learned class-specific prototype. However, this framework suffers from biased classification due to incomplete feature comparisons. To address this issue, we present an adaptive prototype representation by introducing class-specific and class-agnostic prototypes and thus construct complete sample pairs for learning semantic alignment with query features. The complementary features learning manner effectively enriches feature comparison and helps yield an unbiased segmentation model in the few-shot setting. It is implemented with a two-branch end-to-end network (i.e., a class-specific branch and a class-agnostic branch), which generates prototypes and then combines query features to perform comparisons. In addition, the proposed class-agnostic branch is simple yet effective. In practice, it can adaptively generate multiple class-agnostic prototypes for query images and learn feature alignment in a self-contrastive manner. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ demonstrate the superiority of our method. At no expense of inference efficiency, our model achieves state-of-the-art results in both 1-shot and 5-shot settings for semantic segmentation.

An Adversarial Meta-Training Framework for Cross-Domain Few-Shot Learning

Meta-learning provides a promising way for deep learning models to efficiently learn in few-shot learning. With this capacity, many deep learning systems can be applied in many real applications. However, many existing meta-learning based few-shot learning systems suffer from vulnerable generalization when new tasks are from unseen domains (a.k.a, cross-domain few-shot learning). In this work, we consider this problem from the perspective of designing a model-agnostic meta-training framework to improve the generalization of existing meta-learning methods in cross-domain few-shot learning. In this way, compared with focusing on elaborately designing modules for a specific meta-learning model, our method is endowed with the ability to be compatible with different meta-learning models in various few-shot problems. To achieve this goal, a novel adversarial meta-training framework is proposed. The proposed framework utilizes max-min episodic iteration. In the episode of maximization, our framework focuses on how to dynamically generate appropriate pseudo tasks which benefit learning cross-domain knowledge. In the episode of minimization, our method aims to solve how to help meta-learning model learn cross-task and robust meta-knowledge. To comprehensively evaluate our framework, experiments are conducted on two few-shot learning settings, three meta-learning models, and eight datasets. These results demonstrate that our method is applicable to various meta-learning models in different few-shot learning problems. The superiority of our method is verified compared with existing state-of-the-art methods.

Attentive Pooling Network for Few-Shot Learning

Few-shot learning has become essential for generating neural network models that can generalize to novel classes given only a few labeled samples. Previous studies mostly focused only on building a class prototype via the relations between intra-class sample features and adopted the prototype to classify the target samples. Considering that the number of labeled samples is typically limited under few-shot settings, the use of these methods to produce representative prototypes for classification may not be efficient. To this end, in this study, we propose an attentive pooling network (APNet), which establishes the relationship between the prototype and target sample feature to highlight their important regions. APNet selectively assigns higher weights to the local features with higher relative importance scores in the prototype and the target feature map. By minimizing the classification loss through supervised learning, APNet learns to produce prototypes that are specific to the target feature based on the relative importance scores. To verify the effectiveness of APNet, we compared it with the existing methods on two popular few-shot learning datasets, and APNet outperforms the related methods by achieving 71.12% and 77.58% classification accuracy in the 5-shot setting on the miniImageNet and CUB datasets, respectively.

Generalized few-shot object detection in remote sensing images

Recently few-shot object detection (FSOD) in remote sensing images (RSIs) has drawn increasing attention. However, the current FSOD methods in RSIs merely focus on the detection performance of few-shot novel classes while ignoring the severe degradation of the base class performance. Generalized few-shot object detection (G-FSOD) aims to solve the FSOD problem without forgetting previous knowledge. In this paper, we focus on the G-FSOD in RSIs and propose a Generalized Few-Shot Detector (G-FSDet) that can learn novel knowledge without forgetting. Through the comprehensive analysis of each component in the detector, a novel efficient transfer-learning framework is presented as the foundation of our G-FSDet, which is more suitable for FSOD in remote sensing scenes. Considering the greater intra-class diversity and lower inter-class separability of geospatial objects, we design a metric-based discriminative loss to learn a more discriminative classifier in the few-shot fine-tuning stage. Furthermore, a representation compensation module is proposed to alleviate the catastrophic forgetting problem by decoupling the representation learning of previous and novel knowledge. Extensive experiments on DIOR and NWPU VHR-10.v2 datasets demonstrate that our proposed G-FSDet achieves competitive novel class performance with minor degradation in the base class, reaching state-of-the-art overall performance among all few-shot settings. The source code is available at (https://github.com/RSer-XDU/G-FSDet).

Hierarchical message-passing graph neural networks

Graph Neural Networks (GNNs) have become a prominent approach to machine learning with graphs and have been increasingly applied in a multitude of domains. Nevertheless, since most existing GNN models are based on flat message-passing mechanisms, two limitations need to be tackled: (i) they are costly in encoding long-range information spanning the graph structure; (ii) they are failing to encode features in the high-order neighbourhood in the graphs as they only perform information aggregation across the observed edges in the original graph. To deal with these two issues, we propose a novel Hierarchical Message-passing Graph Neural Networks framework. The key idea is generating a hierarchical structure that re-organises all nodes in a flat graph into multi-level super graphs, along with innovative intra- and inter-level propagation manners. The derived hierarchy creates shortcuts connecting far-away nodes so that informative long-range interactions can be efficiently accessed via message passing and incorporates meso- and macro-level semantics into the learned node representations. We present the first model to implement this framework, termed Hierarchical Community-aware Graph Neural Network (HC-GNN), with the assistance of a hierarchical community detection algorithm. The theoretical analysis illustrates HC-GNN’s remarkable capacity in capturing long-range information without introducing heavy additional computation complexity. Empirical experiments conducted on 9 datasets under transductive, inductive, and few-shot settings exhibit that HC-GNN can outperform state-of-the-art GNN models in network analysis tasks, including node classification, link prediction, and community detection. Moreover, the model analysis further demonstrates HC-GNN’s robustness facing graph sparsity and the flexibility in incorporating different GNN encoders.

Aggregating pairwise semantic differences for few-shot claim verification.

As part of an automated fact-checking pipeline, the claim verification task consists in determining if a claim is supported by an associated piece of evidence. The complexity of gathering labelled claim-evidence pairs leads to a scarcity of datasets, particularly when dealing with new domains. In this article, we introduce Semantic Embedding Element-wise Difference (SEED), a novel vector-based method to few-shot claim verification that aggregates pairwise semantic differences for claim-evidence pairs. We build on the hypothesis that we can simulate class representative vectors that capture average semantic differences for claim-evidence pairs in a class, which can then be used for classification of new instances. We compare the performance of our method with competitive baselines including fine-tuned Bidirectional Encoder Representations from Transformers (BERT)/Robustly Optimized BERT Pre-training Approach (RoBERTa) models, as well as the state-of-the-art few-shot claim verification method that leverages language model perplexity. Experiments conducted on the Fact Extraction and VERification (FEVER) and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings. Our code is available.

An effective industrial defect classification method under the few-shot setting via two-stream training

Visual surface defect inspection provides an important tool for product quality assessment in a wide range of industrial applications. In recent years, diverse convolutional neural network (CNN) models have been developed for the high-accuracy image classification of natural objects. However, it remains a challenging task to develop deep-learning-based approaches for surface defect inspection of industrial products. The main challenge is that surface defects of industrial products occur in low probability, therefore it is impractical to build a large image dataset containing different types of defects to train/fine-tune deep CNN models to achieve satisfactory performance when applied to industrial inspection tasks. To overcome the above-mentioned problem, we build a light-weight defect classification model based on the pre-trained SqueezeNet architecture and present three effective techniques to achieve high-accuracy defect classification using very few collected defective training samples. First, we experimentally demonstrate that it is reasonable to freeze shallower convolutional layers and only fine-tune deeper layers in the last convolutional stage when there are not enough training samples. Second, we integrate batch-size independent Group Normalization (GN) into SqueezeNet to compute stable statistics based on a limited number of defective samples for training the classification model. Third, we add an auxiliary task to perform the similarity comparison of two defective samples during the training phase, significantly increasing the size of the training data and generating more distinctive defect-specific representations. The experimental results on the NEU-CLS dataset and USB-FS dataset verify the effectiveness of the proposed method. Our method can obtain accurate defect classification results when very few defective training samples are provided, i.e., about 97.69% and 82.92% average classification accuracy on the NEU-CLS and USB-FS datasets, respectively, under the 5-shot setting (5 images per defect category).

Thermal to Visual Person Re-Identification Using Collaborative Metric Learning Based on Maximum Margin Matrix Factorization

Thermal to visual person re-identification (T2V-ReID) is a cross-domain image retrieval problem. In this problem, the matching of a person’s image takes place, where the image is taken by different cameras (thermal and visual) at different times. This problem has numerous applications in night-time security surveillance. It is challenging due to the large intra-class variations and cross-domain discrepancies. Recently, deep metric learning methods are proposed for this problem. Still, there is a scope to improve the metric learning by generalizing the metric. In this paper, we have proposed the collaborative metric learning using Maximum Margin Matrix Factorization. It uses the group-wise similarities and collaboratively predicts the similarities. We can learn a more generalized metric by utilizing the maximized margin in this method. The proposed method is tested on the RegDB and RGB-D-T data sets, and the method outperforms the existing works in the few-shot learning settings.

FewJoint: few-shot learning for joint dialogue understanding.

Few-shot learning (FSL) is one of the key future steps in machine learning and raises a lot of attention. In this paper, we focus on the FSL problem of dialogue understanding, which contains two closely related tasks: intent detection and slot filling. Dialogue understanding has been proven to benefit a lot from jointly learning the two sub-tasks. However, such joint learning becomes challenging in the few-shot scenarios: on the one hand, the sparsity of samples greatly magnifies the difficulty of modeling the connection between the two tasks; on the other hand, how to jointly learn multiple tasks in the few-shot setting is still less investigated. In response to this, we introduce FewJoint, the first FSL benchmark for joint dialogue understanding. FewJoint provides a new corpus with 59 different dialogue domains from real industrial API and a code platform to ease FSL experiment set-up, which are expected to advance the research of this field. Further, we find that insufficient performance of the few-shot setting often leads to noisy sharing between two sub-task and disturbs joint learning. To tackle this, we guide slot with explicit intent information and propose a novel trust gating mechanism that blocks low-confidence intent information to ensure high quality sharing. Besides, we introduce a Reptile-based meta-learning strategy to achieve better generalization in unseen few-shot domains. In the experiments, the proposed method brings significant improvements on two datasets and achieve new state-of-the-art performance.

Few-shot object detection with semantic enhancement and semantic prototype contrastive learning

Few-shot object detection (FSOD), which aims to teach machines to detect objects belonging to novel classes via extremely few annotated data, has attracted extensive research interest. However, the performance of FSOD is still limited by the lack of data. Visual information of novel objects has significant intraclass variance under the few-shot setting, so single visual information cannot accurately represent the objects themselves. In contrast, humans are good at combining visual and semantic systems to recognize new concepts simultaneously. In this paper, we fully explore utilizing additional semantic knowledge to assist the FSOD task. Concretely, we first obtain the semantic representation of classes by the word embedding model learned from a large corpus of text. We then design a semantic enhancement (SE) module to enhance the incomprehensively visual representation of novel classes. To further improve the classification performance, we define a semantic prototype contrastive (SPC) loss to learn a more discriminative embedding space, where features to be detected belonging to the same class are compactly clustered around the corresponding semantic representation. Furthermore, we also introduce the semantic margin between different semantic representations for SPC loss to adaptively separate the margin between features belonging to different classes. Extensive experiments on the PASCAL VOC and MS-COCO benchmarks demonstrate that the proposed method achieves state-of-the-art performance.

A transfer learning approach to few-shot segmentation of novel white matter tracts.

Convolutional neural networks(CNNs) have achieved state-of-the-art performance for white matter(WM) tract segmentation based on diffusion magnetic resonance imaging(dMRI). The training of the CNN-based segmentation model generally requires a large number of manual delineations of WM tracts, which can be expensive and time-consuming. Although it is possible to carefully curate abundant training data for a set of WM tracts of interest, there can also be novel WM tracts-i.e., WM tracts that are not included in the existing annotated WM tracts-that are specific to a new scientific problem, and it is desired that the novel WM tracts can be segmented without repeating the laborious collection of a large number of manual delineations for these tracts. One possible solution to the problem is to transfer the knowledge learned for segmenting existing WM tracts to the segmentation of novel WM tracts with a fine-tuning strategy, where a CNN pretrained for segmenting existing WM tracts is fine-tuned with only a few annotated scans to segment the novel WM tracts. However, in classic fine-tuning, the information in the last task-specific layer for segmenting existing WM tracts is completely discarded. In this work, based on the pretraining and fine-tuning framework, we propose an improved transfer learning approach to the segmentation of novel WM tracts in the few-shot setting, where all knowledge in the pretrained model is incorporated into the fine-tuning procedure. Specifically, from the weights of the pretrained task-specific layer for segmenting existing WM tracts, we derive a better initialization of the last task-specific layer for the target model that segments novel WM tracts. In addition, to allow further improvement of the initialization of the last layer and thus the segmentation performance in the few-shot setting, we develop a simple yet effective data augmentation strategy that generates synthetic annotated images with tract-aware image mixing. To validate the proposed method, we performed experiments on brain dMRI scans from public and private datasets under various experimental settings, and the results indicate that our method improves the performance of few-shot segmentation of novel WM tracts.

Revisiting Local Descriptor for Improved Few-Shot Classification

Few-shot classification studies the problem of quickly adapting a deep learner to understanding novel classes based on few support images. In this context, recent research efforts have been aimed at designing more and more complex classifiers that measure similarities between query and support images but left the importance of feature embeddings seldom explored. We show that the reliance on sophisticated classifiers is not necessary, and a simple classifier applied directly to improved feature embeddings can instead outperform most of the leading methods in the literature. To this end, we present a new method, named DCAP, for few-shot classification, in which we investigate how one can improve the quality of embeddings by leveraging Dense Classification and Attentive Pooling (DCAP) . Specifically, we propose to train a learner on base classes with abundant samples to solve dense classification problem first and then meta-train the learner on plenty of randomly sampled few-shot tasks to adapt it to few-shot scenario or the test time scenario. During meta-training, we suggest to pool feature maps by applying attentive pooling instead of the widely used global average pooling to prepare embeddings for few-shot classification. Attentive pooling learns to reweight local descriptors, explaining what the learner is looking for as evidence for decision making. Experiments on two benchmark datasets show the proposed method to be superior in multiple few-shot settings while being simpler and more explainable. Code is publicly available at https://github.com/Ukeyboard/dcap/ .

Search and Learn: Improving Semantic Coverage for Data-to-Text Generation

Data-to-text generation systems aim to generate text descriptions based on input data (often represented in the tabular form). A typical system uses huge training samples for learning the correspondence between tables and texts. However, large training sets are expensive to obtain, limiting the applicability of these approaches in real-world scenarios. In this work, we focus on few-shot data-to-text generation. We observe that, while fine-tuned pretrained language models may generate plausible sentences, they suffer from the low semantic coverage problem in the few-shot setting. In other words, important input slots tend to be missing in the generated text. To this end, we propose a search-and-learning approach that leverages pretrained language models but inserts the missing slots to improve the semantic coverage. We further finetune our system based on the search results to smooth out the search noise, yielding better-quality text and improving inference efficiency to a large extent. Experiments show that our model achieves high performance on E2E and WikiBio datasets. Especially, we cover 98.35% of input slots on E2E, largely alleviating the low coverage problem.

Switch-GPT: An Effective Method for Constrained Text Generation under Few-Shot Settings (Student Abstract)

In real-world applications of natural language generation, target sentences are often required to satisfy some lexical constraints. However, the success of most neural-based models relies heavily on data, which is infeasible for data-scarce new domains. In this work, we present FewShotAmazon, the first benchmark for the task of Constrained Text Generation under few-shot settings on multiple domains. Further, we propose the Switch-GPT model, in which we utilize the strong language modeling capacity of GPT-2 to generate fluent and well-formulated sentences, while using a light attention module to decide which constraint to attend to at each step. Experiments show that the proposed Switch-GPT model is effective and remarkably outperforms the baselines. Codes will be available at https://github.com/chang-github-00/Switch-GPT.