Few-shot Segmentation Research Articles

Task-Disruptive Background Suppression for Few-Shot Segmentation

Few-shot segmentation aims to accurately segment novel target objects within query images using only a limited number of annotated support images. The recent works exploit support background as well as its foreground to precisely compute the dense correlations between query and support. However, they overlook the characteristics of the background that generally contains various types of objects. In this paper, we highlight this characteristic of background which can bring problematic cases as follows: (1) when the query and support backgrounds are dissimilar and (2) when objects in the support background are similar to the target object in the query. Without any consideration of the above cases, adopting the entire support background leads to a misprediction of the query foreground as background. To address this issue, we propose Task-disruptive Background Suppression(TBS), a module to suppress those disruptive support background features based on two spatial-wise scores: query-relevant and target-relevant scores. The former aims to mitigate the impact of unshared features solely existing in the support background, while the latter aims to reduce the influence of target-similar support background features. Based on these two scores, we define a query background relevant score that captures the similarity between the backgrounds of the query and the support, and utilize it to scale support background features to adaptively restrict the impact of disruptive support backgrounds. Our proposed method achieves state-of-the-art performance on standard few-shot segmentation benchmarks. Our official code is available at github.com/SuhoPark0706/TBSNet.

Multi-Content Interaction Network for Few-Shot Segmentation

Few-Shot Segmentation (FSS) poses significant challenges due to limited support images and large intra-class appearance discrepancies. Most existing approaches focus on aligning the support-query correlations from the same layer of the frozen backbone while neglecting the bias between different tasks and different layers. In this article, we propose a Multi-Content Interaction Network (MCINet) to remedy these issues by fully exploiting and interacting with the different contextual information contained in distinct branches. Specifically, MCINet improves FSS from three perspectives: (1) boosting the query representations through incorporating the independent information from another learnable branch into the features from the frozen backbone, (2) enhancing the support-query correlations by exploiting both the same-layer and adjacent-layer features, and (3) refining the predicted results with a multi-scale mask prediction strategy. Experiments on three benchmarks demonstrate that our approach reaches state-of-the-art performances and outperforms the best competitors with many desirable advantages, especially on the challenging COCO dataset. Code will be released on GitHub ( https://github.com/chenhao-zju/mcinet ).

Few-shot segmentation based on multi-level and cross-scale clustering

The problem of image segmentation with few-shot learning is addressed in this paper, which is a challenging task due to the lack of sufficient high-precision annotated data. A novel method that consists of two modules is proposed: a multi-level fuzzy clustering guidance module and a cross-scale feature fusion module. The former module can extract image features in a class-independent feature space and fuse them with different scale information, while the latter module can reduce the information loss caused by cross-scale transmission. The feature association map between the support image and the query image can be learned by the proposed method, and the inconsistency of target object categories can be overcome. The proposed method is evaluated on Pascal and COCO datasets, and it is shown that it outperforms the state-of-the-art algorithms in both one-shot and k-shot segmentation scenarios.

A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation

Few-shot segmentation focuses on the generalization of models to segment unseen object with limited annotated samples. However, existing approaches still face two main challenges. First, huge feature distinction between support and query images causes knowledge transferring barrier, which harms the segmentation performance. Second, limited support prototypes cannot adequately represent features of support objects, hard to guide high-quality query segmentation. To deal with the above two issues, we propose self-distillation embedded supervised affinity attention model to improve the performance of few-shot segmentation task. Specifically, the self-distillation guided prototype module uses self-distillation to align the features of support and query. The supervised affinity attention module generates high-quality query attention map to provide sufficient object information. Extensive experiments prove that our model significantly improves the performance compared to existing methods. Comprehensive ablation experiments and visualization studies also show the significant effect of our method on few-shot segmentation task. On COCO-20i dataset, we achieve new state-of-the-art results. Training code and pretrained models are available at https://github.com/cv516Buaa/SD-AANet.

Learning self-target knowledge for few-shot segmentation

Few-shot semantic segmentation uses a few annotated data of a specific class in the support set to segment the target of the same class in the query set. Most existing approaches fail to perform well when there are significant intra-class variances. This paper alleviates the problem by concentrating on mining the query image and using the support set as supplementary information. First, it proposes a Query Prototype Generation Module to generate a query foreground prototype from the query features. Specifically, we use both prototype-level and pixel-level similarity matching to generate two complementary initial prototypes, which we then integrate to create a discriminative query foreground prototype. Second, we propose a Support Auxiliary Refinement Module to further guide the final precise prediction of the query image by leveraging the target category information of the support set through step-by-step mining. Specifically, we generate a query-support mixture prototype based on the support prototype representation obtained using the attention mechanism. Then we generate a support supplement prototype to complement the missing information by encoding over the foreground regions that the query-support mixture prototype fails to segment out. Extensive experiments on PASCAL-5i and COCO-20i demonstrate that our model outperforms the prior works of few-shot segmentation.

Few-shot segmentation framework for lung nodules via an optimized active contour model.

Accurate segmentation of lung nodules is crucial for the early diagnosis and treatment of lung cancer in clinical practice. However, the similarity between lung nodules and surrounding tissues has made their segmentation a longstanding challenge. Existing deep learning and active contour models each have their limitations. This paper aims to integrate the strengths of both approaches while mitigating their respective shortcomings. In this paper, we propose a few-shot segmentation framework that combines a deep neural network with an active contour model. We introduce heat kernel convolutions and high-order total variation into the active contour model and solve the challenging nonsmooth optimization problem using the alternating direction method of multipliers. Additionally, we use the presegmentation results obtained from training a deep neural network on a small sample set as the initial contours for our optimized active contour model, addressing the difficulty of manually setting the initial contours. We compared our proposed method with state-of-the-art methods for segmentation effectiveness using clinical computed tomography (CT) images acquired from two different hospitals and the publicly available LIDC dataset. The results demonstrate that our proposed method achieved outstanding segmentation performance according to both visual and quantitative indicators. Our approach utilizes the output of few-shot network training as prior information, avoiding the need to select the initial contour in the active contour model. Additionally, it provides mathematical interpretability to the deep learning, reducing its dependency on the quantity of training samples.

Few-shot segmentation using distribution adaption networks

Few-shot segmentation aims to achieve object-specific semantic segmentation in query images by conditioning on specific objects identified within support images. Previous methodologies predominantly relied on evaluating similarities between semantic-level prototypes of support features or comprehensive pixelwise support information and query images to inform predictions for queries. However, a significant domain discrepancy has been identified between the extracted query features and their support counterparts, resulting in a significant misalignment in the distribution of support and query features. As a result, direct analogies between support and query information cannot ensure the precision of query predictions. In this investigation, a distribution adaptation network is introduced, meticulously designed to learn discriminative representations and alleviate the distributional disparities between support features and query features. The adaptation network embeds both support and query features into Reproducing Kernel Hilbert Spaces (RKHS) and optimally aligns their distributions. The distributional gap is effectively bridged through optimal kernel embedding alignment, facilitated by a multiple kernel learning strategy. Extensive experimentation on datasets PASCAL-5i, COCO-20i and FSS-1000 unequivocally demonstrates that the proposed Kernel-Aligned Distribution Network (KADN) achieves state-of-the-art performance.

Prompt-and-Transfer: Dynamic Class-aware Enhancement for Few-shot Segmentation.

For more efficient generalization to unseen domains (classes), most Few-shot Segmentation (FSS) would directly exploit pre-trained encoders and only fine-tune the decoder, especially in the current era of large models. However, such fixed feature encoders tend to be class-agnostic, inevitably activating objects that are irrelevant to the target class. In contrast, humans can effortlessly focus on specific objects in the line of sight. This paper mimics the visual perception pattern of human beings and proposes a novel and powerful prompt-driven scheme, called "Prompt and Transfer" (PAT), which constructs a dynamic class-aware prompting paradigm to tune the encoder for focusing on the interested object (target class) in the current task. Three key points are elaborated to enhance the prompting: 1) Cross-modal linguistic information is introduced to initialize prompts for each task. 2) Semantic Prompt Transfer (SPT) that precisely transfers the class-specific semantics within the images to prompts. 3) Part Mask Generator (PMG) that works in conjunction with SPT to adaptively generate different but complementary part prompts for different individuals. Surprisingly, PAT achieves competitive performance on 4 different tasks including standard FSS, Cross-domain FSS (e.g., CV, medical, and remote sensing domains), Weak-label FSS, and Zero-shot Segmentation, setting new state-of-the-arts on 11 benchmarks.

Mask-Guided Correlation Learning for Few-Shot Segmentation in Remote Sensing Imagery

Mask-Guided Correlation Learning for Few-Shot Segmentation in Remote Sensing Imagery

Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyond

Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyond

MSMTSeg: Multi-Stained Multi-Tissue Segmentation of Kidney Histology Images via Generative Self-Supervised Meta-Learning Framework.

Accurately diagnosing chronic kidney disease requires pathologists to assess the structure of multiple tissues under different stains, a process that is timeconsuming and labor-intensive. Current AI-based methods for automatic structure assessment, like segmentation, often demand extensive manual annotation and focus on single stain domain. To address these challenges, we introduce MSMTSeg, a generative self-supervised meta-learning framework for multi-stained multi-tissue segmentation in renal biopsy whole slide images (WSIs). MSMTSeg incorporates multiple stain transform models for style translation of inter-stain domains, a self-supervision module for obtaining pre-trained models with the domain-specific feature representation, and a meta-learning strategy that leverages generated virtual data and pre-trained models to learn the domain-invariant feature representation across multiple stains, thereby enhancing segmentation performance. Experimental results demonstrate that MSMTSeg achieves superior and robust performance, with mDSC of 0.836 and mIoU of 0.718 for multiple tissues under different stains, using only one annotated training sample for each stain. Our ablation study confirms the effectiveness of each component, positioning MSMTSeg ahead of classic advanced segmentation networks, recent few-shot segmentation methods, and unsupervised domain adaptation methods. In conclusion, our proposed few-shot cross-domain technology offers a feasible and cost-effective solution for multi-stained renal histology segmentation, providing convenient assistance to pathologists in clinical practice. The source code and conditionally accessible data are available at https://github.com/SnowRain510/MSMTSeg.

Prototype Correlation Matching and Class-Relation Reasoning for Few-Shot Medical Image Segmentation.

Few-shot medical image segmentation has achieved great progress in improving accuracy and efficiency of medical analysis in the biomedical imaging field. However, most existing methods cannot explore inter-class relations among base and novel medical classes to reason unseen novel classes. Moreover, the same kind of medical class has large intra-class variations brought by diverse appearances, shapes and scales, thus causing ambiguous visual characterization to degrade generalization performance of these existing methods on unseen novel classes. To address the above challenges, in this paper, we propose a Prototype correlation Matching and Class-relation Reasoning (i.e., PMCR) model. The proposed model can effectively mitigate false pixel correlation matches caused by large intra-class variations while reasoning inter-class relations among different medical classes. Specifically, in order to address false pixel correlation match brought by large intra-class variations, we propose a prototype correlation matching module to mine representative prototypes that can characterize diverse visual information of different appearances well. We aim to explore prototypelevel rather than pixel-level correlation matching between support and query features via optimal transport algorithm to tackle false matches caused by intra-class variations. Meanwhile, in order to explore inter-class relations, we design a class-relation reasoning module to segment unseen novel medical objects via reasoning inter-class relations between base and novel classes. Such inter-class relations can be well propagated to semantic encoding of local query features to improve few-shot segmentation performance. Quantitative comparisons illustrates the large performance improvement of our model over other baseline methods.

Dual Branch Multi-Level Semantic Learning for Few-Shot Segmentation.

Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples in support images. Although progress has been made recently by combining prototype-based metric learning, existing methods still face two main challenges. First, various intra-class objects between the support and query images or semantically similar inter-class objects can seriously harm the segmentation performance due to their poor feature representations. Second, the latent novel classes are treated as the background in most methods, leading to a learning bias, whereby these novel classes are difficult to correctly segment as foreground. To solve these problems, we propose a dual-branch learning method. The class-specific branch encourages representations of objects to be more distinguishable by increasing the inter-class distance while decreasing the intra-class distance. In parallel, the class-agnostic branch focuses on minimizing the foreground class feature distribution and maximizing the features between the foreground and background, thus increasing the generalizability to novel classes in the test stage. Furthermore, to obtain more representative features, pixel-level and prototype-level semantic learning are both involved in the two branches. The method is evaluated on PASCAL- 5i 1 -shot, PASCAL- 5i 5 -shot, COCO- 20i 1 -shot, and COCO- 20i 5 -shot, and extensive experiments show that our approach is effective for few-shot semantic segmentation despite its simplicity.

SCTrans: Self-align and cross-align transformer for few-shot segmentation

Few-shot Semantic Segmentation (FSS) refers to train a segmentation model that can be generalized to novel categories with limited labeled images. One challenge of FSS is spatial inconsistency between support and query images, e.g., appearance and texture. Most existing methods are only committed to utilizing the semantic-level prototypes of support images to guide mask predictions. These methods, nevertheless, only focus on the most discriminate regions of the object rather than holonomic feature representations. Besides, another question exists that the lack of interaction between paired support and query images. In this paper, we propose a self-align and cross-align transformer (SCTrans) to remedy the above limitations. Specifically, we design a feature fusion module (FFM) to incorporate low-level information from the query branch into mid-level semantic features, boosting the semantic representations of query images. In addition, a feature alignment module is designed to bidirectionally propagate semantic information from support to query images conditioned on more representative support and query features, increasing both intra-class similarities and inter-class differences. Extensive experiments on PASCAL-5i and COCO-20i show that our SCTrans significantly advances the state-of-the-art methods.

A novel deep learning method for on-line monitoring of grate bar defects in sintering machine system

The defective grate bars at the bottom of the sintering machine system pose a serious safety risk to steel enterprises. Cracking and missing of these grate bars lead to severe production accidents and economic losses. Implementation of an efficient defect monitoring system plays a crucial role in ensuring continuous and stable sintering production. This paper presents a novel on-line grate bar defects monitoring method to address the following two key challenges. Firstly, a light-weight location model called Multi Scale Separated Convolutional Attention Network-LiteSeg (MSSCAN-LiteSeg) is proposed to identify the optimal shooting position of the moving trolley and simultaneously segment the horizontal and vertical grate bars. By separating the attention calculation into independent channel spaces, MSSCAN-LiteSeg can effectively handle the imbalanced sample distributions. Secondly, an innovative few-shot segmentation network named Response Forgetting Attention-Hypercorrelation Squeeze Network (RFA-HSNet) is proposed for defect detection by employing a "Forgetting-Enhancing-Recalling" method. Extensive experiments have demonstrated that our method can achieve a precision rate of 93.3% and provide a satisfactory performance in the on-line grate bar defects monitoring system.

A learnable support selection scheme for boosting few-shot segmentation

Upon reevaluating recent studies of Few-Shot Segmentation (FSS), a key observation is that the random selection of support images is not always the optimal. In this situation, the support images cannot provide the useful guidance for the segmentation task. Therefore, we argue that a similarity-based support selection scheme, which selects support images according to the similarity between the query and candidate support images, is able to boost the performance of an FSS network. To this end, we propose a Siamese Support Selection Network (SSSN) which can be end-to-end trained along with an FSS network. We also leverage the joint utilization of a Convolutional Neural Network (CNN) and a Transformer network on top of a new feature fusion method to further improve the performance. To our knowledge, none of the similarity-based support selection scheme and the dual-stream network have been utilized for the FSS task before. Experimental results show that our FSS approach outperforms its counterparts on three data sets. In particular, the SSSN is able to boost the performance of an FSS network. We believe that these promising results should be due to the ability of the SSSN to select the top similar support images, which are useful for the FSS task.11Code is available at https://indtlab.github.io/projects/SSSN.

Learning discriminative foreground-and-background features for few-shot segmentation

Learning discriminative foreground-and-background features for few-shot segmentation

Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration.

The prowess that makes few-shot learning desirable in medical image analysis is the efficient use of the support image data, which are labelled to classify or segment new classes, a task that otherwise requires substantially more training images and expert annotations. This work describes a fully 3D prototypical few-shot segmentation algorithm, such that the trained networks can be effectively adapted to clinically interesting structures that are absent in training, using only a few labelled images from a different institute. First, to compensate for the widely recognised spatial variability between institutions in episodic adaptation of novel classes, a novel spatial registration mechanism is integrated into prototypical learning, consisting of a segmentation head and an spatial alignment module. Second, to assist the training with observed imperfect alignment, support mask conditioning module is proposed to further utilise the annotation available from the support images. Extensive experiments are presented in an application of segmenting eight anatomical structures important for interventional planning, using a data set of 589 pelvic T2-weighted MR images, acquired at seven institutes. The results demonstrate the efficacy in each of the 3D formulation, the spatial registration, and the support mask conditioning, all of which made positive contributions independently or collectively. Compared with the previously proposed 2D alternatives, the few-shot segmentation performance was improved with statistical significance, regardless whether the support data come from the same or different institutes.

One-shot segmentation of novel white matter tracts via extensive data augmentation and adaptive knowledge transfer.

The use of convolutional neural networks (CNNs) has allowed accurate white matter (WM) tract segmentation on diffusion magnetic resonance imaging (dMRI). To train the CNN-based segmentation models, a large number of scans on which WM tracts are annotated need to be collected, and these annotated scans can be accumulated over a long period of time. However, when novel WM tracts that are different from existing annotated WM tracts are of interest, additional annotations are required for their segmentation. Due to the cost of manual annotations, methods have been developed for few-shot segmentation of novel WM tracts, where the segmentation knowledge is transferred from existing WM tracts to novel WM tracts and the amount of annotated data for novel WM tracts is reduced. Despite these developments, it is desirable to further reduce the amount of annotated data to the one-shot setting with a single annotated image. To address this problem, we develop an approach to one-shot segmentation of novel WM tracts. Our method follows the existing pretraining/fine-tuning framework that transfers segmentation knowledge from existing to novel WM tracts. First, as there is extremely scarce annotated data in the one-shot setting, we design several different data augmentation strategies so that extensive data augmentation can be performed to obtain extra synthetic training data. The data augmentation strategies are based on image masking and thus applicable to the one-shot setting. Second, to address overfitting and knowledge forgetting in the fine-tuning stage that can be more severe given limited training data, we propose an adaptive knowledge transfer strategy that selects the network weights to be updated. The data augmentation and adaptive knowledge transfer strategies are combined to train the segmentation model. Considering that the different data augmentation strategies can generate synthetic data that contain potentially conflicting information, we apply the data augmentation strategies separately, each leading to a different segmentation model. The results predicted by the different models are fused to produce the final segmentation. We validated our method on two brain dMRI datasets, including a public dataset and an in-house dataset. Different settings were considered for the validation, and the results show that the proposed method improves the one-shot segmentation of novel WM tracts.

Distilling base-and-meta network with contrastive learning for few-shot semantic segmentation

Current studies in few-shot semantic segmentation mostly utilize meta-learning frameworks to obtain models that can be generalized to new categories. However, these models trained on base classes with sufficient annotated samples are biased towards these base classes, which results in semantic confusion and ambiguity between base classes and new classes. A strategy is to use an additional base learner to recognize the objects of base classes and then refine the prediction results output by the meta learner. In this way, the interaction between these two learners and the way of combining results from the two learners are important. This paper proposes a new model, namely Distilling Base and Meta (DBAM) network by using self-attention mechanism and contrastive learning to enhance the few-shot segmentation performance. First, the self-attention-based ensemble module (SEM) is proposed to produce a more accurate adjustment factor for improving the fusion of two predictions of the two learners. Second, the prototype feature optimization module (PFOM) is proposed to provide an interaction between the two learners, which enhances the ability to distinguish the base classes from the target class by introducing contrastive learning loss. Extensive experiments have demonstrated that our method improves on the PASCAL-5i under 1-shot and 5-shot settings, respectively.