Few-shot Segmentation Research Articles

IMU Airtime Detection in Snowboard Halfpipe: U-Net Deep Learning Approach Outperforms Traditional Threshold Algorithms.

Airtime is crucial for high-rotation tricks in snowboard halfpipe performance, significantly impacting trick difficulty, the primary judging criterion. This study aims to enhance the detection of take-off and landing events using inertial measurement unit (IMU) data in conjunction with machine learning algorithms since manual video-based methods are too time-consuming. Eight elite German National Team snowboarders performed 626 halfpipe tricks, recorded by two IMUs at the lateral lower legs and a video camera. The IMU data, synchronized with video, were labeled manually and segmented for analysis. Utilizing a 1D U-Net convolutional neural network (CNN), we achieved superior performance in all of our experiments, establishing new benchmarks for this binary segmentation task. In our extensive experiments, we achieved an 80.34% lower mean Hausdorff distance for unseen runs compared with the threshold approach when placed solely on the left lower leg. Using both left and right IMUs further improved performance (83.37% lower mean Hausdorff). For data from an algorithm-unknown athlete (Zero-Shot segmentation), the U-Net outperformed the threshold algorithm by 67.58%, and fine-tuning on athlete-specific (Few-Shot segmentation) runs improved the lower mean Hausdorff to 78.68%. The fine-tuned model detected takeoffs with median deviations of 0.008 s (IQR 0.030 s), landing deviations of 0.005 s (IQR 0.020 s), and airtime deviations of 0.000 s (IQR 0.027 s). These advancements facilitate real-time feedback and detailed biomechanical analysis, enhancing performance and trick execution, particularly during critical events, such as take-off and landing, where precise time-domain localization is crucial for providing accurate feedback to coaches and athletes.

AgMTR: Agent Mining Transformer for Few-Shot Segmentation in Remote Sensing

AgMTR: Agent Mining Transformer for Few-Shot Segmentation in Remote Sensing

On filling the intra-class and inter-class gaps for few-shot segmentation

Few-shot segmentation (FSS) has made remarkable success through prototypical learning. However, owing to the scarcity of support data, FSS methods continue to suffer from huge intra-class and inter-class gaps. In this paper, we introduce a unified network, termed FGNet++, aimed at addressing both gaps. FGNet++ comprises a Self-Adaptive Module (SAM) to emphasize the query features and generate enhanced prototypes for self-alignment. These prototypes contain the intrinsic information of each query sample, thereby mitigating the intra-class appearance gaps. Moreover, we augment SAM with a Feature Transformation Module (FTM) to further reduce intra-class appearance discrepancies. On the other hand, we introduce an Inter-class Feature Separation Module (IFSM) dedicated to bridge the inter-class gap. With a B-SLIC based background loss and a cross-category loss for training, IFSM makes it easy to distinguish the feature space of the target class from other non-target classes. Furthermore, to assess the generality of our approach, we extend our work to 3D point cloud few-shot segmentation and present FGNet-3D. Experimental results demonstrate that our method successfully mitigates both intra-class and inter-class gaps in FSS through SAM and IFSM, respectively, and achieves state-of-the-art performances on multiple datasets compared with previous approaches.

Uncertainty guided semi-supervised few-shot segmentation with prototype level fusion

Few-Shot Semantic Segmentation (FSS) aims to tackle the challenge of segmenting novel categories with limited annotated data. However, given the diversity among support-query pairs, transferring meta-knowledge to unseen categories poses a significant challenge, particularly in scenarios featuring substantial intra-class variance within an episode task. To alleviate this issue, we propose the Uncertainty Guided Adaptive Prototype Network (UGAPNet) for semi-supervised few-shot semantic segmentation. The key innovation lies in the generation of reliable pseudo-prototypes as an additional supplement to alleviate intra-class semantic bias. Specifically, we employ a shared meta-learner to produce segmentation results for unlabeled images in the pseudo-label prediction module. Subsequently, we incorporate an uncertainty estimation module to quantify the difference between prototypes extracted from query and support images, facilitating pseudo-label denoising. Utilizing these refined pseudo-label samples, we introduce a prototype rectification module to obtain effective pseudo-prototypes and generate a generalized adaptive prototype for the segmentation of query images. Furthermore, generalized few-shot semantic segmentation extends the paradigm of few-shot semantic segmentation by simultaneously segmenting both unseen and seen classes during evaluation. To address the challenge of confusion region prediction between these two categories, we further propose a novel Prototype-Level Fusion Strategy in the prototypical contrastive space. Extensive experiments conducted on two benchmarks demonstrate the effectiveness of the proposed UGAPNet and prototype-level fusion strategy. Our source code will be available on https://github.com/WHL182/UGAPNet.

Layer-Wise Mutual Information Meta-Learning Network for Few-Shot Segmentation.

The goal of few-shot segmentation (FSS) is to segment unlabeled images belonging to previously unseen classes using only a limited number of labeled images. The main objective is to transfer label information effectively from support images to query images. In this study, we introduce a novel meta-learning framework called layer-wise mutual information (LayerMI), which enhances the propagation of label information by maximizing the mutual information (MI) between support and query features at each layer. Our approach involves the utilization of a LayerMI Block based on information-theoretic co-clustering. This block performs online co-clustering on the joint probability distribution obtained from each layer, generating a target-specific attention map. The LayerMI Block can be seamlessly integrated into the meta-learning framework and applied to all convolutional neural network (CNN) layers without altering the training objectives. Notably, the LayerMI Block not only maximizes MI between support and query features but also facilitates internal clustering within the image. Extensive experiments demonstrate that LayerMI significantly enhances the performance of baseline and achieves competitive performance compared to state-of-the-art methods on three challenging benchmarks: PASCAL-5 i , COCO-20 i , and FSS-1000.

MIINet: a multi-branch information interaction network for few-shot segmentation

MIINet: a multi-branch information interaction network for few-shot segmentation

Few-shot remote sensing image segmentation based on label propagation and open-set domain adaptation

ABSTRACT Few-shot segmentation aims to segment objects in a task dataset with only a few labelled target examples, aided by a lot of labelled auxiliary datasets. Existing approaches typically assume that the task dataset and auxiliary dataset originate from the same source but have disjoint category sets. However, it is common in remote sensing image segmentation tasks that the datasets are from different sources and the category sets are often partially overlapped, where models trained on the auxiliary dataset degrade in performance on the task dataset. To address these issues, we present a few-shot segmentation method based on graph network and open-set domain adaptation. The proposed method utilizes a graph model to capture the relationships between super pixels and to predict labels for the unlabelled samples by label propagation, meanwhile, leverages open-set domain adaptation to reduce inter-domain discrepancies and enhances model transferability. The proposed method is evaluated on two publicly available datasets, and the effectiveness of the proposed approach is demonstrated by the significant improvement in accuracy compared to existing methods.

Prototype-wise self-knowledge distillation for few-shot segmentation

Few-shot segmentation was proposed to obtain segmentation results for a image with an unseen class by referring to a few labeled samples. However, due to the limited number of samples, many few-shot segmentation models suffer from poor generalization. Prototypical network-based few-shot segmentation still has issues with spatial inconsistency and prototype bias. Since the target class has different appearance in each image, some specific features in the prototypes generated from the support image and its mask do not accurately reflect the generalized features of the target class. To address the support prototype consistency issue, we put forward two modules: Data Augmentation Self-knowledge Distillation (DASKD) and Prototype-wise Regularization (PWR). The DASKD module focuses on enhancing spatial consistency by using data augmentation and self-knowledge distillation. Self-knowledge distillation helps the model acquire generalized features of the target class and learn hidden knowledge from the support images. The PWR module focuses on obtaining a more representative support prototype by conducting prototype-level loss to obtain support prototypes closer to the category center. Broad evaluation experiments on PASCAL-5i and COCO-20i demonstrate that our model outperforms the prior works on few-shot segmentation. Our approach surpasses the state of the art by 7.5% in PASCAL-5i and 4.2% in COCO-20i.

SAM-RSP: A new few-shot segmentation method based on segment anything model and rough segmentation prompts

Few-shot segmentation (FSS) aims to segment novel classes with a few labeled images. The backbones used in existing methods are pre-trained through classification tasks on the ImageNet dataset. Although these backbones can effectively perceive the semantic categories of images, they cannot accurately perceive the regional boundaries within one image, which limits the model performance. Recently, Segment Anything Model (SAM) has achieved precise image segmentation based on point or box prompts, thanks to its excellent perception of region boundaries within one image. However, it cannot effectively provide semantic information of images. This paper proposes a new few-shot segmentation method that can effectively perceive both semantic categories and regional boundaries. This method first utilizes the SAM encoder to perceive regions and obtain the query embedding. Then the support and query images are input into a backbone pre-trained on ImageNet to perceive semantics and generate a rough segmentation prompt (RSP). This query embedding is combined with the prompt to generate a pixel-level query prototype, which can better match the query embedding. Finally, the query embedding, prompt, and prototype are combined and input into the designed multi-layer prompt transformer decoder, which is more efficient and lightweight, and can provide a more accurate segmentation result. In addition, other methods can be easily combined with our framework to improve their performance. Plenty of experiments on PASCAL-5i and COCO-20i under 1-shot and 5-shot settings prove the effectiveness of our method. Our method also achieves new state-of-the-art. Codes are available at https://github.com/Jiaguang-NEU/SAM-RSP.

A Self-Supervised Few-Shot Semantic Segmentation Method Based on Multi-Task Learning and Dense Attention Computation.

Nowadays, autonomous driving technology has become widely prevalent. The intelligent vehicles have been equipped with various sensors (e.g., vision sensors, LiDAR, depth cameras etc.). Among them, the vision systems with tailored semantic segmentation and perception algorithms play critical roles in scene understanding. However, the traditional supervised semantic segmentation needs a large number of pixel-level manual annotations to complete model training. Although few-shot methods reduce the annotation work to some extent, they are still labor intensive. In this paper, a self-supervised few-shot semantic segmentation method based on Multi-task Learning and Dense Attention Computation (dubbed MLDAC) is proposed. The salient part of an image is split into two parts; one of them serves as the support mask for few-shot segmentation, while cross-entropy losses are calculated between the other part and the entire region with the predicted results separately as multi-task learning so as to improve the model's generalization ability. Swin Transformer is used as our backbone to extract feature maps at different scales. These feature maps are then input to multiple levels of dense attention computation blocks to enhance pixel-level correspondence. The final prediction results are obtained through inter-scale mixing and feature skip connection. The experimental results indicate that MLDAC obtains 55.1% and 26.8% one-shot mIoU self-supervised few-shot segmentation on the PASCAL-5i and COCO-20i datasets, respectively. In addition, it achieves 78.1% on the FSS-1000 few-shot dataset, proving its efficacy.

EFTNet: an efficient fine-tuning method for few-shot segmentation

EFTNet: an efficient fine-tuning method for few-shot segmentation

An enhanced approach for few-shot segmentation via smooth downsampling mask and label smoothing loss

Few-shot semantic segmentation aims to segment new categories with only a small number of annotated images. Previous methods mainly focused on exploiting the pixel-level correlation between the support image and the query image, combined with attention-based methods, resulting in significant advancements. In this paper, we introduce a new perspective to enhance few-shot segmentation. We identify that utilizing the bilinear interpolation method to downsample the mask leads to the loss of fine-grained information from the target features. To address this issue, we propose a Smooth Downsampling Mask (SDM) method. The SDM method is designed to retain more effective target semantic features by employing a cascaded downsampling approach with a smooth kernel for mask processing. Additionally, we propose a label smoothing loss to further enhance the performance, which provides direct guidance for low-resolution feature map optimization. Both methods can be used as plug-and-play modules for existing methods. Notably, our proposed method does not involve additional learnable parameters and is computationally efficient, thus achieving painless gains. To validate the effectiveness of our method, we take three publicly available models as baselines and conduct extensive experiments on three public benchmarks PASCAL-5i, COCO-20i and FSS-1000, and achieve considerable improvement.

Information Gap Narrowing for Point Cloud Few-Shot Segmentation

Information Gap Narrowing for Point Cloud Few-Shot Segmentation

Learning robust correlation with foundation model for weakly-supervised few-shot segmentation

Existing few-shot segmentation (FSS) only considers learning support-query correlation and segmenting unseen categories under the precise pixel masks. However, the cost of a large number of pixel masks during training is expensive. This paper considers a more challenging scenario, weakly-supervised few-shot segmentation (WS-FSS), which only provides category (i.e. image-level) labels. It requires the model to learn robust support-query information when the generated mask is inaccurate. In this work, we design a Correlation Enhancement Network (CORENet) with foundation model, which utilizes multi-information guidance to learn robust correlation. Specifically, correlation-guided transformer (CGT) utilizes self-supervised ViT tokens to learn robust correlation from both local and global perspectives. From the perspective of semantic categories, the class-guided module (CGM) guides the model to locate valuable correlations through the pre-trained CLIP. Finally, the embedding-guided module (EGM) implicitly guides the model to supplement the inevitable information loss during the correlation learning by the original appearance embedding and finally generates the query mask. Extensive experiments on PASCAL-5i and COCO-20i have shown that CORENet exhibits excellent performance compared to existing methods.

Self-support matching networks with multiscale attention for few-shot semantic segmentation

Recent advancements in Few-shot segmentation (FSS) have displayed remarkable capabilities in predicting segmentation masks for unseen class images, using only a limited number of annotated images. However, existing methods have overlooked the influence of contextual information on segmentation and primarily rely on supporting prototypes, with limited research on query prototyping. Effectively utilizing multi-scale features and query information poses a challenging problem in this domain. To address these challenges, this paper proposes a novel approach called the multi-scale and attention-based self-support prototype few-shot semantic segmentation network (MASNet). First, a multi-scale feature enhancement module is designed to obtain features at different scales to enrich global context information. Then, simple and efficient channel attention is utilized to guide the query features related to the target class. Finally, the query prototype is matched with the query features using a self-supporting matching module. This strategy efficiently captures class-based features and addresses the issue of intra-class variance in few-shot segmentation. The experimental results on Pascal-5i, COCO-20i and Abdominal MRI datasets demonstrate that the proposed method achieves remarkable robustness and improved accuracy performance.

Attentional adversarial training for few-shot medical image segmentation without annotations.

Medical image segmentation is a critical application that plays a significant role in clinical research. Despite the fact that many deep neural networks have achieved quite high accuracy in the field of medical image segmentation, there is still a scarcity of annotated labels, making it difficult to train a robust and generalized model. Few-shot learning has the potential to predict new classes that are unseen in training with a few annotations. In this study, a novel few-shot semantic segmentation framework named prototype-based generative adversarial network (PG-Net) is proposed for medical image segmentation without annotations. The proposed PG-Net consists of two subnetworks: the prototype-based segmentation network (P-Net) and the guided evaluation network (G-Net). On one hand, the P-Net as a generator focuses on extracting multi-scale features and local spatial information in order to produce refined predictions with discriminative context between foreground and background. On the other hand, the G-Net as a discriminator, which employs an attention mechanism, further distills the relation knowledge between support and query, and contributes to P-Net producing segmentation masks of query with more similar distributions as support. Hence, the PG-Net can enhance segmentation quality by an adversarial training strategy. Compared to the state-of-the-art (SOTA) few-shot segmentation methods, comparative experiments demonstrate that the proposed PG-Net provides noticeably more robust and prominent generalization ability on different medical image modality datasets, including an abdominal Computed Tomography (CT) dataset and an abdominal Magnetic Resonance Imaging (MRI) dataset.

Advanced Defect Detection in High-Speed Train Wheel Treads Using Nanoelectronic Systems

Detecting defects on the wheel tread of high-speed trains is crucial for ensuring train safety. However, the challenge lies in acquiring sufficient defect data for effective detection. To address this, we propose a novel multi-similarity based few-shot segmentation network (MSFSNet), which employs different architecture for training and detection. Taking advantage of the principle of feature tensor similarity, MSFSNet enables precise extraction tread defects. In the training phase, the network comprises three branches: query branch, support branch, and task discrimination branch. To enhance defect detection efficiency, the detection phase employs an alternative architecture, where the support branch is replaced with a database of extracted defect feature tensors. The support branch provides tread defect information to the query branch, which employs the Multi-Similarity (MS) module to integrate for defect segmentation and detection. This integration effectively mitigates the overfitting issue caused by a scarcity of tread defect images. To ensure the effectiveness of defect information from the support branch, a task discrimination branch is constructed to discriminate tasks on query images and select similar support images. The Query Segmentation Module (QSM) is devised to achieve precise detection across various tread defect scales, enabling accurate segmentation amidst background noise. Experimental testing of MSFSNet on public datasets and the collected Wheel Tread Defect Dataset (WTDD) shows promising results: on FSSD-12 dataset, MSFSNet improves few-shot segmentation mIoU by 8.9% (1-shot) and 7.0% (5-shot); its capability for few-shot defect detection was confirmed on industrial dataset industrial-5i; and achieves 99.63% accuracy in detecting wheel tread defects on WTDD.

PFMNet: Prototype-based feature mapping network for few-shot domain adaptation in medical image segmentation

Lack of data is one of the biggest hurdles for rare disease research using deep learning. Due to the lack of rare-disease images and annotations, training a robust network for automatic rare-disease image segmentation is very challenging. To address this challenge, few-shot domain adaptation (FSDA) has emerged as a practical research direction, aiming to leverage a limited number of annotated images from a target domain to facilitate adaptation of models trained on other large datasets in a source domain. In this paper, we present a novel prototype-based feature mapping network (PFMNet) designed for FSDA in medical image segmentation. PFMNet adopts an encoder–decoder structure for segmentation, with the prototype-based feature mapping (PFM) module positioned at the bottom of the encoder–decoder structure. The PFM module transforms high-level features from the target domain into the source domain-like features that are more easily comprehensible by the decoder. By leveraging these source domain-like features, the decoder can effectively perform few-shot segmentation in the target domain and generate accurate segmentation masks. We evaluate the performance of PFMNet through experiments on three typical yet challenging few-shot medical image segmentation tasks: cross-center optic disc/cup segmentation, cross-center polyp segmentation, and cross-modality cardiac structure segmentation. We consider four different settings: 5-shot, 10-shot, 15-shot, and 20-shot. The experimental results substantiate the efficacy of our proposed approach for few-shot domain adaptation in medical image segmentation.

A Mutually Supervised Graph Attention Network for Few-Shot Segmentation: The Perspective of Fully Utilizing Limited Samples.

Fully supervised semantic segmentation has performed well in many computer vision tasks. However, it is time-consuming because training a model requires a large number of pixel-level annotated samples. Few-shot segmentation has recently become a popular approach to addressing this problem, as it requires only a handful of annotated samples to generalize to new categories. However, the full utilization of limited samples remains an open problem. Thus, in this article, a mutually supervised few-shot segmentation network is proposed. First, the feature maps from intermediate convolution layers are fused to enrich the capacity of feature representation. Second, the support image and query image are combined into a bipartite graph, and the graph attention network is adopted to avoid losing spatial information and increase the number of pixels in the support image to guide the query image segmentation. Third, the attention map of the query image is used as prior information to enhance the support image segmentation, which forms a mutually supervised regime. Finally, the attention maps of the intermediate layers are fused and sent into the graph reasoning layer to infer the pixel categories. Experiments are conducted on the PASCAL VOC- 5i dataset and FSS-1000 dataset, and the results demonstrate the effectiveness and superior performance of our method compared with other baseline methods.

Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement

The Few-Shot Segmentation (FSS) aims to accomplish the novel class segmentation task with a few annotated images. Current FSS research based on meta-learning focuses on designing a complex interaction mechanism between the query and support feature. However, unlike humans who can rapidly learn new things from limited samples, the existing approach relies solely on fixed feature matching to tackle new tasks, lacking adaptability. In this paper, we propose a novel framework based on the adapter mechanism, namely Adaptive FSS, which can efficiently adapt the existing FSS model to the novel classes. In detail, we design the Prototype Adaptive Module (PAM), which utilizes accurate category information provided by the support set to derive class prototypes, enhancing class-specific information in the multi-stage representation. In addition, our approach is compatible with diverse FSS methods with different backbones by simply inserting PAM between the layers of the encoder. Experiments demonstrate that our method effectively improves the performance of the FSS models (e.g., MSANet, HDMNet, FPTrans, and DCAMA) and achieves new state-of-the-art (SOTA) results (i.e., 72.4% and 79.1% mIoU on PASCAL-5i 1-shot and 5-shot settings, 52.7% and 60.0% mIoU on COCO-20i 1-shot and 5-shot settings). Our code is available at https://github.com/jingw193/AdaptiveFSS.