Segmentation Dataset Research Articles

RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM

Satellite remote sensing images contain complex and diverse ground object information and the images exhibit spatial multi-scale characteristics, making the panoptic segmentation of satellite remote sensing images a highly challenging task. Due to the lack of large-scale annotated datasets for panoramic segmentation, existing methods still suffer from weak model generalization capabilities. To mitigate this issue, this paper leverages the advantages of the Segment Anything Model (SAM), which can segment any object in remote sensing images without requiring any annotations and proposes a high-resolution remote sensing image panoptic segmentation method called Remote Sensing Panoptic Segmentation SAM (RSPS-SAM). Firstly, to address the problem of global information loss caused by cropping large remote sensing images for training, a Batch Attention Pyramid was designed to extract multi-scale features from remote sensing images and capture long-range contextual information between cropped patches, thereby enhancing the semantic understanding of remote sensing images. Secondly, we constructed a Mask Decoder to address the limitation of SAM requiring manual input prompts and its inability to output category information. This decoder utilized mask-based attention for mask segmentation, enabling automatic prompt generation and category prediction of segmented objects. Finally, the effectiveness of the proposed method was validated on the high-resolution remote sensing image airport scene dataset RSAPS-ASD. The results demonstrate that the proposed method achieves segmentation and recognition of foreground instances and background regions in high-resolution remote sensing images without the need for prompt input, while providing smooth segmentation boundaries with a panoptic segmentation quality (PQ) of 57.2, outperforming current mainstream methods.

Topology-aware mamba for crack segmentation in structures

CrackMamba, a Mamba-based model, is designed for efficient and accurate crack segmentation for monitoring the structural health of infrastructure. Traditional Convolutional Neural Network (CNN) models struggle with limited receptive fields, and while Vision Transformers (ViT) improve segmentation accuracy, they are computationally intensive. CrackMamba addresses these challenges by utilizing the VMambaV2 with pre-trained ImageNet-1 k weights as the encoder and a newly designed decoder for better performance. To handle the random and complex nature of crack development, a Snake Scan module is proposed to reshape crack feature sequences, enhancing feature extraction. Additionally, the three-branch Snake Conv VSS (SCVSS) block is proposed to target cracks more effectively. Experiments show that CrackMamba achieves state-of-the-art (SOTA) performance on the CrackSeg9k and SewerCrack datasets, and demonstrates competitive performance on the retinal vessel segmentation dataset CHASE_DB1, highlighting its generalization capability.

Multi-Source Fusion Enhanced Feature Segmentation in Remote Sensing Imagery

Abstract. With deepening application of deep learning technology in the field of remote sensing, several challenges persist in the segmentation of remote sensing optical images. These challenges include: (1) insufficient availability of deep learning-based remote sensing semantic segmentation datasets; (2) inadequate utilization of multi-source remote sensing data in the field of semantic segmentation; (3) limited sample size for effective model training, as well as the need to enhance both the speed and accuracy of model training. To address these challenges, this study introduces a multi-source remote sensing dataset consisting of 15000 data pairs, each comprising remote sensing multispectral data, synthetic aperture radar(SAR) images, land use and land cover(LULC) data, digital elevation model (DEM) data as well as the analysis data including Slope, Aspect, and Hillshade. Through the application of an end-to-end network based on pix2pix, effective fusion and feature enhancement of multi-source remote sensing data were achieved. The structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and Spectral Angle Mapper(SAM) values reached 0.84, 23.14, and 0.19, respectively, representing significant improvements over the baseline Pix2pix model’s performance metrics of 0.62, 18.84, and 0.23. In downstream semantic segmentation applications, the enhanced dataset was utilized to train semantic segmentation models for remote sensing image analysis. This approach effectively improved the training speed and segmentation accuracy of the models, with the mean intersection over union (mIoU) increasing from 0.467 to 0.481 and accuracy rising from 0.734 to 0.746. Moreover, the visual representation of remote sensing image segmentation demonstrated noticeable enhancements.

Stroke‐Seg: A deep learning‐based framework for chinese stroke segmentation

AbstractChinese stroke segmentation is a crucial and challenging task for various downstream applications such as font generation, aesthetic evaluation etc. Conventional semantic segmentation techniques typically face difficulties in accurately segmenting strokes, as intersection regions in Chinese characters can belong to multiple strokes simultaneously, and these approaches often lack a holistic understanding of character composition. This paper proposes a character stroke segmentation framework named Stroke‐Seg that integrates with various semantic segmentation architectures, demonstrating adaptability to different backbone networks to tackle the above tasks. A multi‐label output strategy is proposed to effectively classify strokes in intersection areas, overcoming the limitations of traditional semantic segmentation approaches. Additionally, a prior knowledge vector is incorporated into the input layer to provide character‐specific information on stroke composition, enhancing the ability of the framework to precisely identify and segment strokes. The effectiveness of the proposed framework is demonstrated through evaluations of a comprehensive dataset (brush calligraphy stroke segmentation dataset). Evaluations show that the proposed framework significantly improves the capability of semantic segmentation networks, achieving a remarkable improvement of up to 19.9% in the true stroke rate, compared to traditional stroke segmentation techniques. The Stroke‐Seg framework integrated with TransUNet demonstrates its high performance with an impressive 98.2% true stroke rate. Furthermore, the proposed framework combined with FCN still achieves good performance while consuming the least amount of computational resources and memory, demonstrating its potential for lightweight design.

Contour-constrained branch U-Net for accurate left ventricular segmentation in echocardiography.

Using echocardiography to assess the left ventricular function is one of the most crucial cardiac examinations in clinical diagnosis, and LV segmentation plays a particularly vital role in medical image processing as many important clinical diagnostic parameters are derived from the segmentation results, such as ejection function. However, echocardiography typically has a lower resolution and contains a significant amount of noise and motion artifacts, making it a challenge to accurate segmentation, especially in the region of the cardiac chamber boundary, which significantly restricts the accurate calculation of subsequent clinical parameters. In this paper, our goal is to achieve accurate LV segmentation through a simplified approach by introducing a branch sub-network into the decoder of the traditional U-Net. Specifically, we employed the LV contour features to supervise the branch decoding process and used a cross attention module to facilitate the interaction relationship between the branch and the original decoding process, thereby improving the segmentation performance in the region LV boundaries. In the experiments, the proposed branch U-Net (BU-Net) demonstrated superior performance on CAMUS and EchoNet-dynamic public echocardiography segmentation datasets in comparison to state-of-the-art segmentation models, without the need for complex residual connections or transformer-based architectures. Our codes are publicly available at Anonymous Github https://anonymous.4open.science/r/Anoymous_two-BFF2/ .

Semi-supervised segmentation of cardiac chambers from LGE-CMR using feature consistency awareness

BackgroundLate gadolinium enhancement cardiac magnetic resonance imaging (LGE-CMR) is a valuable cardiovascular imaging technique. Segmentation of cardiac chambers from LGE-CMR is a fundamental step in electrophysiological modeling and cardiovascular disease diagnosis. Deep learning methods have demonstrated extremely promising performance. However, excellent performance often depended on a large amount of finely annotated data. The purpose of this manuscript was to develop a semi-supervised segmentation method to use unlabeled data to improve model performance.MethodsThis manuscript proposed a semi-supervised network that integrates triple-consistency constraints (data-level, task-level, and feature-level) for cardiac chambers segmentation from LGE-CMR. Specifically, we designed a network that integrated segmentation and edge prediction tasks based on the mean teacher architecture. This addressed the problem of ignoring some challenging regions because of excluding low-confidence regions of previous research. We also applied a voxel-level contrastive learning strategy to achieve feature-level consistency, helping the model pay attention to the consistency between features overlooked in previous research.ResultsIn terms of the Dice, Jaccard, Average Surface Distance (ASD), and 95% Hausdorff Distance (95HD) metrics, for the atrial segmentation dataset, the proposed method achieved scores of 88.34%, 79.30%, 7.92, and 2.02 when trained with 10% labeled data, and 90.70%, 83.09%, 6.41, and 1.72 when trained with 20% labeled data. For the ventricular segmentation task, the results were 87.22%, 77.95%, 2.27, and 0.61 with 10% labeled data, and 88.99%, 80.45%, 1.87, and 0.51 with 20% labeled data, respectively.ConclusionExperiments demonstrated that our method outperforms previous semi-supervised methods, showing the potential of the proposed network for semi-supervised segmentation problems.

Interactive Segmentation for Medical Images Using Spatial Modeling Mamba

Interactive segmentation methods utilize user-provided positive and negative clicks to guide the model in accurately segmenting target objects. Compared to fully automatic medical image segmentation, these methods can achieve higher segmentation accuracy with limited image data, demonstrating significant potential in clinical applications. Typically, for each new click provided by the user, conventional interactive segmentation methods reprocess the entire network by re-inputting the click into the segmentation model, which greatly increases the user’s interaction burden and deviates from the intended goal of interactive segmentation tasks. To address this issue, we propose an efficient segmentation network, ESM-Net, for interactive medical image segmentation. It obtains high-quality segmentation masks based on the user’s initial clicks, reducing the complexity of subsequent refinement steps. Recent studies have demonstrated the strong performance of the Mamba model in various vision tasks; however, its application in interactive segmentation remains unexplored. In our study, we incorporate the Mamba module into our framework for the first time and enhance its spatial representation capabilities by developing a Spatial Augmented Convolution (SAC) module. These components are combined as the fundamental building blocks of our network. Furthermore, we designed a novel and efficient segmentation head to fuse multi-scale features extracted from the encoder, optimizing the generation of the predicted segmentation masks. Through comprehensive experiments, our method achieved state-of-the-art performance on three medical image datasets. Specifically, we achieved 1.43 NoC@90 on the Kvasir-SEG dataset, 1.57 NoC@90 on the CVC-ClinicDB polyp segmentation dataset, and 1.03 NoC@90 on the ADAM retinal disk segmentation dataset. The assessments on these three medical image datasets highlight the effectiveness of our approach in interactive medical image segmentation.

A Synthetic Dataset for Semantic Segmentation of Waterbodies in Out-of-Distribution Situations

In the past decade, substantial global efforts have been devoted to the development of reliable and efficient solutions for early flood warning and monitoring. One of the most common strategies for tackling this challenge involves the application of computer vision techniques to images obtained from the numerous surveillance cameras present in urban settings today. While there are various datasets available for training and testing these techniques, none of them specifically addresses the issue of out-of-distribution (OoD) behavior. This issue becomes particularly critical when evaluating the reliability of these methods under challenging environmental conditions. Our work stands as the first attempt to bridge this gap by introducing a new, highly controlled synthetic dataset that encompasses the essential attributes required for analyzing OoD behavior. The very high correlation between the accuracy of artificial intelligence (AI) models trained on our synthetic dataset and models trained on real-world data proves our dataset’s ability to predict real-world OoD behavior reliably.

Advances in medical image analysis: A comprehensive survey of lung infection detection

AbstractThis research investigates advanced approaches in medical image analysis, specifically focusing on segmentation and classification techniques, as well as their integration into multi‐task architectures for lung infections. This research begins by explaining key architectural models used in segmentation and classification tasks. The study extends to the enhancement of these architectures through attention modules and conditional random fields. Relevant datasets and evaluation metrics, incorporating discussions on loss functions are also reviewed. This review encompasses recent advancements in single‐task and multi‐task models, highlighting innovations in semi‐supervised, self‐supervised, few‐shot, and zero‐shot learning techniques. Empirical analysis is conducted on both single‐task and multi‐task architectures, predominantly utilizing the U‐Net framework, and is applied across multiple datasets for segmentation and classification tasks. Results demonstrate the effectiveness of these models and provide insights into the strengths and limitations of different approaches. This research contributes to improved detection and diagnosis of lung infections by offering a comprehensive overview of current methodologies and their practical applications.

Semantic Segmentation of Satellite Images for Landslide Detection Using Foreground-Aware and Multi-Scale Convolutional Attention Mechanism.

Advancements in satellite and aerial imagery technology have made it easier to obtain high-resolution remote sensing images, leading to widespread research and applications in various fields. Remote sensing image semantic segmentation is a crucial task that provides semantic and localization information for target objects. In addition to the large-scale variation issues common in most semantic segmentation datasets, aerial images present unique challenges, including high background complexity and imbalanced foreground-background ratios. However, general semantic segmentation methods primarily address scale variations in natural scenes and often neglect the specific challenges in remote sensing images, such as inadequate foreground modeling. In this paper, we present a foreground-aware remote sensing semantic segmentation model. The model introduces a multi-scale convolutional attention mechanism and utilizes a feature pyramid network architecture to extract multi-scale features, addressing the multi-scale problem. Additionally, we introduce a Foreground-Scene Relation Module to mitigate false alarms. The model enhances the foreground features by modeling the relationship between the foreground and the scene. In the loss function, a Soft Focal Loss is employed to focus on foreground samples during training, alleviating the foreground-background imbalance issue. Experimental results indicate that our proposed method outperforms current state-of-the-art general semantic segmentation methods and transformer-based methods on the LS dataset benchmark.

A dual-labeled dataset and fusion model for automatic teeth segmentation, numbering, and state assessment on panoramic radiographs

BackgroundRecently, deep learning has been increasingly applied in the field of dentistry. The aim of this study is to develop a model for the automatic segmentation, numbering, and state assessment of teeth on panoramic radiographs.MethodsWe created a dual-labeled dataset on panoramic radiographs for training, incorporating both numbering and state labels. We then developed a fusion model that combines a YOLOv9-e instance segmentation model with an EfficientNetv2-l classification model. The instance segmentation model is used for tooth segmentation and numbering, whereas the classification model is used for state evaluation. The final prediction results integrate tooth position, numbering, and state information. The model’s output includes result visualization and automatic report generation.ResultsPrecision, Recall, mAP50 (mean Average Precision), and mAP50-95 for the tooth instance segmentation task are 0.989, 0.955, 0.975, and 0.840, respectively. Precision, Recall, Specificity, and F1 Score for the tooth classification task are 0.943, 0.933, 0.985, and 0.936, respectively.ConclusionsThis fusion model is the first to integrate automatic dental segmentation, numbering, and state assessment. It provides highly accurate results, including detailed visualizations and automated report generation.

Pre‐trained SAM as data augmentation for image segmentation

AbstractData augmentation plays an important role in training deep neural model by expanding the size and diversity of the dataset. Initially, data augmentation mainly involved some simple transformations of images. Later, in order to increase the diversity and complexity of data, more advanced methods appeared and evolved to sophisticated generative models. However, these methods required a mass of computation of training or searching. In this paper, a novel training‐free method that utilises the Pre‐Trained Segment Anything Model (SAM) model as a data augmentation tool (PTSAM‐DA) is proposed to generate the augmented annotations for images. Without the need for training, it obtains prompt boxes from the original annotations and then feeds the boxes to the pre‐trained SAM to generate diverse and improved annotations. In this way, annotations are augmented more ingenious than simple manipulations without incurring huge computation for training a data augmentation model. Multiple comparative experiments on three datasets are conducted, including an in‐house dataset, ADE20K and COCO2017. On this in‐house dataset, namely Agricultural Plot Segmentation Dataset, maximum improvements of 3.77% and 8.92% are gained in two mainstream metrics, mIoU and mAcc, respectively. Consequently, large vision models like SAM are proven to be promising not only in image segmentation but also in data augmentation.

A lighter hybrid feature fusion framework for polyp segmentation

Colonoscopy is widely recognized as the most effective method for the detection of colon polyps, which is crucial for early screening of colorectal cancer. Polyp identification and segmentation in colonoscopy images require specialized medical knowledge and are often labor-intensive and expensive. Deep learning provides an intelligent and efficient approach for polyp segmentation. However, the variability in polyp size and the heterogeneity of polyp boundaries and interiors pose challenges for accurate segmentation. Currently, Transformer-based methods have become a mainstream trend for polyp segmentation. However, these methods tend to overlook local details due to the inherent characteristics of Transformer, leading to inferior results. Moreover, the computational burden brought by self-attention mechanisms hinders the practical application of these models. To address these issues, we propose a novel CNN-Transformer hybrid model for polyp segmentation (CTHP). CTHP combines the strengths of CNN, which excels at modeling local information, and Transformer, which excels at modeling global semantics, to enhance segmentation accuracy. We transform the self-attention computation over the entire feature map into the width and height directions, significantly improving computational efficiency. Additionally, we design a new information propagation module and introduce additional positional bias coefficients during the attention computation process, which reduces the dispersal of information introduced by deep and mixed feature fusion in the Transformer. Extensive experimental results demonstrate that our proposed model achieves state-of-the-art performance on multiple benchmark datasets for polyp segmentation. Furthermore, cross-domain generalization experiments show that our model exhibits excellent generalization performance.

Mutually-Guided Hierarchical Multi-Modal Feature Learning for Referring Image Segmentation

Referring image segmentation aims to locate and segment the target region based on a given textual expression query. The primary challenge is to understand semantics from visual and textual modalities and achieve alignment and matching. Prior works have attempted to address this challenge by leveraging separately pretrained unimodal models to extract global visual and textual features, and perform straightforwardly fusion to establish cross-modal semantic associations. However, these methods often concentrate solely on the global semantics, disregarding the hierarchical semantics of expression and image, and struggling with complex and open real scenarios, thus failing to capture critical cross-modal information. To address these limitations, this paper introduces an innovative mutually-guided hierarchical multi-modal feature learning scheme. By leveraging the guidance of global visual features, the model mines hierarchical text features from different stages of the text encoder. Simultaneously, the guidance of global textual features is leveraged to aggregate multi-scale visual features. This mutually guided hierarchical feature learning effectively addresses the semantically inaccurate cause by free-form text and naturally occurring scale variations. Furthermore, a Segment Detail Refinement (SDR) module is designed to enhance the model’s spatial detail awareness through attention mapping of low-level visual features and cross-modal features. To evaluate the effectiveness of the proposed approach, extensive experiments are conducted on three widely used referring image object segmentation datasets. The results demonstrate the superiority of the presented method in accurately locating and segmenting objects in images.

Click to Correction: Interactive Bidirectional Dynamic Propagation Video Object Segmentation Network.

High-quality video object segmentation is a challenging visual computing task. Interactive segmentation can improve segmentation results. This paper proposes a multi-round interactive dynamic propagation instance-level video object segmentation network based on click interaction. The network consists of two parts: a user interaction segmentation module and a bidirectional dynamic propagation module. A prior segmentation network was designed in the user interaction segmentation module to better segment objects of different scales that users click on. The dynamic propagation network achieves high-precision video object segmentation through the bidirectional propagation and fusion of segmentation masks obtained from multiple rounds of interaction. Experiments on interactive segmentation datasets and video object segmentation datasets show that our method achieves state-of-the-art segmentation results with fewer click interactions.

Annotated Dataset for Training Cloud Segmentation Neural Networks Using High-Resolution Satellite Remote Sensing Imagery

The integration of satellite data with deep learning has revolutionized various tasks in remote sensing, including classification, object detection, and semantic segmentation. Cloud segmentation in high-resolution satellite imagery is a critical application within this domain, yet progress in developing advanced algorithms for this task has been hindered by the scarcity of specialized datasets and annotation tools. This study addresses this challenge by introducing CloudLabel, a semi-automatic annotation technique leveraging region growing and morphological algorithms including flood fill, connected components, and guided filter. CloudLabel v1.0 streamlines the annotation process for high-resolution satellite images, thereby addressing the limitations of existing annotation platforms which are not specifically adapted to cloud segmentation, and enabling the efficient creation of high-quality cloud segmentation datasets. Notably, we have curated the Annotated Dataset for Training Cloud Segmentation (ADTCS) comprising 32,065 images (512 × 512) for cloud segmentation based on CloudLabel. The ADTCS dataset facilitates algorithmic advancement in cloud segmentation, characterized by uniform cloud coverage distribution and high image entropy (mainly 5–7). These features enable deep learning models to capture comprehensive cloud characteristics, enhancing recognition accuracy and reducing ground object misclassification. This contribution significantly advances remote sensing applications and cloud segmentation algorithms.

HyFormer: a hybrid transformer-CNN architecture for retinal OCT image segmentation

Optical coherence tomography (OCT) has become the leading imaging technique in diagnosing and treatment planning for retinal diseases. Retinal OCT image segmentation involves extracting lesions and/or tissue structures to aid in the decisions of ophthalmologists, and multi-class segmentation is commonly needed. As the target regions often spread widely inside the retina, and the intensities and locations of different categories can be close, good segmentation networks must possess both global modeling capabilities and the ability to capture fine details. To address the challenge in capturing both global and local features simultaneously, we propose HyFormer, an efficient, lightweight, and robust hybrid network architecture. The proposed architecture features parallel Transformer and convolutional encoders for independent feature capture. A multi-scale gated attention block and a group positional embedding block are introduced within the Transformer encoder to enhance feature extraction. Feature integration is achieved in the decoder composed of the proposed three-path fusion modules. A class activation map-based cross-entropy loss function is also proposed to improve segmentation results. Evaluations are performed on a private dataset with myopic traction maculopathy lesions and the public AROI dataset for retinal layer and lesion segmentation with age-related degeneration. The results demonstrate HyFormer's superior segmentation performance and robustness compared to existing methods, showing promise for accurate and efficient OCT image segmentation. .

RdmkNet & Toronto-RDMK: Large-Scale Datasets for Road Marking Classification and Segmentation

RdmkNet & Toronto-RDMK: Large-Scale Datasets for Road Marking Classification and Segmentation

Tumor Detection in MRI Data using Deep Learning Techniques for Image Classification and Semantic Segmentation

This study aims to apply image classification and semantic segmentation methods using deep learning for accurate and efficient tumor detection in brain MRI data. It looks at ways to get over issues including interpretability of the model and a lack of annotated data. A pre-trained VGG19 model and a basic convolutional neural network (CNN) were trained using a public dataset of 273 brain MRI images for the purpose of image classification. Techniques for regularization, transfer learning, and data augmentation were used. A U-Net architecture was trained using manually created masks from the same dataset for semantic segmentation, with several training setups investigated. With a mean test accuracy of 77.19%, the pre-trained VGG19 model surpassed the basic CNN in picture categorization. The U-Net model performed better for semantic segmentation after training on the entire dataset for two epochs as opposed to training on a subset for ten epochs. The work acknowledges issues including data scarcity and the requirement for model interpretability and scalability, which call for additional research, but also underlines the potential of deep learning for brain tumor identification.

Prototype-Based Semantic Segmentation.

Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20 K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.