Polyp Segmentation Research Articles

Boundary Refinement Network for Colorectal Polyp Segmentation in Colonoscopy Images

Boundary Refinement Network for Colorectal Polyp Segmentation in Colonoscopy Images

Polyp Segmentation Based on Multilevel Information Correction Transformer

Polyp Segmentation Based on Multilevel Information Correction Transformer

Optimizing Pix2Pix GAN with Attention Mechanisms for AI-Driven Polyp Segmentation in IoMT-Enabled Smart Healthcare.

This paper introduces an innovative approach for automated polyp segmentation in colonoscopy images, deploying an enhanced Pix2Pix Generative Adversarial Network (GAN) equipped with an integrated attention mechanism in the discriminator. Addressing prevalent challenges in conventional segmentation methods, such as variable polyp appearances, inconsistent image quality, and limited training data, our model significantly augments the precision and reliability of polyp segmentation. The integration of an attention mechanism enables our model to meticulously focus on the intricate features of polyps, improving segmentation accuracy. A unique training strategy, employing both real and synthetic data, is adopted to ensure the model's robust performance under a variety of conditions. The results, validated through rigorous tests on multiple public colonoscopy datasets, indicate a notable improvement in segmentation performance over existing state-of-the-art methods. Our model's enhanced ability to detect critical details early plays a pivotal role in proactive colorectal cancer detection, a key aspect of smart healthcare systems. This work represents an effective amalgamation of advanced AI techniques and the Internet of Medical Things (IoMT), signifying a noteworthy contribution to the evolution of smart healthcare systems. In conclusion, our attention-enhanced Pix2Pix GAN not only offers efficient and reliable polyp segmentation, but also showcases considerable potential for seamless integration into remote health monitoring systems, underlining the increasing relevance and efficacy of AI in advancing IoMT-enabled healthcare.

DLGRAFE-Net: A double loss guided residual attention and feature enhancement network for polyp segmentation.

Colon polyps represent a common gastrointestinal form. In order to effectively treat and prevent complications arising from colon polyps, colon polypectomy has become a commonly used therapeutic approach. Accurately segmenting polyps from colonoscopy images can provide valuable information for early diagnosis and treatment. Due to challenges posed by illumination and contrast variations, noise and artifacts, as well as variations in polyp size and blurred boundaries in polyp images, the robustness of segmentation algorithms is a significant concern. To address these issues, this paper proposes a Double Loss Guided Residual Attention and Feature Enhancement Network (DLGRAFE-Net) for polyp segmentation. Firstly, a newly designed Semantic and Spatial Information Aggregation (SSIA) module is used to extract and fuse edge information from low-level feature graphs and semantic information from high-level feature graphs, generating local loss-guided training for the segmentation network. Secondly, newly designed Deep Supervision Feature Fusion (DSFF) modules are utilized to fuse local loss feature graphs with multi-level features from the encoder, addressing the negative impact of background imbalance caused by varying polyp sizes. Finally, Efficient Feature Extraction (EFE) decoding modules are used to extract spatial information at different scales, establishing longer-distance spatial channel dependencies to enhance the overall network performance. Extensive experiments conducted on the CVC-ClinicDB and Kvasir-SEG datasets demonstrate that the proposed network outperforms all mainstream networks and state-of-the-art networks, exhibiting superior performance and stronger generalization capabilities.

Generalizable Polyp Segmentation via Randomized Global Illumination Augmentation.

Accurately segmenting polyps from colonoscopy images is essential for diagnosing colorectal cancer. Despite the tremendous success of the deep convolutional neural networks in automatic polyp segmentation, it suffers from domain shift issues, where the trained model yields performance deterioration on unseen test datasets. This paper proposes an illumination enhancement-based domain generalization approach to improve the generalization capability of the model on unseen test datasets and alleviate this issue. In particular, an image decomposition module (IDM) was developed to separate colonoscopy images into reflectance, local, and global illumination components. An illumination transform module (ITM) was proposed to augment images with different global illuminations by synthesizing target-like global illumination maps. A novel illumination variance insensitiveness (IViSen) is also introduced to evaluate the robustness of the model against illumination disturbance. IViSen is easy to compute and correlates well with model generalizability. The segmentation performance of the proposed model on four colonoscopy datasets was examined: CVC-ClinicDB, CVC-ColonDB, ETIS-Larib, and Kvasir-SEG. The method outperformed the competitive methods when tested on unseen domains. In particular, the proposed approach yielded 60.82% and 53.19% in terms of mean Dice and IoU, respectively, with 2.06% and 2.31% improvements.

Pixel-Centric Context Perception Network for Camouflaged Object Detection.

Camouflaged object detection (COD) aims to identify object pixels visually embedded in the background environment. Existing deep learning methods fail to utilize the context information around different pixels adequately and efficiently. In order to solve this problem, a novel pixel-centric context perception network (PCPNet) is proposed, the core of which is to customize the personalized context of each pixel based on the automatic estimation of its surroundings. Specifically, PCPNet first employs an elegant encoder equipped with the designed vital component generation (VCG) module to obtain a set of compact features rich in low-level spatial and high-level semantic information across multiple subspaces. Then, we present a parameter-free pixel importance estimation (PIE) function based on multiwindow information fusion. Object pixels with complex backgrounds will be assigned with higher PIE values. Subsequently, PIE is utilized to regularize the optimization loss. In this way, the network can pay more attention to those pixels with higher PIE values in the decoding stage. Finally, a local continuity refinement module (LCRM) is used to refine the detection results. Extensive experiments on four COD benchmarks, five salient object detection (SOD) benchmarks, and five polyp segmentation benchmarks demonstrate the superiority of PCPNet with respect to other state-of-the-art methods.

Cross-Level Context Fusion Network for Polyp Segmentation in Colonoscopy Images

Cross-Level Context Fusion Network for Polyp Segmentation in Colonoscopy Images

Tri-branch feature pyramid network based on federated particle swarm optimization for polyp segmentation.

Deep learning technology has shown considerable potential in various domains. However, due to privacy issues associated with medical data, legal and ethical constraints often result in smaller datasets. The limitations of smaller datasets hinder the applicability of deep learning technology in the field of medical image processing. To address this challenge, we proposed the Federated Particle Swarm Optimization algorithm, which is designed to increase the efficiency of decentralized data utilization in federated learning and to protect privacy in model training. To stabilize the federated learning process, we introduced Tri-branch feature pyramid network (TFPNet), a multi-branch structure model. TFPNet mitigates instability during the aggregation model deployment and ensures fast convergence through its multi-branch structure. We conducted experiments on four different public datasets：CVC-ClinicDB, Kvasir, CVC-ColonDB and ETIS-LaribPolypDB. The experimental results show that the Federated Particle Swarm Optimization algorithm outperforms single dataset training and the Federated Averaging algorithm when using independent scattered data, and TFPNet converges faster and achieves superior segmentation accuracy compared to other models.

PolyDSS: computer-aided decision support system for multiclass polyp segmentation and classification using deep learning

Colorectal cancer (CRC) is a malignant condition that affects the colon or rectum, and it is distinguished by abnormal cell growth in these areas. Colon polyps, which are abnormalities, can turn into cancer. To stop the spread of cancer, early polyp detection is essential. The timely removal of polyps without submitting a sample for histology is made possible by computer-assisted polyp classification. In addition to Locally Shared Features (LSF) and ensemble learning majority voting, this paper introduces a computer-aided decision support system named PolyDSS to assist endoscopists in segmenting and classifying various polyp classes using deep learning models like ResUNet and ResUNet++ and transfer learning models like EfficientNet. The PICCOLO dataset is used to train and test the PolyDSS model. To address the issue of class imbalance, data augmentation techniques were used on the dataset. To investigate the impact of each technique on the model, extensive experiments were conducted. While the classification module achieved the highest accuracy of 0.9425 by utilizing the strength of ensemble learning using majority voting, the proposed segmenting module achieved the highest Dice Similarity Coefficient (DSC) of 0.9244 using ResUNet++ and LSF. In conjunction with the Paris classification system, the PolyDSS model, with its significant results, can assist clinicians in identifying polyps early and choosing the best approach to treatment.

D-SAT: dual semantic aggregation transformer with dual attention for medical image segmentation

Objective. Medical image segmentation is significantly essential to assist clinicians in facilitating a quick and accurate diagnoses. However, most of the existing methods are still challenged by the loss of semantic information, blurred boundaries and the huge semantic gap between the encoder and decoder. Approach. To tackle these issues, a dual semantic aggregation transformer with dual attention is proposed for medical image segmentation. Firstly, the dual-semantic feature aggregation module is designed to build a bridge between convolutional neural network (CNN) and Transformer, effectively aggregating CNN’s local feature detail ability and Transformer’s long-range modeling ability to mitigate semantic information loss. Thereafter, the strip spatial attention mechanism is put forward to alleviate the blurred boundaries during encoding by constructing pixel-level feature relations across CSWin Transformer blocks from different spatial dimensions. Finally, a feature distribution gated attention module is constructed in the skip connection between the encoder and decoder to decrease the large semantic gap by filtering out the noise in low-level semantic information when fusing low-level and high-level semantic features during decoding. Main results. Comprehensive experiments conducted on abdominal multi-organ segmentation, cardiac diagnosis, polyp segmentation and skin lesion segmentation serve to validate the generalization and effectiveness of the proposed dual semantic aggregation transformer with dual attention (D-SAT). The superiority of D-SAT over current state-of-the-art methods is substantiated by both subjective and objective evaluations, revealing its remarkable performance in terms of segmentation accuracy and quality. Significance. The proposed method subtly preserves the local feature details and global context information in medical image segmentation, providing valuable support to improve diagnostic efficiency for clinicians and early disease control for patients. Code is available at https://github.com/Dxkm/D-SAT.

Multi-level feature fusion network combining attention mechanisms for polyp segmentation

Clinically, automated polyp segmentation techniques have the potential to significantly improve the efficiency and accuracy of medical diagnosis, thereby reducing the risk of colorectal cancer in patients. Unfortunately, existing methods suffer from two significant weaknesses that can impact the accuracy of segmentation. Firstly, features extracted by encoders are not adequately filtered and utilized. Secondly, semantic conflicts and information redundancy caused by feature fusion are not attended to. To overcome these limitations, we propose a novel approach for polyp segmentation, named MLFF-Net, which leverages multi-level feature fusion and attention mechanisms. Specifically, MLFF-Net comprises three modules: Multi-scale Attention Module (MAM), High-level Feature Enhancement Module (HFEM), and Global Attention Module (GAM). Among these, MAM is used to extract multi-scale information and polyp details from the shallow output of the encoder. In HFEM, the deep features of the encoders complement each other by aggregation. Meanwhile, the attention mechanism redistributes the weight of the aggregated features, weakening the conflicting redundant parts and highlighting the information useful to the task. GAM combines features from the encoder and decoder features, as well as computes global dependencies to prevent receptive field locality. Experimental results on five public datasets show that the proposed method not only can segment multiple types of polyps but also has advantages over current state-of-the-art methods in both accuracy and generalization ability.

Exploring the Potential of Ensembles of Deep Learning Networks for Image Segmentation

To identify objects in images, a complex set of skills is needed that includes understanding the context and being able to determine the borders of objects. In computer vision, this task is known as semantic segmentation and it involves categorizing each pixel in an image. It is crucial in many real-world situations: for autonomous vehicles, it enables the identification of objects in the surrounding area; in medical diagnosis, it enhances the ability to detect dangerous pathologies early, thereby reducing the risk of serious consequences. In this study, we compare the performance of various ensembles of convolutional and transformer neural networks. Ensembles can be created, e.g., by varying the loss function, the data augmentation method, or the learning rate strategy. Our proposed ensemble, which uses a simple averaging rule, demonstrates exceptional performance across multiple datasets. Notably, compared to prior state-of-the-art methods, our ensemble consistently shows improvements in the well-studied polyp segmentation problem. This problem involves the precise delineation and identification of polyps within medical images, and our approach showcases noteworthy advancements in this domain, obtaining an average Dice of 0.887, which outperforms the current SOTA with an average Dice of 0.885.

Dual‐branch feature extraction network combined with Transformer and CNN for polyp segmentation

AbstractTo overcome the difficulty of accurate polyp segmentation, a novel encoder–decoder network DFETC‐Net is proposed, in which two encoders based on Swin Transformer and CNN are utilized to extract the global and local features respectively. Further, a new self‐attention and convolution feature fusion module is designed to fuse the two branch features to enhance the feature representative capability and alleviate the influence of the semantic gap. In the bottleneck, a new multi‐feature pyramid pooling module fuses all deep features from two branches to obtain multi‐scale information and promote segmentation accuracy. The coordinate attention is used in the skip connections between two shallow CNN blocks and corresponding decoder blocks to pay more attention to doubtful and complicated regions. Extensive experiments demonstrate the proposed network outperforms several state‐of‐the‐art methods in terms of both qualitative effects and quantitative measurements. All codes are available on https://github.com/LYJieH/DFETC-NET.

Learning Dynamic Prototypes for Visual Pattern Debiasing

AbstractDeep learning has achieved great success in academic benchmarks but fails to work effectively in the real world due to the potential dataset bias. The current learning methods are prone to inheriting or even amplifying the bias present in a training dataset and under-represent specific demographic groups. More recently, some dataset debiasing methods have been developed to address the above challenges based on the awareness of protected or sensitive attribute labels. However, the number of protected or sensitive attributes may be considerably large, making it laborious and costly to acquire sufficient manual annotation. To this end, we propose a prototype-based network to dynamically balance the learning of different subgroups for a given dataset. First, an object pattern embedding mechanism is presented to make the network focus on the foreground region. Then we design a prototype learning method to discover and extract the visual patterns from the training data in an unsupervised way. The number of prototypes is dynamic depending on the pattern structure of the feature space. We evaluate the proposed prototype-based network on three widely used polyp segmentation datasets with abundant qualitative and quantitative experiments. Experimental results show that our proposed method outperforms the CNN-based and transformer-based state-of-the-art methods in terms of both effectiveness and fairness metrics. Moreover, extensive ablation studies are conducted to show the effectiveness of each proposed component and various parameter values. Lastly, we analyze how the number of prototypes grows during the training process and visualize the associated subgroups for each learned prototype. The code and data will be released at https://github.com/zijinY/dynamic-prototype-debiasing.

CoInNet: A Convolution-Involution Network With a Novel Statistical Attention for Automatic Polyp Segmentation.

Polyps are very common abnormalities in human gastrointestinal regions. Their early diagnosis may help in reducing the risk of colorectal cancer. Vision-based computer-aided diagnostic systems automatically identify polyp regions to assist surgeons in their removal. Due to their varying shape, color, size, texture, and unclear boundaries, polyp segmentation in images is a challenging problem. Existing deep learning segmentation models mostly rely on convolutional neural networks that have certain limitations in learning the diversity in visual patterns at different spatial locations. Further, they fail to capture inter-feature dependencies. Vision transformer models have also been deployed for polyp segmentation due to their powerful global feature extraction capabilities. But they too are supplemented by convolution layers for learning contextual local information. In the present paper, a polyp segmentation model CoInNet is proposed with a novel feature extraction mechanism that leverages the strengths of convolution and involution operations and learns to highlight polyp regions in images by considering the relationship between different feature maps through a statistical feature attention unit. To further aid the network in learning polyp boundaries, an anomaly boundary approximation module is introduced that uses recursively fed feature fusion to refine segmentation results. It is indeed remarkable that even tiny-sized polyps with only 0.01% of an image area can be precisely segmented by CoInNet. It is crucial for clinical applications, as small polyps can be easily overlooked even in the manual examination due to the voluminous size of wireless capsule endoscopy videos. CoInNet outperforms thirteen state-of-the-art methods on five benchmark polyp segmentation datasets.

Attention combined pyramid vision transformer for polyp segmentation

Colorectal cancer (CRC) has become one of the most frequent cancers in the world. To prevent CRC, proper polyp localization in endoscopy images plays a vital role for detecting and removing colorectal polyps. Most polyp segmentation methods use convolutional neural networks (CNN) as their backbone, and have achieved promising results to effectively assist clinicians in their diagnosis. However, those CNN-based approaches have limitations in modeling accurate location and shape of polyps, due to the intrinsic locality property of convolutional operations. To address these limitations, this study proposes a novel network, namely Att-PVT, that combines CNN and Pyramid Vision Transformer (PVT) together for poly segmentation. The main challenge lies in maintaining long-range semantic information without sacrificing low-level features. Att-PVT applies multidimensional information extraction (MIE) to generate refined feature maps extracted from PVT for better feature representation. Cascaded context integration (CCI) is designed to adaptively aggregating the three highest layers of refined polyp features for learning semantic and location information. To accurately segment colorectal polyps, Att-PVT introduces multilevel feature fusion (MFF) module that explores the boundary information in the shallower layer based on the global map. The proposed workflow has undergone comparative experiments on three public datasets, namely Kvasir, ColonDB, and ETIS. The results show that the proposed approach achieves impressive mDice scores of 0.926, 0.817, and 0.794 for polyp segmentation tasks on these datasets, surpassing other state-of-the-art methods. This indicates the superior generalization and scalability of the proposed approach.

Dual-branch multi-information aggregation network with transformer and convolution for polyp segmentation

Computer-Aided Diagnosis (CAD) for polyp detection offers one of the most notable showcases. By using deep learning technologies, the accuracy of polyp segmentation is surpassing human experts. In such CAD process, a critical step is concerned with segmenting colorectal polyps from colonoscopy images. Despite remarkable successes attained by recent deep learning related works, much improvement is still anticipated to tackle challenging cases. For instance, the effects of motion blur and light reflection can introduce significant noise into the image. The same type of polyps has a diversity of size, color and texture. To address such challenges, this paper proposes a novel dual-branch multi-information aggregation network (DBMIA-Net) for polyp segmentation, which is able to accurately and reliably segment a variety of colorectal polyps with efficiency. Specifically, a dual-branch encoder with transformer and convolutional neural networks (CNN) is employed to extract polyp features, and two multi-information aggregation modules are applied in the decoder to fuse multi-scale features adaptively. Two multi-information aggregation modules include global information aggregation (GIA) module and edge information aggregation (EIA) module. In addition, to enhance the representation learning capability of the potential channel feature association, this paper also proposes a novel adaptive channel graph convolution (ACGC). To validate the effectiveness and advantages of the proposed network, we compare it with several state-of-the-art (SOTA) methods on five public datasets. Experimental results consistently demonstrate that the proposed DBMIA-Net obtains significantly superior segmentation performance across six popularly used evaluation matrices. Especially, we achieve 94.12% mean Dice on CVC-ClinicDB dataset which is 4.22% improvement compared to the previous state-of-the-art method PraNet. Compared with SOTA algorithms, DBMIA-Net has a better fitting ability and stronger generalization ability.

MMFIL-Net: Multi-level and multi-source feature interactive lightweight network for polyp segmentation

Accuracy, generalized performance, and model size are critical considerations for designing a real-time polyp segmentation model. However, existing techniques primarily focus on accuracy and do not consider the other two vital characteristics. This paper proposes MMFIL-Net, a novel polyp segmentation technique. As part of MMFIL-Net, the Hierarchical Multi-source Feature Interaction Module (HMFIM) comprises Multi-source Feature Interaction Blocks (MFIB). MFIB manipulates multi-level and multi-sourced features to reduce the gap between low and high-level feature maps to achieve generalized performance. Additionally, the Multiple Receptive Field Feature Interaction Block (MRFFIB) targets the issues of segmenting polyps of different sizes. Finally, Dual Source Attention Fusion Block (DSAFB) is introduced to deal with hazy boundary information for early-stage polyps detection and segmentation. The proposed model outperformed existing lightweight models in conducted evaluation on different datasets. In addition to the achieved generalized performance and higher accuracy, the proposed model presents a significant reduction in the model size than existing approaches. The proposed model only contains 6.68 million parameters and has 4.32G MACs (Multiply–Accumulate Operations), which is better than the current approaches.

Boundary uncertainty aware network for automated polyp segmentation

Recently, leveraging deep neural networks for automated colorectal polyp segmentation has emerged as a hot topic due to the favored advantages in evading the limitations of visual inspection, e.g., overwork and subjectivity. However, most existing methods do not pay enough attention to the uncertain areas of colonoscopy images and often provide unsatisfactory segmentation performance. In this paper, we propose a novel boundary uncertainty aware network (BUNet) for precise and robust colorectal polyp segmentation. Specifically, considering that polyps vary greatly in size and shape, we first adopt a pyramid vision transformer encoder to learn multi-scale feature representations. Then, a simple yet effective boundary exploration module (BEM) is proposed to explore boundary cues from the low-level features. To make the network focus on the ambiguous area where the prediction score is biased to neither the foreground nor the background, we further introduce a boundary uncertainty aware module (BUM) that explores error-prone regions from the high-level features with the assistance of boundary cues provided by the BEM. Through the top-down hybrid deep supervision, our BUNet implements coarse-to-fine polyp segmentation and finally localizes polyp regions precisely. Extensive experiments on five public datasets show that BUNet is superior to thirteen competing methods in terms of both effectiveness and generalization ability.

CSwinDoubleU-Net: A double U-shaped network combined with convolution and Swin Transformer for colorectal polyp segmentation

The early detection of polyps from colonoscopy can reduce the risk of colorectal cancer development and give timely treatment. Accurate polyp segmentation during colonoscopy examinations can help clinicians locate the polyps which is of great significance in the clinical prevention of colorectal cancer. However, due to large variations in terms of size, color, texture, and morphology of polyps, the similarity between the polyp lesions and their background, the variations of illumination, motion blur, low-contrast areas, intestinal contents during image acquisitions, precisely polyp segmentation is still an open issue. To overcome the challenges above, a new double U-shaped image segmentation network combining convolution structure and Shifted Windows (Swin) Transformer, namely CSwinDoubleU-Net, is proposed in this paper. The developed CSwinDoubleU-Net consists of two U-shaped encode-decode structures associated with a pure CNN-based structure and a pure Transformer-based one, and a feature fusion module that interacts with the two U-shaped parts. The first U-shaped codecs network utilizes the multi-convolution layers to extract the local feature information and the coordinate attention module in each skip connection to effectively reduce the loss of spatial information and obtain the accurate position information of the encoded features. Next, the second U-shaped codecs network uses the Swin Transformer layers with the sliding window to extract global feature information further. At last, a convolutional feature and self-attention feature fusion module (CSFFM) is designed to deeply fuse the local convolutional features extracted from the first U-shape structure and the global self-Attention features extracted from the second U-shape structure. The obtained multi-category and multi-dimensional fused feature information can help to recover the boundary features of polyps. Extensive experiments are conducted to validate the proposed CSwinDoubleU-Net on five publicly available datasets, including CVC-ClinicDB, Kvasir, CVC-ColonDB, CVC-T, and ETIS-Larib. The results show that the proposed model can outperform some state-of-the-art methods and achieve high segmentation performance for polyp images.