Robust Image Segmentation Research Articles

Optimizing waste handling with interactive AI: Prompt-guided segmentation of construction and demolition waste using computer vision

Optimized and automated methods for handling construction and demolition waste (CDW) are crucial for improving the resource recovery process in waste management. Automated waste recognition is a critical step in this process, and it relies on robust image segmentation techniques. Prompt-guided segmentation methods provide promising results for specific user needs in image recognition. However, the current state-of-the-art segmentation methods trained for generic images perform unsatisfactorily on CDW recognition tasks, indicating a domain gap. To address this gap, a user-guided segmentation pipeline is developed in this study that leverages prompts such as bounding boxes, points, and text to segment CDW in cluttered environments. The adopted approach achieves a class-wise performance of around 70 % in several waste categories, surpassing the state-of-the-art algorithms by 9 % on average. This method allows users to create accurate segmentations by drawing a bounding box, clicking, or providing a text prompt, minimizing the time spent on detailed annotations. Integrating this human–machine system as a user-friendly interface into material recovery facilities enhances the monitoring and processing of waste, leading to better resource recovery outcomes in waste management.

TBConvL-Net: A hybrid deep learning architecture for robust medical image segmentation

Deep learning has shown great potential for automated medical image segmentation to improve the precision and speed of disease diagnostics. However, the task presents significant difficulties due to variations in the scale, shape, texture, and contrast of the pathologies. Traditional convolutional neural network (CNN) models have certain limitations when it comes to effectively modelling multiscale context information and facilitating information interaction between skip connections across levels. To overcome these limitations, a novel deep learning architecture is introduced for medical image segmentation, which takes advantage of CNNs and vision transformers. Our proposed model, named TBConvL-Net, involves a hybrid network that combines the local features of a CNN encoder–decoder architecture with long-range and temporal dependencies using biconvolutional long-short-term memory (LSTM) networks and vision transformers (ViT). This enables the model to capture contextual channel relationships in the data and account for the uncertainty of the segmentation over time. Additionally, we introduce a novel composite loss function that considers both segmentation robustness and boundary agreement of the predicted output with the gold standard. Our proposed model shows consistent improvement over the state of the art on ten publicly available datasets of seven different medical imaging modalities.

TPAFNet: Transformer-Driven Pyramid Attention Fusion Network for 3D Medical Image Segmentation.

The field of 3D medical image segmentation is witnessing a growing trend in the utilization of combined networks that integrate convolutional neural networks and transformers. Nevertheless, prevailing hybrid networks are confronted with limitations in their straightforward serial or parallel combination methods and lack an effective mechanism to fuse channel and spatial feature attention. To address these limitations, we present a robust multi-scale 3D medical image segmentation network, the Transformer-Driven Pyramid Attention Fusion Network, which is denoted as TPAFNet, leveraging a hybrid structure of CNN and transformer. Within this framework, we exploit the characteristics of atrous convolution to extract multi-scale information effectively, thereby enhancing the encoding results of the transformer. Furthermore, we introduce the TPAF block in the encoder to seamlessly fuse channel and spatial feature attention from multi-scale feature inputs. In contrast to conventional skip connections that simply concatenate or add features, our decoder is enriched with a TPAF connection, elevating the integration of feature attention between low-level and high-level features. Additionally, we propose a low-level encoding shortcut from the original input to the decoder output, preserving more original image features and contributing to enhanced results. Finally, the deep supervision is implemented using a novel CNN-based voxel-wise classifier to facilitate better network convergence. Experimental results demonstrate that TPAFNet significantly outperforms other state-of-the-art networks on two public datasets, indicating that our research can effectively improve the accuracy of medical image segmentation, thereby assisting doctors in making more precise diagnoses.

A robust image segmentation and synthesis pipeline for histopathology

Significant diagnostic variability between and within observers persists in pathology, despite the fact that digital slide images provide the ability to measure and quantify features much more precisely compared to conventional methods. Automated and accurate segmentation of cancerous cell and tissue regions can streamline the diagnostic process, providing insights into the cancer progression, and helping experts decide on the most effective treatment. Here, we evaluate the performance of the proposed PathoSeg model, with an architecture comprising of a modified HRNet encoder and a UNet++ decoder integrated with a CBAM block to utilize attention mechanism for an improved segmentation capability. We demonstrate that PathoSeg outperforms the current state-of-the-art (SOTA) networks in both quantitative and qualitative assessment of instance and semantic segmentation. Notably, we leverage the use of synthetic data generated by PathopixGAN, which effectively addresses the data imbalance problem commonly encountered in histopathology datasets, further improving the performance of PathoSeg. It utilizes spatially adaptive normalization within a generative and discriminative mechanism to synthesize diverse histopathological environments dictated through semantic information passed through pixel-level annotated Ground Truth semantic masks.Besides, we contribute to the research community by providing an in-house dataset that includes semantically segmented masks for breast carcinoma tubules (BCT), micro/macrovesicular steatosis of the liver (MSL), and prostate carcinoma glands (PCG). In the first part of the dataset, we have a total of 14 whole slide images from 13 patients’ liver, with fat cell segmented masks, totaling 951 masks of size 512 × 512 pixels. In the second part, it includes 17 whole slide images from 13 patients with prostate carcinoma gland segmentation masks, amounting to 30,000 masks of size 512 × 512 pixels. In the third part, the dataset contains 51 whole slides from 36 patients, with breast carcinoma tubule masks totaling 30,000 masks of size 512 × 512 pixels. To ensure transparency and encourage further research, we will make this dataset publicly available for non-commercial and academic purposes. To facilitate reproducibility and encourage further research, we will also make our code and pre-trained models publicly available at https://github.com/DeepMIALab/PathoSeg.

Learning a Single Network for Robust Medical Image Segmentation With Noisy Labels.

Robust segmenting with noisy labels is an important problem in medical imaging due to the difficulty of acquiring high-quality annotations. Despite the enormous success of recent developments, these developments still require multiple networks to construct their frameworks and focus on limited application scenarios, which leads to inflexibility in practical applications. They also do not explicitly consider the coarse boundary label problem, which results in sub-optimal results. To overcome these challenges, we propose a novel Simultaneous Edge Alignment and Memory-Assisted Learning (SEAMAL) framework for noisy-label robust segmentation. It achieves single-network robust learning, which is applicable for both 2D and 3D segmentation, in both Set-HQ-knowable and Set-HQ-agnostic scenarios. Specifically, to achieve single-model noise robustness, we design a Memory-assisted Selection and Correction module (MSC) that utilizes predictive history consistency from the Prediction Memory Bank to distinguish between reliable and non-reliable labels pixel-wisely, and that updates the reliable ones at the superpixel level. To overcome the coarse boundary label problem, which is common in practice, and to better utilize shape-relevant information at the boundary, we propose an Edge Detection Branch (EDB) that explicitly learns the boundary via an edge detection layer with only slight additional computational cost, and we improve the sharpness and precision of the boundary with a thinning loss. Extensive experiments verify that SEAMAL outperforms previous works significantly.

SSCFormer: Revisiting ConvNet-Transformer Hybrid Framework From Scale-Wise and Spatial-Channel-Aware Perspectives for Volumetric Medical Image Segmentation.

Accurate and robust medical image segmentation is crucial for assisting disease diagnosis, making treatment plan, and monitoring disease progression. Adaptive to different scale variations and regions of interest is essential for high accuracy in automatic segmentation methods. Existing methods based on the U-shaped architecture respectively tackling intra- and inter-scale problem with a hierarchical encoder, however, are restricted by the scope of multi-scale modeling. In addition, global attention and scaling attention in regions of interest have not been appropriately adopted, especially for the salient features. To address these two issues, we propose a ConvNet-Transformer hybrid framework named SSCFormer for accurate and versatile medical image segmentation. The intra-scale ResInception and inter-scale transformer bridge are designed to collaboratively capture the intra- and inter-scale features, facilitating the interaction of small-scale disparity information at a single stage with large-scale from multiple stages. Global attention and scaling attention are cleverly integrated from a spatial-channel-aware perspective. The proposed SSCFormer is tested on four different medical image segmentation tasks. Comprehensive experimental results show that SSCFormer outperforms the current state-of-the-art methods.

A geometric approach to robust medical image segmentation

Robustness of deep learning segmentation models is crucial for their safe incorporation into clinical practice. However, these models can falter when faced with distributional changes. This challenge is evident in magnetic resonance imaging (MRI) scans due to the diverse acquisition protocols across various domains, leading to differences in image characteristics such as textural appearances. We posit that the restricted anatomical differences between subjects could be harnessed to refine the latent space into a set of shape components. The learned set then aims to encompass the relevant anatomical shape variation found within the patient population.We explore this by utilising multiple MRI sequences to learn texture invariant and shape equivariant features which are used to construct a shape dictionary using vector quantisation. We investigate shape equivariance to a number of different types of groups. We hypothesise and prove that the greater the group order, i.e., the denser the constraint, the better becomes the model robustness. We achieve shape equivariance either with a contrastive based approach or by imposing equivariant constraints on the convolutional kernels. The resulting shape equivariant dictionary is then sampled to compose the segmentation output. Our method achieves state-of-the-art performance for the task of single domain generalisation for prostate and cardiac MRI segmentation. Code is available at https://github.com/AinkaranSanthi/A_Geometric_Perspective_For_Robust_Segmentation.

DentalSegmentator: Robust open source deep learning-based CT and CBCT image segmentation

ObjectivesSegmentation of anatomical structures on dento-maxillo-facial (DMF) computed tomography (CT) or cone beam computed tomography (CBCT) scans is increasingly needed in digital dentistry. The main aim of this research was to propose and evaluate a novel open source tool called DentalSegmentator for fully automatic segmentation of five anatomical structures on DMF CT and CBCT scans: maxilla/upper skull, mandible, upper teeth, lower teeth, and the mandibular canal. MethodsA retrospective sample of 470 CT and CBCT scans was used as a training/validation set. The performance and generalizability of the tool was evaluated by comparing segmentations provided by experts and automatic segmentations in two hold-out test datasets: an internal dataset of 133 CT and CBCT scans acquired before orthognathic surgery and an external dataset of 123 CBCT scans randomly sampled from routine examinations in 5 institutions. ResultsThe mean overall results in the internal test dataset (n = 133) were a Dice similarity coefficient (DSC) of 92.2 ± 6.3 % and a normalised surface distance (NSD) of 98.2 ± 2.2 %. The mean overall results on the external test dataset (n = 123) were a DSC of 94.2 ± 7.4 % and a NSD of 98.4 ± 3.6 %. ConclusionsThe results obtained from this highly diverse dataset demonstrate that this tool can provide fully automatic and robust multiclass segmentation for DMF CT and CBCT scans. To encourage the clinical deployment of DentalSegmentator, the pre-trained nnU-Net model has been made publicly available along with an extension for the 3D Slicer software. Clinical significanceDentalSegmentator open source 3D Slicer extension provides a free, robust, and easy-to-use approach to obtaining patient-specific three-dimensional models from CT and CBCT scans. These models serve various purposes in a digital dentistry workflow, such as visualization, treatment planning, intervention, and follow-up.

An empirical study on the robustness of the segment anything model (SAM)

The Segment Anything Model (SAM) is a foundation model for general image segmentation. Although it exhibits impressive performance predominantly on natural images, understanding its robustness against various image perturbations and domains is critical for real-world applications where such challenges frequently arise. In this study we conduct a comprehensive robustness investigation of SAM under diverse real-world conditions. Our experiments encompass a wide range of image perturbations. Our experimental results demonstrate that SAM’s performance generally declines under perturbed images, with varying degrees of vulnerability across different perturbations. By customizing prompting techniques and leveraging domain knowledge based on the unique characteristics of each dataset, the model’s resilience to these perturbations can be enhanced, addressing dataset-specific challenges. This work sheds light on the limitations and strengths of SAM in real-world applications, promoting the development of more robust and versatile image segmentation solutions. Our code is available at https://github.com/EternityYW/SAM-Robustness/.

Robust Semi-Supervised 3D Medical Image Segmentation With Diverse Joint-Task Learning and Decoupled Inter-Student Learning.

Semi-supervised segmentation is highly significant in 3D medical image segmentation. The typical solutions adopt a teacher-student dual-model architecture, and they constrain the two models' decision consistency on the same segmentation task. However, the scarcity of medical samples can lower the diversity of tasks, reducing the effectiveness of consistency constraint. The issue can further worsen as the weights of the models gradually become synchronized. In this work, we have proposed to construct diverse joint-tasks using masked image modelling for enhancing the reliability of the consistency constraint, and develop a novel architecture consisting of a single teacher but multiple students to enjoy the additional knowledge decoupled from the synchronized weights. Specifically, the teacher and student models 'see' varied randomly-masked versions of an input, and are trained to segment the same targets but reconstruct different missing regions concurrently. Such joint-task of segmentation and reconstruction can have the two learners capture related but complementary features to derive instructive knowledge when constraining their consistency. Moreover, two extra students join the original one to perform an inter-student learning. The three students share the same encoding but different decoding designs, and learn decoupled knowledge by constraining their mutual consistencies, preventing themselves from suboptimally converging to the biased predictions of the dictatorial teacher. Experimental on four medical datasets show that our approach performs better than six mainstream semi-supervised methods. Particularly, our approach achieves at least 0.61% and 0.36% higher Dice and Jaccard values, respectively, than the most competitive approach on our in-house dataset. The code will be released at https://github.com/zxmboshi/DDL.

Enhanced Chimp Optimization Algorithm Using Attack Defense Strategy and Golden Update Mechanism for Robust COVID-19 Medical Image Segmentation

Enhanced Chimp Optimization Algorithm Using Attack Defense Strategy and Golden Update Mechanism for Robust COVID-19 Medical Image Segmentation

Robust image segmentation and bias field correction model based on image structural prior constraint

In this paper, we propose an advanced variational model for image segmentation and bias correction. In contrast to the majority of existing level set segmentation models that only consider illumination bias fields, we additionally consider the impact of image reflectance on segmentation accuracy. Our method is capable of effectively segmenting images with blurry edge structures affected by non-uniform illumination. In order to enhance segmentation efficiency, we directly segment the underlying structures of the images, construct spatial prior and apply adaptive regularization constraints on the structural component. Therefore, in the process of segmentation, the proposed algorithm can accurately identify object boundaries without being affected by the environment. Besides, the GL operator is applied to enhance the robustness of the model against noise. Furthermore, we use the alternating direction method of multipliers and the operator splitting algorithm for numerical solution. The experimental results obtained from various sorts of images illustrate that our model outperforms many leading-edge level set models with regard to robustness, corrected results and accuracy.

CI-UNet: melding convnext and cross-dimensional attention for robust medical image segmentation.

Deep learning-based methods have recently shown great promise in medical image segmentation task. However, CNN-based frameworks struggle with inadequate long-range spatial dependency capture, whereas Transformers suffer from computational inefficiency and necessitate substantial volumes of labeled data for effective training. To tackle these issues, this paper introduces CI-UNet, a novel architecture that utilizes ConvNeXt as its encoder, amalgamating the computational efficiency and feature extraction capabilities. Moreover, an advanced attention mechanism is proposed to captures intricate cross-dimensional interactions and global context. Extensive experiments on two segmentation datasets, namely BCSD, and CT2USforKidneySeg, confirm the excellent performance of the proposed CI-UNet as compared to other segmentation methods.

Toward Robust Referring Image Segmentation.

Referring Image Segmentation (RIS) is a fundamental vision-language task that outputs object masks based on text descriptions. Many works have achieved considerable progress for RIS, including different fusion method designs. In this work, we explore an essential question, "What if the text description is wrong or misleading?" For example, the described objects are not in the image. We term such a sentence as a negative sentence. However, existing solutions for RIS cannot handle such a setting. To this end, we propose a new formulation of RIS, named Robust Referring Image Segmentation (R-RIS). It considers the negative sentence inputs besides the regular positive text inputs. To facilitate this new task, we create three R-RIS datasets by augmenting existing RIS datasets with negative sentences and propose new metrics to evaluate both types of inputs in a unified manner. Furthermore, we propose a new transformer-based model, called RefSegformer, with a token-based vision and language fusion module. Our design can be easily extended to our R-RIS setting by adding extra blank tokens. Our proposed RefSegformer achieves state-of-the-art results on both RIS and R-RIS datasets, establishing a solid baseline for both settings. Our project page is at https://github.com/jianzongwu/robust-ref-seg.

Attention-Guided and Noise-Resistant Learning for Robust Medical Image Segmentation

Attention-Guided and Noise-Resistant Learning for Robust Medical Image Segmentation

Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation

Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation

Learning generalizable visual representation via adaptive spectral random convolution for medical image segmentation

Medical image segmentation models often fail to generalize well when applied to new datasets, hindering their usage in clinical practice. Existing random-convolution-based domain generalization approaches, which involve randomizing the convolutional kernel weights in the initial layers of CNN models, have shown promise in improving model generalizability. Nevertheless, the indiscriminate introduction of high-frequency noise during early feature extraction may pollute the critical fine details and degrade the model’s performance on new datasets. To mitigate this problem, we propose an adaptive spectral random convolution (ASRConv) module designed to selectively randomize low-frequency features while avoiding the introduction of high-frequency artifacts. Unlike prior arts, ASRConv dynamically generates convolution kernel weights, enabling more effective control over feature frequencies than randomized kernels. Specifically, ASRConv achieves this selective randomization through a novel weight generation module conditioned on random noise inputs. The adversarial domain augmentation strategy guides the weight generation module in adaptively suppressing high-frequency noise during training, allowing ASRConv to improve feature diversity and reduce overfitting to specific domains. Extensive experimental results show that our proposed ASRConv method consistently outperforms the state-of-the-art methods, with average DSC improvements of 3.07% and 1.18% on fundus and polyp datasets, respectively. We also qualitatively demonstrate the robustness of our model against domain distribution shifts. All these results validate the effectiveness of the proposed ASRConv in learning domain-invariant representations for robust medical image segmentation.

Optimized detection and segmentation of nuclei in gastric cancer images using stain normalization and blurred artifact removal

Histological analysis with microscopy is the gold standard to diagnose and stage cancer, where slides or whole slide images are analyzed for cell morphological and spatial features by pathologists. The nuclei of cancerous cells are characterized by nonuniform chromatin distribution, irregular shapes, and varying size. As nucleus area and shape alone carry prognostic value, detection and segmentation of nuclei are among the most important steps in disease grading. However, evaluation of nuclei is a laborious, time-consuming, and subjective process with large variation among pathologists. Recent advances in digital pathology have allowed significant applications in nuclei detection, segmentation, and classification, but automated image analysis is greatly affected by staining factors, scanner variability, and imaging artifacts, requiring robust image preprocessing, normalization, and segmentation methods for clinically satisfactory results. In this paper, we aimed to evaluate and compare the digital image analysis techniques used in clinical pathology and research in the setting of gastric cancer. A literature review was conducted to evaluate potential methods of improving nuclei detection. Digitized images of 35 patients from a retrospective cohort of gastric adenocarcinoma at Oulu University Hospital in 1987–2016 were annotated for nuclei (n = 9085) by expert pathologists and 14 images of different cancer types from public TCGA dataset with annotated nuclei (n = 7000) were used as a comparison to evaluate applicability in other cancer types. The detection and segmentation accuracy with the selected color normalization and stain separation techniques were compared between the methods. The extracted information can be supplemented by patient’s medical data and fed to the existing statistical clinical tools or subjected to subsequent AI-assisted classification and prediction models. The performance of each method is evaluated by several metrics against the annotations done by expert pathologists. The F1-measure of 0.854 ± 0.068 is achieved with color normalization for the gastric cancer dataset, and 0.907 ± 0.044 with color deconvolution for the public dataset, showing comparable results to the earlier state-of-the-art works. The developed techniques serve as a basis for further research on application and interpretability of AI-assisted tools for gastric cancer diagnosis.

GSAL: Geometric structure adversarial learning for robust medical image segmentation

Automatic medical image segmentation plays a crucial role in clinical diagnosis and treatment. However, it is still a challenging task due to the complex interior characteristics (e.g., inconsistent intensity, low contrast, texture heterogeneity) and ambiguous external boundary structures. In this paper, we introduce a novel geometric structure learning mechanism (GSLM) to overcome the limitations of existing segmentation models that lack learning ”focus, path, and difficulty.” The geometric structure in this mechanism is jointly characterized by the skeleton-like structure extracted by the mask distance transform (MDT) and the boundary structure extracted by the mask distance inverse transform (MDIT). Among them, the skeleton-like and boundary pay attention to the trend of interior characteristics consistency and external structure continuity, respectively. With this idea, we design GSAL, a novel end-to-end geometric structure adversarial learning for robust medical image segmentation. GSAL has four components: a geometric structure generator, which yields the geometric structure to learn the most discriminative features that preserve interior characteristics consistency and external boundary structure continuity, skeleton-like and boundary structure discriminators, which enhance and correct the characterization of internal and external geometry to mutually promote the capture of global contextual dependencies, and a geometric structure fusion sub-network, which fuses the two complementary and refined skeleton-like and boundary structures to generate the high-quality segmentation results. The proposed approach has been successfully applied to three different challenging medical image segmentation tasks, including polyp segmentation, COVID-19 lung infection segmentation, and lung nodule segmentation. Extensive experimental results demonstrate that the proposed GSAL achieves favorably against most state-of-the-art methods under different evaluation metrics. The code is available at: https://github.com/DLWK/GSAL.

Analyzing a series of thermal infrared images to identify defects using a hybrid approach that combines robust principal component analysis and image segmentation

Thermography is widely used to identify defects; however, analyzing just the images containing important information instead of analyzing each individual image from a series of thermal infrared images is an important capability. Principal component analysis (PCA) is the traditionally used dimensionality reduction approach for projecting a given dataset onto a low-dimensional space. In doing so, several images can be extracted from this low-dimensional space that contain significant information about the given data. The problem with this is that PCA is sensitive to noise and shadows. Robust PCA (RPCA), Sparse PCA (SPCA), and Kernel PCA (KPCA) have therefore been developed and employed for defect detection. The question then arises as to which method can preserve the most information from a series of thermal infrared images after the data are projected onto the low-dimensional space. This study analyzed a series of thermal infrared images covered with shadows using different PCAs. By comparing the results, it was found that RPCA can preserve the most information of the given data. Image segmentation was employed to segment the extracted images such that the thermal patterns could be easily identified from the segmented regions. The shadow locations could be identified by selecting the areas whose shadow effects estimated using image segmentation were less than 1.0. The results were encouraging and the proposed scheme can be feasibly employed for defect detection.