3D Image Segmentation Research Articles

Neuroimaging of intracranial neoplasms: volume segmentation of 3D images

The progressive development of medical bioinformatics, namely computer vision and deep learning technologies, will allow accurate screening of neurooncological diseases and evaluate the effectiveness of the treatment methods performed in dynamics at all levels of healthcare institutions. At present, the Matlab platform is the most functional environment for image processing with the implemented possibility of using semantic segmentation. An algorithm for creating masks of the area of interest of specialists and a listing for training a neural network in the diagnostic screening of neurooncological patients are presented.

Identification of high-risk population of pneumoconiosis using deep learning segmentation of lung 3D images and radiomics texture analysis

ObjectionThe aim of this study is to develop an early-warning model for identifying high-risk populations of pneumoconiosis by combining lung 3D images and radiomics lung texture features. MethodsA retrospective study was conducted, including 600 dust-exposed workers and 300 confirmed pneumoconiosis patients. Chest computed tomography (CT) images were divided into a training set and a test set in a 2:1 ratio. Whole-lung segmentation was performed using deep learning models for feature extraction of radiomics. Two feature selection algorithms and five classification models were used. The optimal model was selected using a 10-fold cross-validation strategy, and the calibration curve and decision curve were evaluated. To verify the applicability of the model, the diagnostic efficiency and accuracy between the model and human interpretation were compared. Additionally, the risk probabilities for different risk groups defined by the model were compared at different time intervals. ResultsFour radiomics features were ultimately used to construct the predictive model. The logistic regression model was the most stable in both the training set and testing set, with an area under curve (AUC) of 0.964 (95 % confidence interval [CI], 0.950–0.976) and 0.947 (95 %CI, 0.925–0.964). In the training and testing sets, the Brier scores were 0.092 and 0.14, respectively, with threshold probability ranges of 2 %-99 % and 2 %-85 %. These results indicate that the model exhibits good calibration and clinical benefit. The comparison between the model and human interpretation showed that the model was not inferior in terms of diagnostic efficiency and accuracy. Additionally, the high-risk population identified by the model was diagnosed as pneumoconiosis two years later. ConclusionThis study provides a meticulous and quantifiable method for detecting and assessing the risk of pneumoconiosis, building upon accurate diagnosis. Employing risk scoring and probability estimation, not only enhances the efficiency of diagnostic physicians but also provides a valuable reference for controlling the occurrence of pneumoconiosis.

Topology-based segmentation of 3D confocal images of emerging hematopoietic stem cells in the zebrafish embryo

Abstract We develop a novel method for image segmentation of 3D confocal microscopy images of emerging hematopoietic stem cells. The method is based on the theory of persistent homology and uses an optimal threshold to select the most persistent cycles in the persistence diagram. This enables the segmentation of the image’s most contrasted and representative shapes. Coupling this segmentation method with a meshing algorithm, we define a pipeline for 3D reconstruction of confocal volumes. Compared to related methods, this approach improves shape segmentation, is more ergonomic to automatize, and has fewer parameters. We apply it to the segmentation of membranes, at subcellular resolution, of cells involved in the endothelial-to-hematopoietic transition (EHT) in the zebrafish embryos.

Deep-DM: Deep-driven deformable model for 3D image segmentation using limited data.

Objective - Medical image segmentation is essential for several clinical tasks, including diagnosis, surgical and treatment planning, and image-guided interventions. Deep Learning (DL) methods have become the state-of-the-art for several image segmentation scenarios. However, a large and well-annotated dataset is required to effectively train a DL model, which is usually difficult to obtain in clinical practice, especially for 3D images. Methods - In this paper, we proposed Deep-DM, a learning-guided deformable model framework for 3D medical imaging segmentation using limited training data. In the proposed method, an energy function is learned by a Convolutional Neural Network (CNN) and integrated into an explicit deformable model to drive the evolution of an initial surface towards the object to segment. Specifically, the learning-based energy function is iteratively retrieved from localized anatomical representations of the image containing the image information around the evolving surface at each iteration. By focusing on localized regions of interest, this representation excludes irrelevant image information, facilitating the learning process. Results and conclusion - The performance of the proposed method is demonstrated for the tasks of left ventricle and fetal head segmentation in ultrasound, left atrium segmentation in Magnetic Resonance, and bladder segmentation in Computed Tomography, using different numbers of training volumes in each study. The results obtained showed the feasibility of the proposed method to segment different anatomical structures in different imaging modalities. Moreover, the results also showed that the proposed approach is less dependent on the size of the training dataset in comparison with state-of-the-art DL-based segmentation methods, outperforming them for all tasks when a low number of samples is available. Significance - Overall, by offering a more robust and less data-intensive approach to accurately segmenting anatomical structures, the proposed method has the potential to enhance clinical tasks that require image segmentation strategies.

One-Shot Weakly-Supervised Segmentation in 3D Medical Images.

Deep neural networks typically require accurate and a large number of annotations to achieve outstanding performance in medical image segmentation. One-shot and weakly-supervised learning are promising research directions that reduce labeling effort by learning a new class from only one annotated image and using coarse labels instead, respectively. In this work, we present an innovative framework for 3D medical image segmentation with one-shot and weakly-supervised settings. Firstly a propagation-reconstruction network is proposed to propagate scribbles from one annotated volume to unlabeled 3D images based on the assumption that anatomical patterns in different human bodies are similar. Then a multi-level similarity denoising module is designed to refine the scribbles based on embeddings from anatomical- to pixel-level. After expanding the scribbles to pseudo masks, we observe the miss-classified voxels mainly occur at the border region and propose to extract self-support prototypes for the specific refinement. Based on these weakly-supervised segmentation results, we further train a segmentation model for the new class with the noisy label training strategy. Experiments on three CT and one MRI datasets show the proposed method obtains significant improvement over the state-of-the-art methods and performs robustly even under severe class imbalance and low contrast. Code is publicly available at https://github.com/LWHYC/OneShot_WeaklySeg.

SASAN: Spectrum-Axial Spatial Approach Networks for Medical Image Segmentation.

Ophthalmic diseases such as central serous chorioretinopathy (CSC) significantly impair the vision of millions of people globally. Precise segmentation of choroid and macular edema is critical for diagnosing and treating these conditions. However, existing 3D medical image segmentation methods often fall short due to the heterogeneous nature and blurry features of these conditions, compounded by medical image clarity issues and noise interference arising from equipment and environmental limitations. To address these challenges, we propose the Spectrum Analysis Synergy Axial-Spatial Network (SASAN), an approach that innovatively integrates spectrum features using the Fast Fourier Transform (FFT). SASAN incorporates two key modules: the Frequency Integrated Neural Enhancer (FINE), which mitigates noise interference, and the Axial-Spatial Elementum Multiplier (ASEM), which enhances feature extraction. Additionally, we introduce the Self-Adaptive Multi-Aspect Loss ( LSM ), which balances image regions, distribution, and boundaries, adaptively updating weights during training. We compiled and meticulously annotated the Choroid and Macular Edema OCT Mega Dataset (CMED-18k), currently the world's largest dataset of its kind. Comparative analysis against 13 baselines shows our method surpasses these benchmarks, achieving the highest Dice scores and lowest HD95 in the CMED and OIMHS datasets. Our code is publicly available at https://github.com/IMOP-lab/SASAN-Pytorch.

A 3D Medical Image Segmentation Method Based on MCRAIP-Net

A 3D Medical Image Segmentation Method Based on MCRAIP-Net

Mutual Information Guided Diffusion for Zero-Shot Cross-Modality Medical Image Translation.

Cross-modality data translation has attracted great interest in medical image computing. Deep generative models show performance improvement in addressing related challenges. Nevertheless, as a fundamental challenge in image translation, the problem of zero-shot learning cross-modality image translation with fidelity remains unanswered. To bridge this gap, we propose a novel unsupervised zero-shot learning method called Mutual Information guided Diffusion Model, which learns to translate an unseen source image to the target modality by leveraging the inherent statistical consistency of Mutual Information between different modalities. To overcome the prohibitive high dimensional Mutual Information calculation, we propose a differentiable local-wise mutual information layer for conditioning the iterative denoising process. The Local-wise-Mutual-Information-Layer captures identical cross-modality features in the statistical domain, offering diffusion guidance without relying on direct mappings between the source and target domains. This advantage allows our method to adapt to changing source domains without the need for retraining, making it highly practical when sufficient labeled source domain data is not available. We demonstrate the superior performance of MIDiffusion in zero-shot cross-modality translation tasks through empirical comparisons with other generative models, including adversarial-based and diffusion-based models. Finally, we showcase the real-world application of MIDiffusion in 3D zero-shot learning-based cross-modality image segmentation tasks.

SDSL: Spectral Distance Scaling Loss Pretraining SwinUNETR for 3D Medical Image Segmentation

SDSL: Spectral Distance Scaling Loss Pretraining SwinUNETR for 3D Medical Image Segmentation

Three-Dimensional Leaf Edge Reconstruction Combining Two- and Three-Dimensional Approaches

Leaves, crucial for plant physiology, exhibit various morphological traits that meet diverse functional needs. Traditional leaf morphology quantification, largely 2-dimensional (2D), has not fully captured the 3-dimensional (3D) aspects of leaf function. Despite improvements in 3D data acquisition, accurately depicting leaf morphologies, particularly at the edges, is difficult. This study proposes a method for 3D leaf edge reconstruction, combining 2D image segmentation with curve-based 3D reconstruction. Utilizing deep-learning-based instance segmentation for 2D edge detection, structure from motion for estimation of camera positions and orientations, leaf correspondence identification for matching leaves among images, and curve-based 3D reconstruction for estimating 3D curve fragments, the method assembles 3D curve fragments into a leaf edge model through B-spline curve fitting. The method's performances were evaluated on both virtual and actual leaves, and the results indicated that small leaves and high camera noise pose greater challenges to reconstruction. We developed guidelines for setting a reliability threshold for curve fragments, considering factors occlusion, leaf size, the number of images, and camera error; the number of images had a lesser impact on this threshold compared to others. The method was effective for lobed leaves and leaves with fewer than 4 holes. However, challenges still existed when dealing with morphologies exhibiting highly local variations, such as serrations. This nondestructive approach to 3D leaf edge reconstruction marks an advancement in the quantitative analysis of plant morphology. It is a promising way to capture whole-plant architecture by combining 2D and 3D phenotyping approaches adapted to the target anatomical structures.

Visual Boundary-Guided Pseudo-Labeling for Weakly Supervised 3D Point Cloud Segmentation in Indoor Environments.

Accurate segmentation of 3D point clouds in indoor scenes remains a challenging task, often hindered by the labor-intensive nature of data annotation. While weakly supervised learning approaches have shown promise in leveraging partial annotations, they frequently struggle with imbalanced performance between foreground and background elements due to the complex structures and proximity of objects in indoor environments. To address this issue, we propose a novel foreground-aware label enhancement method utilizing visual boundary priors. Our approach projects 3D point clouds onto 2D planes and applies 2D image segmentation to generate pseudo-labels for foreground objects. These labels are subsequently back-projected into 3D space and used to train an initial segmentation model. We further refine this process by incorporating prior knowledge from projected images to filter the predicted labels, followed by model retraining. We introduce this technique as the Foreground Boundary Prior (FBP), a versatile, plug-and-play module designed to enhance various weakly supervised point cloud segmentation methods. We demonstrate the efficacy of our approach on the widely-used 2D-3D-Semantic dataset, employing both random-sample and bounding-box based weak labeling strategies. Our experimental results show significant improvements in segmentation performance across different architectural backbones, highlighting the method's effectiveness and portability.

SkinFormer: Learning Statistical Texture Representation With Transformer for Skin Lesion Segmentation.

Accurate skin lesion segmentation from dermoscopic images is of great importance for skin cancer diagnosis. However, automatic segmentation of melanoma remains a challenging task because it is difficult to incorporate useful texture representations into the learning process. Texture representations are not only related to the local structural information learned by CNN, but also include the global statistical texture information of the input image. In this paper, we propose a transFormer network (SkinFormer) that efficiently extracts and fuses statistical texture representation for Skin lesion segmentation. Specifically, to quantify the statistical texture of input features, a Kurtosis-guided Statistical Counting Operator is designed. We propose Statistical Texture Fusion Transformer and Statistical Texture Enhance Transformer with the help of Kurtosis-guided Statistical Counting Operator by utilizing the transformer's global attention mechanism. The former fuses structural texture information and statistical texture information, and the latter enhances the statistical texture of multi-scale features. Extensive experiments on three publicly available skin lesion datasets validate that our SkinFormer outperforms other SOAT methods, and our method achieves 93.2% Dice score on ISIC 2018. It can be easy to extend SkinFormer to segment 3D images in the future.

HCA-former: Hybrid Convolution Attention Transformer for 3D Medical Image Segmentation

In recent years, Transformers have achieved success in the field of medical image segmentation due to their outstanding capability to model long-range dependencies. However, many existing segmentation methods only use Transformer as an auxiliary module to capture global context information in images, limiting the potential of the Transformers. Additionally, self-attention mechanism within the Transformers can lead to attention collapse issues, thus triggering semantic gap between the encoder and decoder. Furthermore, most networks have difficulties in effectively handling multi-scale and multi-channel feature information. To address the above problems, we propose a hybrid Convolutional Neural Networks (CNNs) and Transformers method for medical image segmentation (HCA-Former). We design a local multi-channel attention block (LMCA) to effectively combine the features of CNN and Transformers, enabling multi-channel information extraction and interaction. Using the Double-Former Block (DFB) alleviates the semantic gap between the encoder and decoder, restoring more detailed information. Moreover, the utilization of the global multi-scale attention block (GMSA) can establish information interaction among multi-scale targets, thereby enhancing generalization capability of the model. To validate the effectiveness of our approach, we evaluate the proposed method on three challenging tasks: the MICCAI 2015 Multi-Image Abdominal Marker Challenge (Synapse), Automated Cardiac Diagnosis Dataset (ACDC), and Medical Segmentation Decathlon Brain Tumor Segmentation (MSD brain tumor). Extensive experiments demonstrate that our HCA-Former achieved competitive or better performance than state-of-the-art approaches for 3D medical image segmentation.

Medical image segmentation of gastric adenocarcinoma based on dense connection of residuals.

Accurate segmentation of gastric cancer based on CT images of gastric adenocarcinoma is crucial for physicians to screen gastric diseases, clinical diagnosis, preoperative prediction, and postoperative evaluation plans. To address the issue of the inability of the segmentation algorithm to depict the correct boundaries due to unclear gastric contours in the lesion area and the visible irregular band-like dense shadow extending to the perigastric region, a 3D medical image segmentation model 3D UNet based on residual dense jumping method is proposed. In the method we proposed, Residual Dense Block, which is applied to the image super-resolution module to remove CT artifacts, and Residual Block in ResNet are further fused. The quality of CT images is improved by Residual Dense Skip Block, which removes banded dense shadows, preserves image details and edge information, captures features, and improves the segmentation performance of gastric adenocarcinoma. The Instance Normalization layer position is modified to select the best result. Different loss functions are also combined in order to obtain the best gastric adenocarcinoma segmentation performance. We tested the model on a hospital-provided gastric adenocarcinoma dataset. The experimental results show that our model outperforms the existing methods in CT gastric adenocarcinoma segmentation, in which the method combining the hybrid loss function of Dice and CE obtains an average dice score of 82.3%, which is improved by 5.3% and 3.8% compared to TransUNet and Hiformer, respectively, and improves the cross-merge rate to 70.8%, compared to nnFormer, nnUNet by 1% and 0.9%, respectively. The residual jump connection structure indeed improves segmentation performance. The proposed method has the potential to be used as a screen for gastric diseases and to assist physicians in diagnosis.

SEMANTIC ENRICHMENT OF 3D POINT CLOUDS USING 2D IMAGE SEGMENTATION

Abstract. 3D point cloud segmentation is computationally intensive due to the lack of inherent structural information and the unstructured nature of the point cloud data, which hinders the identification and connection of neighboring points. Understanding the structure of the point cloud data plays a crucial role in obtaining a meaningful and accurate representation of the underlying 3D environment. In this paper, we propose an algorithm that builds on existing state-of-the-art techniques of 2D image segmentation and point cloud registration to enrich point clouds with semantic information. DeepLab2 with ResNet50 as backbone architecture trained on the COCO dataset is used for indoor scene semantic segmentation into several classes like wall, floor, ceiling, doors, and windows. Semantic information from 2D images is propagated along with other input data, i.e., RGB images, depth images, and sensor information to generate 3D point clouds with semantic information. Iterative Closest Point (ICP) algorithm is used for the pair-wise registration of consecutive point clouds and finally, optimization is applied using the pose graph optimization on the whole set of point clouds to generate the combined point cloud of the whole scene. 3D point cloud of the whole scene contains pseudo-color information which denotes the semantic class to which each point belongs. The proposed methodology use an off-the-shelf 2D semantic segmentation deep learning model to semantically segment 3D point clouds collected using handheld mobile LiDAR sensor. We demonstrate a comparison of the accuracy achieved compared to a manually segmented point cloud on an in-house dataset as well as a 2D3DS benchmark dataset.

Improved 3D image segmentation for X-ray tomographic data of biaxial warp-knitted composites

Identifying different features and phases within the biaxial warp-knitted composites enables accurate characterization of the internal changes of the composites under complex loading conditions and different processing parameters. However, obtaining 3D reconstructions that can be accurately segmented and quantitatively analyzed is still a challenge, primarily due to the low attenuation of materials, especially the low contrast between the weft yarn and warp yarns. Here, we use deep learning approaches to identify the boundary of the section of the tow on the 2D image, extract individual warp or weft tows automatically, and this is followed by computational modeling of composite materials. The results show that the performance of ResUNet++ model is better than that of other models for the biaxial warp-knitted composite specimen segmentation tasks. The special improved dataset augmentation algorithm is also a positive effective way to enhance network performance.

UNesT: Local spatial representation learning with hierarchical transformer for efficient medical segmentation.

Transformer-based models, capable of learning better global dependencies, have recently demonstrated exceptional representation learning capabilities in computer vision and medical image analysis. Transformer reformats the image into separate patches and realizes global communication via the self-attention mechanism. However, positional information between patches is hard to preserve in such 1D sequences, and loss of it can lead to sub-optimal performance when dealing with large amounts of heterogeneous tissues of various sizes in 3D medical image segmentation. Additionally, current methods are not robust and efficient for heavy-duty medical segmentation tasks such as predicting a large number of tissue classes or modeling globally inter-connected tissue structures. To address such challenges and inspired by the nested hierarchical structures in vision transformer, we proposed a novel 3D medical image segmentation method (UNesT), employing a simplified and faster-converging transformer encoder design that achieves local communication among spatially adjacent patch sequences by aggregating them hierarchically. We extensively validate our method on multiple challenging datasets, consisting of multiple modalities, anatomies, and a wide range of tissue classes, including 133 structures in the brain, 14 organs in the abdomen, 4 hierarchical components in the kidneys, inter-connected kidney tumors and brain tumors. We show that UNesT consistently achieves state-of-the-art performance and evaluate its generalizability and data efficiency. Particularly, the model achieves whole brain segmentation task complete ROI with 133 tissue classes in a single network, outperforming prior state-of-the-art method SLANT27 ensembled with 27 networks. Our model performance increases the mean DSC score of the publicly available Colin and CANDI dataset from 0.7264 to 0.7444 and from 0.6968 to 0.7025, respectively. Code, pre-trained models, and use case pipeline are available at: https://github.com/MASILab/UNesT.

DS-Former: A dual-stream encoding-based transformer for 3D medical image segmentation

Models that utilize self-attention mechanisms, including but not limited to Vision Transformers (ViTs), have shown promising performance in visual tasks like semantic segmentation. This is attributed to their capacity to capture global features of images, enabling them to learn more comprehensive representations. However, transformer-based models typically demand a considerable amount of training data to achieve satisfactory performance, while being deficient in the ability to efficiently extract local image features. As a result, these models may not be as effective in some computer vision tasks that involve small-scale datasets, like medical image segmentation. To address these issues, this paper proposes a dual-stream encoding-based transformer dubbed as Dual-stream Transformer (DS-Former). The dual-stream module in DS-Former can simultaneously acquire local and global features in the image and construct relation between the two kinds of features via self-attention. Compared with the simple splicing or serial connection, the dual-stream module can extract more comprehensive and hierarchical feature information from the fusion interaction of the two features. Our method is evaluated on the UK Biobank (UKBB) cardiac magnetic resonance imaging (CMR) dataset and The Beyond the Cranial Vault (BTCV) abdominal challenge dataset. The experimental results indicate that our DS-Former outperforms other state-of-the-art approaches on both datasets, indicating its potential for medical images semantic segmentation.

An evolutionary Chameleon Swarm Algorithm based network for 3D medical image segmentation

Developing 3D Convolutional Neural Networks (CNNs) for medical image segmentation is challenging due to the limited number of available labeled medical images and computational resources. Effective design of a 3D CNN requires a better structure and appropriate hyperparameters. Manually designing a neural network and choosing appropriate hyperparameters requires extensive knowledge, a time-consuming and challenging process, resulting in complex and over-parameterized networks. Neuroevolution is an automatic and effective approach to finding an optimal network structure and hyperparameters. This paper proposes an evolutionary based method using Chameleon Swarm Algorithm (CSA) to develop 3D CNN networks for volume segmentation. We also design a search space to build optimal 3D block-based encoder–decoder structure networks for medical image segmentation. We evaluated the proposed method on two different datasets (CT Spleen and MRI Heart) and compared the results of the obtained model with state-of-the-art models. Furthermore, the comparative results show that our proposed method yields 3D CNN networks with superior performance and fewer parameters than other state-of-the-art models utilized in 3D medical image segmentation.

ReFs: A hybrid pre-training paradigm for 3D medical image segmentation

Self-supervised learning (SSL) has achieved remarkable progress in medical image segmentation. The application of an SSL algorithm often follows a two-stage training process: using unlabeled data to perform label-free representation learning and fine-tuning the pre-trained model on the downstream tasks. One issue of this paradigm is that the SSL step is unaware of the downstream task, which may lead to sub-optimal feature representation for a target task. In this paper, we propose a hybrid pre-training paradigm that is driven by both self-supervised and supervised objectives. To achieve this, a supervised reference task is involved in self-supervised learning, aiming to improve the representation quality. Specifically, we employ the off-the-shelf medical image segmentation task as reference, and encourage learning a representation that (1) incurs low prediction loss on both SSL and reference tasks and (2) leads to a similar gradient when updating the feature extractor from either task. In this way, the reference task pilots SSL in the direction beneficial for the downstream segmentation. To this end, we propose a simple but effective gradient matching method to optimize the model towards a consistent direction, thus improving the compatibility of both SSL and supervised reference tasks. We call this hybrid pre-training paradigm reference-guided self-supervised learning (ReFs), and perform it on a large-scale unlabeled dataset and an additional reference dataset. The experimental results demonstrate its effectiveness on seven downstream medical image segmentation benchmarks.