3D Segmentation Research Articles

Petal segmentation in CT images based on divide-and-conquer strategy.

Manual segmentation of the petals of flower computed tomography (CT) images is time-consuming and labor-intensive because the flower has many petals. In this study, we aim to obtain a three-dimensional (3D) structure of Camellia japonica flowers and propose a petal segmentation method using computer vision techniques. Petal segmentation on the slice images fails by simply applying the segmentation methods because the shape of the petals in CT images differs from that of the objects targeted by the latest instance segmentation methods. To overcome these challenges, we crop two-dimensional (2D) long rectangles from each slice image and apply the segmentation method to segment the petals on the images. Thanks to cropping, it is easier to segment the shape of the petals in the cropped images using the segmentation methods. We can also use the latest segmentation method for the task because the number of images used for training is augmented by cropping. Subsequently, the results are integrated into 3D to obtain 3D segmentation volume data. The experimental results show that the proposed method can segment petals on slice images with higher accuracy than the method without cropping. The 3D segmentation results were also obtained and visualized successfully.

Differences in the Prognostic Role of Age, Extent of Resection, and Tumor Grade between Astrocytoma IDHmt and Oligodendroglioma: A Single-Center Cohort Study.

IDH-mutant glioma is classified as oligodendroglioma or astrocytoma based on 1p19q-codeletion. Whether prognostic factors are similar between these tumor types is not well understood. Retrospective cohort study. Molecular characterization was performed with targeted next-generation sequencing. Tumor volumes were calculated using semiautomatic 3D segmentation on all pre- and post-operative MRI scans. Overall survival was assessed with the Cox-proportional hazards model. A total of 383 patients with newly diagnosed IDH-mutant glioma were followed up for a median of 7.2 years. Grades 3 and 4 patients had significantly lower Karnofsky performance, with tumors having more contrast enhancement. Patients also received more aggressive postsurgery treatment. Postoperative tumor volume is significantly and independently associated with survival (HR, per cm3 1.19; 95% CI, 1.03-1.39) in IDH-mutant glioma. A separate analysis of oligodendroglioma and astrocytoma showed a significant association of postoperative tumor volume in astrocytoma but not in oligodendroglioma. Higher age and histologic tumor grade were associated with worse survival in patients with oligodendroglioma but not with astrocytoma. Our data support an initial strategy of extensive resection in patients with oligodendroglioma and astrocytoma. Other important prognostic factors differ between these tumor types, urging researchers and clinicians to keep treating these tumors as separate entities.

Bone remodeling after revision total hip arthroplasty for large acetabular defects.

Large acetabular bone defects are challenging in hip revision surgery. Clinical assessment is crucial to evaluate modern technologies in surgical reconstruction. We aimed to better understand the bone remodeling that occurs following acetabular reconstruction. Our objectives were: (1) To characterize changes in the shape of the pelvis by studying sequential computed tomography (CT) scans collected immediately and 1-year postoperatively and (2) to identify which part of the pelvis is most susceptible to remodeling. We used the CT scans taken at two timepoints, of 24 patients with acetabular bone defects classified as Paprosky IIIB, treated with three-dimensional (3D)-printed custom-made acetabular implants. Segmented 3D models of the bony pelvis were co-registered using three different techniques. A global co-registration of the full pelvis was conducted, followed by the co-registration of the innominate bone and then ilium only, on the ipsilateral reconstructed side. The relative movements of the ilium, ischium, and pubis were analyzed from visual inspection and using co-registration metrics (root mean square error and intersection over union). No bone remodeling was found in 14/24 patients (58%). The co-registration of the innominate bone indicated bone remodeling in five cases (21%), while the remaining five cases (21%) presented remodeling in the global co-registration but not the innominate bone co-registration, suggestive of changes occurring at the sacroiliac joint. Changes in the pelvic shape were greatest at the pubis and ischium. Bone remodeling may occur in complex cases of Paprosky type IIIB defects, after acetabular reconstruction (occurrence of 21%, 5/24 cases). Surgeons and engineers should consider this when monitoring implant migration.

Brain image data processing using collaborative data workflows on Texera.

In the realm of neuroscience, mapping the three-dimensional (3D) neural circuitry and architecture of the brain is important for advancing our understanding of neural circuit organization and function. This study presents a novel pipeline that transforms mouse brain samples into detailed 3D brain models using a collaborative data analytics platform called "Texera." The user-friendly Texera platform allows for effective interdisciplinary collaboration between team members in neuroscience, computer vision, and data processing. Our pipeline utilizes the tile images from a serial two-photon tomography/TissueCyte system, then stitches tile images into brain section images, and constructs 3D whole-brain image datasets. The resulting 3D data supports downstream analyses, including 3D whole-brain registration, atlas-based segmentation, cell counting, and high-resolution volumetric visualization. Using this platform, we implemented specialized optimization methods and obtained significant performance enhancement in workflow operations. We expect the neuroscience community can adopt our approach for large-scale image-based data processing and analysis.

Adherent Peanut Image Segmentation Based on Multi-Modal Fusion.

Aiming at the problem of the difficult segmentation of adherent images due to the not fully convex shape of peanut pods, their complex surface texture, and their diverse structures, a multimodal fusion algorithm is proposed to achieve a 2D segmentation of adherent peanut images with the assistance of 3D point clouds. Firstly, the point cloud of a running peanut is captured line by line using a line structured light imaging system, and its three-dimensional shape is obtained through splicing and combining it with a local surface-fitting algorithm to calculate a normal vector and curvature. Seed points are selected based on the principle of minimum curvature, and neighboring points are searched using the KD-Tree algorithm. The point cloud is filtered and segmented according to the normal angle and the curvature threshold until achieving the completion of the point cloud segmentation of the individual peanut, and then the two-dimensional contour of the individual peanut model is extracted by using the rolling method. The search template is established, multiscale feature matching is implemented on the adherent image to achieve the region localization, and finally, the segmentation region is optimized by an opening operation. The experimental results show that the algorithm improves the segmentation accuracy, and the segmentation accuracy reaches 96.8%.

Memory-Augmented 3D Point Cloud Semantic Segmentation Network for Intelligent Mining Shovels.

The semantic segmentation of the 3D operating environment represents the key to intelligent mining shovels' autonomous digging and loading operation. However, the complexity of the operating environment of intelligent mining shovels presents challenges, including the variety of scene targets and the uneven number of samples. This results in low accuracy of 3D semantic segmentation and reduces the autonomous operation accuracy of the intelligent mine shovels. To solve these issues, this paper proposes a 3D point cloud semantic segmentation network based on memory enhancement and lightweight attention mechanisms. This model addresses the challenges of an uneven number of sampled scene targets, insufficient extraction of key features to reduce the semantic segmentation accuracy, and an insufficient lightweight level of the model to reduce deployment capability. Firstly, we investigate the memory enhancement learning mechanism, establishing a memory module for key semantic features of the targets. Furthermore, we address the issue of forgetting non-dominant target point cloud features caused by the unbalanced number of samples and enhance the semantic segmentation accuracy. Subsequently, the channel attention mechanism is studied. An attention module based on the statistical characteristics of the channel is established. The adequacy of the expression of the key features is improved by adjusting the weights of the features. This is done in order to improve the accuracy of semantic segmentation further. Finally, the lightweight mechanism is studied by adopting the deep separable convolution instead of conventional convolution to reduce the number of model parameters. Experiments demonstrate that the proposed method can improve the accuracy of semantic segmentation in the 3D scene and reduce the model's complexity. Semantic segmentation accuracy is improved by 7.15% on average compared with the experimental control methods, which contributes to the improvement of autonomous operation accuracy and safety of intelligent mining shovels.

Instance Consistency Regularization for Semi-Supervised 3D Instance Segmentation.

Large-scale datasets with point-wise semantic and instance labels are crucial to 3D instance segmentation but also expensive. To leverage unlabeled data, previous semi-supervised 3D instance segmentation approaches have explored self-training frameworks, which rely on high-quality pseudo labels for consistency regularization. They intuitively utilize both instance and semantic pseudo labels in a joint learning manner. However, semantic pseudo labels contain numerous noise derived from the imbalanced category distribution and natural confusion of similar but distinct categories, which leads to severe collapses in self-training. Motivated by the observation that 3D instances are non-overlapping and spatially separable, we ask whether we can solely rely on instance consistency regularization for improved semi-supervised segmentation. To this end, we propose a novel self-training network InsTeacher3D to explore and exploit pure instance knowledge from unlabeled data. We first build a parallel base 3D instance segmentation model DKNet, which distinguishes each instance from the others via discriminative instance kernels without reliance on semantic segmentation. Based on DKNet, we further design a novel instance consistency regularization framework to generate and leverage high-quality instance pseudo labels. Experimental results on multiple large-scale datasets show that the InsTeacher3D significantly outperforms prior state-of-the-art semi-supervised approaches.

FDiff-Fusion: Denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation

In recent years, the denoising diffusion model has achieved remarkable success in image segmentation modeling. With its powerful nonlinear modeling capabilities and superior generalization performance, denoising diffusion models have gradually been applied to medical image segmentation tasks, bringing new perspectives and methods to this field. However, existing methods overlook the uncertainty of segmentation boundaries and the fuzziness of regions, resulting in the instability and inaccuracy of the segmentation results. To solve this problem, a denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation (FDiff-Fusion) is proposed in this paper. By integrating the denoising diffusion model into the classical U-Net network, this model can effectively extract rich semantic information from input medical images, thus providing excellent pixel-level representation for medical image segmentation. In this paper, a fuzzy learning module is designed on the skip path of U-Net network because of the widespread boundary uncertainty and region blurring of medical image segmentation. The module sets several fuzzy membership functions for the input encoded features to describe the similarity degree between the feature points, and applies fuzzy rules to the fuzzy membership functions, thus enhancing the modeling ability of the model for uncertain boundaries and fuzzy regions. In addition, in order to improve the accuracy and robustness of the model segmentation results, we introduced an iterative attention feature fusion method in the test phase, which added local context information to the global context information in the attention module to fuse the prediction results of each denoising time step. Finally, to validate the effectiveness of FDiff-Fusion, we compare it with existing advanced segmentation networks on the BRATS 2020 brain tumor dataset and the BTCV abdominal multi-organ dataset. The results show that FDiff-Fusion significantly improves the Dice scores and HD95 distance on these two datasets, demonstrating its superiority in medical image segmentation tasks.

3D segmentation and 3D printing in presurgical planning and patient education in sinonasal tumoral pathologies

3D segmentation and 3D printing in presurgical planning and patient education in sinonasal tumoral pathologies

The added value of an AI-based body composition analysis in a lung cancer screening population: preliminary results

Background and aimsBody composition has been linked with clinical and prognostic outcomes in patients with cancer and cardiovascular diseases. Body composition analysis in lung cancer screening (LCS) is very limited. This study aimed at assessing the association of subcutaneous fat volume (SFV) and subcutaneous fat density (SFD), measured on chest ultra-low dose computed tomography (ultra-LDCT) images by a fully automated artificial intelligence (AI)-based software, with clinical and anthropometric characteristics in a LCS population. Methods and resultsDemographic, clinical, and dietary data were obtained from the written questionnaire completed by each participant at the first visit, when anthropometric measurements, blood sample collection and chest ultra-LDCT were performed. Images were analyzed for automated 3D segmentation of subcutaneous fat and muscle.The analysis included 938 volunteers (372 females); men with a smoking history of ≥40 pack-years had higher SFV (p = 0.0009), while former smokers had lower SFD (p = 0.0019). In female participants, SFV and SFD differed significantly according to age. SFV increased with rising BMI, waist circumference, waist-hip ratio, and CRP levels ≥2 mg/L (p < 0.0001), whereas SFD decreased with rising BMI, waist circumference, waist-hip ratio, and CRP levels ≥2 mg/L (p < 0.001) in both sexes. SFV was associated with glycemia and triglycerides levels (p = 0.0067 and p=<0.0001 in males, p = 0.0074 and p < 0.0001 in females, respectively), while SFD with triglycerides levels (p < 0.0001). ConclusionWe observed different associations of SFV and SFD with age and smoking history between men and women, whereas the association with anthropometric data, CRP, glycemia and triglycerides levels was similar in the two sexes.

Clustering-Based Dual-Population Co-Evolutionary Algorithm for 2D Segmentation Coding in NAND Flash Memory

Clustering-Based Dual-Population Co-Evolutionary Algorithm for 2D Segmentation Coding in NAND Flash Memory

Language-Level Semantics-Conditioned 3D Point Cloud Segmentation

In this work, a language-level Semantics-Conditioned framework for 3D Point cloud segmentation, called SeCondPoint, is proposed, where language-level semantics are introduced to condition the modeling of the point feature distribution, as well as the pseudo-feature generation, and a feature–geometry-based Mixup approach is further proposed to facilitate the distribution learning. Since a large number of point features could be generated from the learned distribution thanks to the semantics-conditioned modeling, any existing segmentation network could be embedded into the proposed framework to boost its performance. In addition, the proposed framework has the inherent advantage of dealing with novel classes, which seems an impossible feat for the current segmentation networks. Extensive experimental results on two public datasets demonstrate that three typical segmentation networks could achieve significant improvements over their original performances after enhancement by the proposed framework in the conventional 3D segmentation task. Two benchmarks are also introduced for a newly introduced zero-shot 3D segmentation task, and the results also validate the proposed framework.

Beyond local patches: Preserving global–local interactions by enhancing self-attention via 3D point cloud tokenization

Transformer-based architectures have recently shown impressive performance on various point cloud understanding tasks such as 3D object shape classification and semantic segmentation. Particularly, this can be attributed to their self-attention mechanism, which has the ability to capture long-range dependencies. However, current methods have constrained it to operate in local patches due to its quadratic memory constraints. This hinders their generalization ability and scaling capacity due to the loss of non-locality in early layers. To tackle this issue, we propose a window-based transformer architecture that captures long-range dependencies while aggregating information in the local patches. We do this by interacting each window with a set of global point cloud tokens — a representative subset of the entire scene — and augmenting the local geometry through a 3D Histogram of Oriented Gradients (HOG) descriptor. Through a series of experiments on segmentation and classification tasks, we show that our model exceeds the state-of-the-art on S3DIS semantic segmentation (+1.67% mIoU), ShapeNetPart part segmentation (+1.03% instance mIoU) and performs competitively on ScanObjectNN 3D object classification.11The code and trained models shall be made publicly available.

Learning to learn point signature for 3D shape geometry

Point signature is a representation that describes the structural geometry of a point within a neighborhood in 3D shapes. Conventional approaches apply a weight-sharing network, e.g., Graph Neural Network (GNN), to all neighborhoods of all points to directly generate point signatures and gain the generalization ability of the network by extensive training over amounts of samples from scratch. However, such approaches lack the flexibility to rapidly adapt to unseen neighborhood structures and thus cannot generalize well to new point sets. In this paper, we propose a novel meta-learning 3D point signature model, 3D meta point signature (MEPS) network, which is capable of learning robust 3D point signatures. Regarding each point signature learning process as a task, our method obtains an optimized model over the best performance on the distribution of all tasks, generating reliable signatures for new tasks, i.e., signatures of unseen point neighborhoods. Specifically, our MEPS consists of two modules: a base signature learner and a meta signature learner. During training, a base-learner is trained to perform specific signature learning tasks. Meanwhile, a meta-learner is trained to update the base-learner with optimal parameters. During testing, the meta-learner learned with the distribution of all tasks can adaptively change the base-learner parameters to accommodate unseen local neighborhoods. We evaluate our MEPS model on 3D shape correspondence and segmentation. Experimental results demonstrate that our method not only gains significant improvements over the baseline model to achieve state-of-the-art performance, but also is capable of handling unseen 3D geometry. Our implementation is available at https://github.com/hhuang-code/MEPS.

Machine learning of dissection photographs and surface scanning for quantitative 3D neuropathology

We present open-source tools for three-dimensional (3D) analysis of photographs of dissected slices of human brains, which are routinely acquired in brain banks but seldom used for quantitative analysis. Our tools can: (1) 3D reconstruct a volume from the photographs and, optionally, a surface scan; and (2) produce a high-resolution 3D segmentation into 11 brain regions per hemisphere (22 in total), independently of the slice thickness. Our tools can be used as a substitute for ex vivo magnetic resonance imaging (MRI), which requires access to an MRI scanner, ex vivo scanning expertise, and considerable financial resources. We tested our tools on synthetic and real data from two NIH Alzheimer’s Disease Research Centers. The results show that our methodology yields accurate 3D reconstructions, segmentations, and volumetric measurements that are highly correlated to those from MRI. Our method also detects expected differences between post mortem confirmed Alzheimer’s disease cases and controls. The tools are available in our widespread neuroimaging suite ‘FreeSurfer’ (https://surfer.nmr.mgh.harvard.edu/fswiki/PhotoTools).

Machine learning of dissection photographs and surface scanning for quantitative 3D neuropathology.

We present open-source tools for three-dimensional (3D) analysis of photographs of dissected slices of human brains, which are routinely acquired in brain banks but seldom used for quantitative analysis. Our tools can: (1) 3D reconstruct a volume from the photographs and, optionally, a surface scan; and (2) produce a high-resolution 3D segmentation into 11 brain regions per hemisphere (22 in total), independently of the slice thickness. Our tools can be used as a substitute for ex vivo magnetic resonance imaging (MRI), which requires access to an MRI scanner, ex vivo scanning expertise, and considerable financial resources. We tested our tools on synthetic and real data from two NIH Alzheimer's Disease Research Centers. The results show that our methodology yields accurate 3D reconstructions, segmentations, and volumetric measurements that are highly correlated to those from MRI. Our method also detects expected differences between post mortem confirmed Alzheimer's disease cases and controls. The tools are available in our widespread neuroimaging suite 'FreeSurfer' (https://surfer.nmr.mgh.harvard.edu/fswiki/PhotoTools).

Multi-scale Knowledge Transfer Vision Transformer for 3D vessel shape segmentation

In order to facilitate the robust and precise 3D vessel shape extraction and quantification from in-vivo Magnetic Resonance Imaging (MRI), this paper presents a novel multi-scale Knowledge Transfer Vision Transformer (i.e., KT-ViT) for 3D vessel shape segmentation. First, it uniquely integrates convolutional embeddings with transformer in a U-net architecture, which simultaneously responds to local receptive fields with convolution layers and global contexts with transformer encoders in a multi-scale fashion. Therefore, it intrinsically enriches local vessel feature and simultaneously promotes global connectivity and continuity for a more accurate and reliable vessel shape segmentation. Furthermore, to enable using relatively low-resolution (LR) images to segment fine scale vessel shapes, a novel knowledge transfer network is designed to explore the inter-dependencies of data and automatically transfer the knowledge gained from high-resolution (HR) data to the low-resolution handling network at multiple levels, including the multi-scale feature levels and the decision level, through an integration of multi-level loss functions. The modeling capability of fine-scale vessel shape data distribution, possessed by the HR image transformer network, can be transferred to the LR image transformer to enhance its knowledge for fine vessel shape segmentation. Extensive experimental results on public image datasets have demonstrated that our method outperforms all other state-of-the-art deep learning methods.

Balancing High-performance and Lightweight: HL-UNet for 3D Cardiac Medical Image Segmentation

Cardiac magnetic resonance imaging is a crucial tool for analyzing, diagnosing, and formulating treatment plans for cardiovascular diseases. Currently, there is very little research focused on balancing cardiac segmentation performance with lightweight methods. Despite the existence of numerous efficient image segmentation algorithms, they primarily rely on complex and computationally intensive network models, making it challenging to implement them on resource-constrained medical devices. Furthermore, simplified models designed to meet the requirements of device lightweighting may have limitations in comprehending and utilizing both global and local information for cardiac segmentation. Therefore, we propose a novel 3D high-performance lightweight medical image segmentation network, HL-UNet, for application in cardiac image segmentation. Specifically, in HL-UNet, we propose a novel residual-enhanced Adaptive attention (REAA) module that combines residual-enhanced connectivity with an adaptive attention mechanism to efficiently capture key features of input images and optimize their representation capabilities, and integrates the Visual Mamba (VSS) module to enhance the performance of HL-UNet. Compared with large-scale models such as TransUNet, HL-UNet increased the Dice of the right ventricular cavity (RV), left ventricular myocardia (MYO) and left ventricular cavity (LV), the key indicators of cardiac image segmentation, by 1.61%, 5.03% and 0.19%, respectively. At the same time, the Params and FLOPs of the model decreased by 41.3M and 31.05G, respectively. Compared to lightweight models such as the MISSFormer, the HL-UNet improves the Dice of RV, MYO, and LV by 4.11%, 3.82%, and 4.33%, respectively, when the number of parameters and computational complexity are close to or even lower.

DIAS: A dataset and benchmark for intracranial artery segmentation in DSA sequences

The automated segmentation of Intracranial Arteries (IA) in Digital Subtraction Angiography (DSA) plays a crucial role in the quantification of vascular morphology, significantly contributing to computer-assisted stroke research and clinical practice. Current research primarily focuses on the segmentation of single-frame DSA using proprietary datasets. However, these methods face challenges due to the inherent limitation of single-frame DSA, which only partially displays vascular contrast, thereby hindering accurate vascular structure representation. In this work, we introduce DIAS, a dataset specifically developed for IA segmentation in DSA sequences. We establish a comprehensive benchmark for evaluating DIAS, covering full, weak, and semi-supervised segmentation methods. Specifically, we propose the vessel sequence segmentation network, in which the sequence feature extraction module effectively captures spatiotemporal representations of intravascular contrast, achieving intracranial artery segmentation in 2D+Time DSA sequences. For weakly-supervised IA segmentation, we propose a novel scribble learning-based image segmentation framework, which, under the guidance of scribble labels, employs cross pseudo-supervision and consistency regularization to improve the performance of the segmentation network. Furthermore, we introduce the random patch-based self-training framework, aimed at alleviating the performance constraints encountered in IA segmentation due to the limited availability of annotated DSA data. Our extensive experiments on the DIAS dataset demonstrate the effectiveness of these methods as potential baselines for future research and clinical applications. The dataset and code are publicly available at https://doi.org/10.5281/zenodo.11401368 and https://github.com/lseventeen/DIAS.

IMG-13. PREDICTION OF PEDIATRIC MEDULLOBLASTOMA SUBGROUPS USING CLINICO-RADIOMIC ANALYSIS

Abstract BACKGROUND Pediatric medulloblastoma is an aggressive brain tumor and tailored treatment has the potential to lead to better patient outcomes. The WHO 2021 classification of medulloblastoma involves an integrated diagnosis that incorporates genetically defined characteristics, including molecular subgroups (WNT-activated, SHH-activated TP53 wildtype, SHH-activated TP53-mutant, and non-WNT/non-SHH). Clinically acquired radiology (MRI) imaging characteristics could provide a non-invasive, pre-treatment biomarker for such subgroups faster than standard methylation-based turn-around times, and for use in countries where methylation is not available. METHODS Herein, we utilize a multi-institutional dataset of multi-parametric, clinical MRIs of pediatric medulloblastoma patients from the Children’s Brain Tumor Network (median age = 8.9 years) to assess the predictive value of patient-level radiomic and clinical factors. 89 subjects with treatment-naïve T1w/T1w contrast-enhanced/T2w/FLAIR images and molecular subgroups (derived from methylation profiling or RNA sequencing) were included (SHH=18, WNT=10, non-SHH/WNT=61). Radiomic features were extracted from a radiologist-defined volumetric (3D) segmentation for each subject separately (including solid tumor, cystic, and peritumoral edema regions). Clinical variables included sex, age at diagnosis, and metastatic disease. 20 top performing features were selected based on an ANOVA between features/classes and subsequently used to train and evaluate three separate classification models based on radiomic, clinical, or clinico-radiomic features (Linear SVM; leave-one-subject-out cross-validation). RESULTS The combined clinico-radiomic model had the top performance (AUC=0.83) followed by radiomic (0.78) and clinical (0.5) for predicting SHH/WNT vs. non-SHH/WNT groups. Predictive radiomic features included intensity-based statistics (T1w, T2w, T1w-CE), texture (T2w), and morphological shape characteristics. CONCLUSIONS Our findings show potential for early prediction of molecular subgroups using baseline imaging that could lend to faster decision-making in patient treatment planning. Future work aims to evaluate the inclusion of histological characteristics for a full integrated diagnostic approach as well as assessment of prognostic value of the combined multi-omic features.