Volumetric Medical Images Research Articles

DrasCLR: A self-supervised framework of learning disease-related and anatomy-specific representation for 3D lung CT images

Large-scale volumetric medical images with annotation are rare, costly, and time prohibitive to acquire. Self-supervised learning (SSL) offers a promising pre-training and feature extraction solution for many downstream tasks, as it only uses unlabeled data. Recently, SSL methods based on instance discrimination have gained popularity in the medical imaging domain. However, SSL pre-trained encoders may use many clues in the image to discriminate an instance that are not necessarily disease-related. Moreover, pathological patterns are often subtle and heterogeneous, requiring the ability of the desired method to represent anatomy-specific features that are sensitive to abnormal changes in different body parts. In this work, we present a novel SSL framework, named DrasCLR, for 3D lung CT images to overcome these challenges. We propose two domain-specific contrastive learning strategies: one aims to capture subtle disease patterns inside a local anatomical region, and the other aims to represent severe disease patterns that span larger regions. We formulate the encoder using conditional hyper-parameterized network, in which the parameters are dependant on the anatomical location, to extract anatomically sensitive features. Extensive experiments on large-scale datasets of lung CT scans show that our method improves the performance of many downstream prediction and segmentation tasks. The patient-level representation improves the performance of the patient survival prediction task. We show how our method can detect emphysema subtypes via dense prediction. We demonstrate that fine-tuning the pre-trained model can significantly reduce annotation efforts without sacrificing emphysema detection accuracy. Our ablation study highlights the importance of incorporating anatomical context into the SSL framework. Our codes are available at https://github.com/batmanlab/DrasCLR.

A statistical deformation model-based data augmentation method for volumetric medical image segmentation

The accurate delineation of organs-at-risk (OARs) is a crucial step in treatment planning during radiotherapy, as it minimizes the potential adverse effects of radiation on surrounding healthy organs. However, manual contouring of OARs in computed tomography (CT) images is labor-intensive and susceptible to errors, particularly for low-contrast soft tissue. Deep learning-based artificial intelligence algorithms surpass traditional methods but require large datasets. Obtaining annotated medical images is both time-consuming and expensive, hindering the collection of extensive training sets. To enhance the performance of medical image segmentation, augmentation strategies such as rotation and Gaussian smoothing are employed during preprocessing. However, these conventional data augmentation techniques cannot generate more realistic deformations, limiting improvements in accuracy. To address this issue, this study introduces a statistical deformation model-based data augmentation method for volumetric medical image segmentation. By applying diverse and realistic data augmentation to CT images from a limited patient cohort, our method significantly improves the fully automated segmentation of OARs across various body parts. We evaluate our framework on three datasets containing tumor OARs from the head, neck, chest, and abdomen. Test results demonstrate that the proposed method achieves state-of-the-art performance in numerous OARs segmentation challenges. This innovative approach holds considerable potential as a powerful tool for various medical imaging-related sub-fields, effectively addressing the challenge of limited data access.

SwinResNet: Volumetric Medical Image Segmentation by Fusing Swin Transformer and ResNet

SwinResNet: Volumetric Medical Image Segmentation by Fusing Swin Transformer and ResNet

Accelerating Volumetric CT and MRI Imaging by Reference-Free Deep Learning Transformation from Low-Resolution to High-Resolution

High-resolution (HR) images are important in precision radiation oncology. However, acquiring HR volumetric CT and MRI images is often time consuming; also, the resolution in some direction(s) (e.g., z-direction in the case of CT) is often limited by imaging hardware or fundamental imaging principle. Super-resolution (SR) imaging, i.e., the low-resolution (LR) to HR image transformation, is widely used to improve image resolution. Data-driven deep learning (DL) methods have achieved great success in SR imaging, yet they can hardly be applied to medical imaging as they require large amount of LR-HR image pairs to train the model. We therefore propose a reference-free DL method to increase resolutions of volumetric medical images in an efficient way. We propose a maximum likelihood estimation (MLE)-based implicit neural representation (INR) network for SR imaging. The INR network aims to represent an image as a continuous function parameterized by a coordinate-based multi-layer perceptron. The INR network takes image coordinates as input and outputs corresponding pixel intensities. To train the network without using any HR images, we use a MLE framework to model LR observations' statistics and their relation to the latent HR image. The predicted HR image from the INR's output is transformed to LR images based on the MLE, and the network parameters are then optimized by minimizing the distance between the transformed LR images and actual LR observations. We demonstrate the efficacy of the proposed method on CT and MRI images for 2x, 4x, and 8x SR using only one or two LR image(s). The performance is compared with a conventional SR method named plain MLE, in terms of visual quality and numerical qualities of PSNR and SSIM. Our method outperformed the plain MLE method in the experiment. Table 1 reports the numerical improvements of our method over the compared plain MLE method. For 2x SR with a single LR image, our method achieved significant improvements in both PSNR and SSIM. When using two LR images, the better structural restoration capability of our method became more obvious with higher SR magnifications, as indicated by the increased SSIM differences. Better noise suppression capability of our method is observed in all our studies, as indicated by the PSNR values. In visual quality evaluation, we observed sharper image details with less noise in SR images generated by the proposed method, compared with the plain MLE method. The proposed novel reference-free DL method can efficiently provide high-quality HR images with only one or two LR images for CT and MRI imaging. This method can be easily generalized to many other radiation therapy related applications without the requirement for HR reference images.

Multimodal Image Registration Using a Viscous Fluid Model with the Bhattacharyya Distance.

Image registration is an elementary task in medical image processing and analysis, which can be divided into monomodal and multimodal. Direct 3D multimodal registration in volumetric medical images can provide more insight into the interpretation of subsequent image processing applications than 2D methods. This paper is dedicated to the development of a 3D multimodal image registration algorithm based on a viscous fluid model associated with the Bhattacharyya distance. In our approach, a modified Navier-Stoke's equation is exploited as the foundation of the multimodal image registration framework. The hopscotch method is numerically implemented to solve the velocity field, whose values at the explicit locations are first computed and the values at the implicit positions are solved by transposition. The differential of the Bhattacharyya distance is incorporated into the body force function, which is the main driving force for deformation, to enable multimodal registration. A variety of simulated and real brain MR images were utilized to assess the proposed 3D multimodal image registration system. Preliminary experimental results indicated that our algorithm produced high registration accuracy in various registration scenarios and outperformed other competing methods in many multimodal image registration tasks.Clinical Relevance- This facilitates the disease diagnosis and treatment planning that requires accurate 3D multimodal image registration without massive image data and extensive training regardless of the imaging modality.

Uncertainty-guided dual-views for semi-supervised volumetric medical image segmentation

Uncertainty-guided dual-views for semi-supervised volumetric medical image segmentation

Positional Information is a Strong Supervision for Volumetric Medical Image Segmentation

Positional Information is a Strong Supervision for Volumetric Medical Image Segmentation

3D Multiple-Contextual ROI-Attention Network for Efficient and Accurate Volumetric Medical Image Segmentation

3D Multiple-Contextual ROI-Attention Network for Efficient and Accurate Volumetric Medical Image Segmentation

Machine learning and reduced order modelling for the simulation of braided stent deployment.

Endoluminal reconstruction using flow diverters represents a novel paradigm for the minimally invasive treatment of intracranial aneurysms. The configuration assumed by these very dense braided stents once deployed within the parent vessel is not easily predictable and medical volumetric images alone may be insufficient to plan the treatment satisfactorily. Therefore, here we propose a fast and accurate machine learning and reduced order modelling framework, based on finite element simulations, to assist practitioners in the planning and interventional stages. It consists of a first classification step to determine a priori whether a simulation will be successful (good conformity between stent and vessel) or not from a clinical perspective, followed by a regression step that provides an approximated solution of the deployed stent configuration. The latter is achieved using a non-intrusive reduced order modelling scheme that combines the proper orthogonal decomposition algorithm and Gaussian process regression. The workflow was validated on an idealized intracranial artery with a saccular aneurysm and the effect of six geometrical and surgical parameters on the outcome of stent deployment was studied. We trained six machine learning models on a dataset of varying size and obtained classifiers with up to 95% accuracy in predicting the deployment outcome. The support vector machine model outperformed the others when considering a small dataset of 50 training cases, with an accuracy of 93% and a specificity of 97%. On the other hand, real-time predictions of the stent deployed configuration were achieved with an average validation error between predicted and high-fidelity results never greater than the spatial resolution of 3D rotational angiography, the imaging technique with the best spatial resolution (0.15mm). Such accurate predictions can be reached even with a small database of 47 simulations: by increasing the training simulations to 147, the average prediction error is reduced to 0.07mm. These results are promising as they demonstrate the ability of these techniques to achieve simulations within a few milliseconds while retaining the mechanical realism and predictability of the stent deployed configuration.

A Novel Implicit Neural Representation for Volume Data

The storage of medical images is one of the challenges in the medical imaging field. There are variable works that use implicit neural representation (INR) to compress volumetric medical images. However, there is room to improve the compression rate for volumetric medical images. Most of the INR techniques need a huge amount of GPU memory and a long training time for high-quality medical volume rendering. In this paper, we present a novel implicit neural representation to compress volume data using our proposed architecture, that is, the Lanczos downsampling scheme, SIREN deep network, and SRDenseNet high-resolution scheme. Our architecture can effectively reduce training time, and gain a high compression rate while retaining the final rendering quality. Moreover, it can save GPU memory in comparison with the existing works. The experiments show that the quality of reconstructed images and training speed using our architecture is higher than current works which use the SIREN only. Besides, the GPU memory cost is evidently decreased.

A General Global and Local Pre-Training Framework for 3D Medical Image Segmentation.

Accurate target segmentation from computed tomography (CT) scans is crucial for surgical robots to perform clinical surgeries successfully. However, the lack of medical image data and annotations has been the biggest obstacle to learning robust medical image segmentation models. Self-supervised learning can effectively address this problem by providing a strategy to pre-train a model with unlabeled data, and then fine-tune downstream tasks with limited labeled data. Existing self-supervised methods fail to simultaneously utilize the abundant global anatomical structure information and local feature differences in medical imaging. In this work, we propose a new strategy for the pre-training framework, which uses the three-dimensional anatomical structure of medical images and specific task and background cues to segment volumetric medical images with limited annotations. Specifically, we propose (1) learning intrinsic patterns of volumetric medical image structures through multiple sub-tasks, and (2) designing a multi-level background cube contrastive learning strategy to enhance the target feature representation by exploiting the differences between the specific target and background. We conduct extensive evaluations on two publicly available datasets. Under limited annotation settings, the proposed method yields significant improvements compared to other self-supervised learning techniques. The proposed method achieves within 6% of the baseline performance using only five labeled CT volumes for training. Once the paper is online, the code and dataset will be available at https://github.com/PinkGhost0812/SGL.

NnFormer: Volumetric Medical Image Segmentation via a 3D Transformer.

Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to learn more contextualized visual representations. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer (i.e., not-another transFormer), a 3D transformer for volumetric medical image segmentation. nnFormer not only exploits the combination of interleaved convolution and self-attention operations, but also introduces local and global volume-based self-attention mechanism to learn volume representations. Moreover, nnFormer proposes to use skip attention to replace the traditional concatenation/summation operations in skip connections in U-Net like architecture. Experiments show that nnFormer significantly outperforms previous transformer-based counterparts by large margins on three public datasets. Compared to nnUNet, the most widely recognized convnet-based 3D medical segmentation model, nnFormer produces significantly lower HD95 and is much more computationally efficient. Furthermore, we show that nnFormer and nnUNet are highly complementary to each other in model ensembling. Codes and models of nnFormer are available at https://git.io/JSf3i.

TransMorph: Transformer for unsupervised medical image registration.

In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently, Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehension of the spatial correspondence between moving and fixed images. Here, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. This paper also presents diffeomorphic and Bayesian variants of TransMorph: the diffeomorphic variants ensure the topology-preserving deformations, and the Bayesian variant produces a well-calibrated registration uncertainty estimate. We extensively validated the proposed models using 3D medical images from three applications: inter-patient and atlas-to-patient brain MRI registration and phantom-to-CT registration. The proposed models are evaluated in comparison to a variety of existing registration methods and Transformer architectures. Qualitative and quantitative results demonstrate that the proposed Transformer-based model leads to a substantial performance improvement over the baseline methods, confirming the effectiveness of Transformers for medical image registration.

Multiplicative Multiresolution Decomposition for Lossless Volumetric Medical Images Compression

With the emergence of medical imaging, the compression of volumetric medical images is essential. For this purpose, we propose a novel Multiplicative Multiresolution Decomposition (MMD) wavelet coding scheme for lossless compression of volumetric medical images. The MMD is used in speckle reduction technique but offers some proprieties which can be exploited in compression. Thus, as the wavelet transform the MMD provides a hierarchical representation and offers a possibility to realize lossless compression. We integrate in proposed scheme an inter slice filter based on wavelet transform and motion compensation to reduce data energy efficiently. We compare lossless results of classical wavelet coders such as 3D SPIHT and JP3D to the proposed scheme. This scheme incorporates MMD in lossless compression technique by applying MMD/wavelet or MMD transform to each slice, after inter slice filter is employed and the resulting sub-bands are coded by the 3D zero-tree algorithm SPIHT. Lossless experimental results show that the proposed scheme with the MMD can achieve lowest bit rates compared to 3D SPIHT and JP3D.

Distributed contrastive learning for medical image segmentation.

Supervised deep learning needs a large amount of labeled data to achieve high performance. However, in medical imaging analysis, each site may only have a limited amount of data and labels, which makes learning ineffective. Federated learning (FL) can learn a shared model from decentralized data. But traditional FL requires fully-labeled data for training, which is very expensive to obtain. Self-supervised contrastive learning (CL) can learn from unlabeled data for pre-training, followed by fine-tuning with limited annotations. However, when adopting CL in FL, the limited data diversity on each site makes federated contrastive learning (FCL) ineffective. In this work, we propose two federated self-supervised learning frameworks for volumetric medical image segmentation with limited annotations. The first one features high accuracy and fits high-performance servers with high-speed connections. The second one features lower communication costs, suitable for mobile devices. In the first framework, features are exchanged during FCL to provide diverse contrastive data to each site for effective local CL while keeping raw data private. Global structural matching aligns local and remote features for a unified feature space among different sites. In the second framework, to reduce the communication cost for feature exchanging, we propose an optimized method FCLOpt that does not rely on negative samples. To reduce the communications of model download, we propose the predictive target network update (PTNU) that predicts the parameters of the target network. Based on PTNU, we propose the distance prediction (DP) to remove most of the uploads of the target network. Experiments on a cardiac MRI dataset show the proposed two frameworks substantially improve the segmentation and generalization performance compared with state-of-the-art techniques.

Efficient contour-based annotation by iterative deep learning for organ segmentation from volumetric medical images

PurposeTraining deep neural networks usually require a large number of human-annotated data. For organ segmentation from volumetric medical images, human annotation is tedious and inefficient. To save human labour and to accelerate the training process, the strategy of annotation by iterative deep learning recently becomes popular in the research community. However, due to the lack of domain knowledge or efficient human-interaction tools, the current AID methods still suffer from long training time and high annotation burden.MethodsWe develop a contour-based annotation by iterative deep learning (AID) algorithm which uses boundary representation instead of voxel labels to incorporate high-level organ shape knowledge. We propose a contour segmentation network with a multi-scale feature extraction backbone to improve the boundary detection accuracy. We also developed a contour-based human-intervention method to facilitate easy adjustments of organ boundaries. By combining the contour-based segmentation network and the contour-adjustment intervention method, our algorithm achieves fast few-shot learning and efficient human proofreading.ResultsFor validation, two human operators independently annotated four abdominal organs in computed tomography (CT) images using our method and two compared methods, i.e. a traditional contour-interpolation method and a state-of-the-art (SOTA) convolutional network (CNN) method based on voxel label representation. Compared to these methods, our approach considerably saved annotation time and reduced inter-rater variabilities. Our contour detection network also outperforms the SOTA nnU-Net in producing anatomically plausible organ shape with only a small training set.ConclusionTaking advantage of the boundary shape prior and the contour representation, our method is more efficient, more accurate and less prone to inter-operator variability than the SOTA AID methods for organ segmentation from volumetric medical images. The good shape learning ability and flexible boundary adjustment function make it suitable for fast annotation of organ structures with regular shape.

Multilevel segmentation of 2D and volumetric medical images using hybrid Coronavirus Optimization Algorithm

Medical image segmentation is a crucial step in Computer-Aided Diagnosis systems, where accurate segmentation is vital for perfect disease diagnoses. This paper proposes a multilevel thresholding technique for 2D and 3D medical image segmentation using Otsu and Kapur's entropy methods as fitness functions to determine the optimum threshold values. The proposed algorithm applies the hybridization concept between the recent Coronavirus Optimization Algorithm (COVIDOA) and Harris Hawks Optimization Algorithm (HHOA) to benefit from both algorithms' strengths and overcome their limitations. The improved performance of the proposed algorithm over COVIDOA and HHOA algorithms is demonstrated by solving 5 test problems from IEEE CEC 2019 benchmark problems. Medical image segmentation is tested using two groups of images, including 2D medical images and volumetric (3D) medical images, to demonstrate its superior performance. The utilized test images are from different modalities such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and X-ray images. The proposed algorithm is compared with seven well-known metaheuristic algorithms, where the performance is evaluated using four different metrics, including the best fitness values, Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Normalized Correlation Coefficient (NCC). The experimental results demonstrate the superior performance of the proposed algorithm in terms of convergence to the global optimum and making a good balance between exploration and exploitation properties. Moreover, the quality of the segmented images using the proposed algorithm at different threshold levels is better than the other methods according to PSNR, SSIM, and NCC values. Additionally, the Wilcoxon rank-sum test is conducted to prove the statistical significance of the proposed algorithm.

A reciprocal learning strategy for semisupervised medical image segmentation.

Semisupervised strategy has been utilized to alleviate issues from segmentation applications due to challenges in collecting abundant annotated segmentation masks, which is an essential prerequisite for training high-performance 3D convolutional neural networks (CNNs) . Existing semisupervised segmentation methods are mainly concerned with how to generate the pseudo labels with regularization but not evaluate the quality of the pseudo labels explicitly. To alleviate this problem, we offer a simple yet effective reciprocal learning strategy for semisupervised volumetric medical image segmentation, which generates more reliable pseudo labels for the unannotateddata. Our proposed reciprocal learning is achieved through a pair of networks, one as a teacher network and the other as a student network. The student network learns from pseudo labels generated by the teacher network. In addition, the teacher network autonomously optimizes its parameters based on the reciprocal feedback signals from the student's performance on the annotated images. The efficacy of the proposed method is evaluated on three medical image data sets, including 82 pancreas computed tomography (CT) scans (training/testing: 62/20), 100 left atrium gadolinium-enhanced magnetic resonance (MR) scans (training/testing: 80/20), and 200 breast cancer MR scans (training/testing: 68/132). The comparison methods include mean teacher (MT) model, uncertainty-aware MT (UA-MT) model, shape-aware adversarial network (SASSNet), and transformation-consistent self-ensembling model (TCSM). The evaluation metrics are Dice similarity coefficient (Dice), Jaccard index (Jaccard), 95% Hausdorff distance (95HD), and average surface distance (ASD). The Wilcoxon signed-rank test is used to conduct the statisticalanalyses. By utilizing 20% labeled data and 80% unlabeled data for training, our proposed method achieves an average Dice of 84.77%/90.46%/78.53%, Jaccard of 73.71%/82.67%/69.00%, ASD of 1.58/1.90/0.57, and 95HD of 6.24/5.97/4.34 on pancreas/left atrium/breast data sets, respectively. These results outperform several cutting-edge semisupervised approaches, showing the feasibility of our method for the challenging semisupervised segmentationapplications. The proposed reciprocal learning strategy is a general semisupervised solution and has the potential to be applied for other 3D segmentationtasks.

Explainable multiple abnormality classification of chest CT volumes

Understanding model predictions is critical in healthcare, to facilitate rapid verification of model correctness and to guard against use of models that exploit confounding variables. We introduce the challenging new task of explainable multiple abnormality classification in volumetric medical images, in which a model must indicate the regions used to predict each abnormality. To solve this task, we propose a multiple instance learning convolutional neural network, AxialNet, that allows identification of top slices for each abnormality. Next we incorporate HiResCAM, an attention mechanism, to identify sub-slice regions. We prove that for AxialNet, HiResCAM explanations are guaranteed to reflect the locations the model used, unlike Grad-CAM which sometimes highlights irrelevant locations. Armed with a model that produces faithful explanations, we then aim to improve the model’s learning through a novel mask loss that leverages HiResCAM and 3D allowed regions to encourage the model to predict abnormalities based only on the organs in which those abnormalities appear. The 3D allowed regions are obtained automatically through a new approach, PARTITION, that combines location information extracted from radiology reports with organ segmentation maps obtained through morphological image processing. Overall, we propose the first model for explainable multi-abnormality prediction in volumetric medical images, and then use the mask loss to achieve a 33% improvement in organ localization of multiple abnormalities in the RAD-ChestCT dataset of 36,316 scans, representing the state of the art. This work advances the clinical applicability of multiple abnormality modeling in chest CT volumes.

Bridging 2D and 3D segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5D solutions.

Recently, deep convolutional neural networks have achieved great success for medical image segmentation. However, unlike segmentation of natural images, most medical images such as MRI and CT are volumetric data. In order to make full use of volumetric information, 3D CNNs are widely used. However, 3D CNNs suffer from higher inference time and computation cost, which hinders their further clinical applications. Additionally, with the increased number of parameters, the risk of overfitting is higher, especially for medical images where data and annotations are expensive to acquire. To issue this problem, many 2.5D segmentation methods have been proposed to make use of volumetric spatial information with less computation cost. Despite these works lead to improvements on a variety of segmentation tasks, to the best of our knowledge, there has not previously been a large-scale empirical comparison of these methods. In this paper, we aim to present a review of the latest developments of 2.5D methods for volumetric medical image segmentation. Additionally, to compare the performance and effectiveness of these methods, we provide an empirical study of these methods on three representative segmentation tasks involving different modalities and targets. Our experimental results highlight that 3D CNNs may not always be the best choice. Despite all these 2.5D methods can bring performance gains to 2D baseline, not all the methods hold the benefits on different datasets. We hope the results and conclusions of our study will prove useful for the community on exploring and developing efficient volumetric medical image segmentation methods.