3D U-Net Research Articles

Investigating deep learning strategies for fast denoising of 5D cardiac photon-counting micro-CT images


Photon-counting detectors (PCDs) for CT imaging use energy thresholds to simultaneously acquire projections at multiple energies, making them suitable for spectral imaging and material decomposition. Unfortunately, setting multiple energy thresholds results in noisy analytical reconstructions due to low photon counts in high-energy bins. Iterative reconstruction provides high quality photon-counting CT (PCCT) images but requires enormous computation time for 5D (3D + energy + time) in vivo cardiac imaging. 

Approach. 
We recently introduced UnetU, a deep learning (DL) approach that accurately denoises axial slices from 4D (3D + energy) PCCT reconstructions at various acquisition settings. In this study, we explore UnetU configurations for 5D cardiac PCCT denoising, focusing on singular value decomposition (SVD) modifications along the energy and time dimensions and alternate network architectures such as 3D U-net, FastDVDNet, and Swin Transformer UNet. We compare our networks to multi-energy non-local means (ME NLM), an established PCCT denoising algorithm. 

Main results. 
Our evaluation, using real mouse data and the digital MOBY phantom, revealed that all DL methods were more than 16 times faster than iterative reconstruction. DL denoising with SVD along the energy dimension was most effective, consistently providing low root mean square error and spatio-temporal reduced reference entropic difference, alongside strong qualitative agreement with iterative reconstruction. This superiority was attributed to lower effective rank along the energy dimension than the time dimension in 5D cardiac PCCT reconstructions. ME NLM sometimes outperformed DL with time SVD or time and energy SVD, but lagged behind iterative reconstruction and DL with energy SVD. Among alternate DL architectures with energy SVD, none consistently outperformed UnetU Energy (2D). 

Significance. 
Our study establishes UnetU Energy as an accurate and efficient method for 5D cardiac PCCT denoising, offering a 32-fold speed increase from iterative reconstruction. This advancement sets a new benchmark for DL applications in cardiovascular imaging.

UNet-based multi-organ segmentation in photon counting CT using virtual monoenergetic images.

Multi-organ segmentation aids in disease diagnosis, treatment, and radiotherapy. The recently emerged photon counting detector-based CT (PCCT) provides spectral information of the organs and the background tissue and may improve segmentationperformance. We propose UNet-based multi-organ segmentation in PCCT using virtual monoenergetic images (VMI) to exploit spectral informationeffectively. The proposed method consists of the following steps: Noise reduction in bin-wise images, image-based material decomposition, generating VMIs, and deep learning-based segmentation. VMIs are synthesized for various x-ray energies using basis images. The UNet-based networks (3D UNet, Swin UNETR) were used for segmentation, and dice similarity coefficients (DSC) and 3D visualization of the segmented result were evaluation indicators. We validated the proposed method for the liver, pancreas, and spleen segmentation using abdominal phantoms from 55 subjects for dual- and quad-energy bins. We compared it to the conventional PCCT-based segmentation, which uses only the (noise-reduced) bin-wise images. The experiments were conducted on two cases by adjusting the doselevels. The proposed method improved the training stability for most cases. With the proposed method, the average DSC for the three organs slightly increased from 0.933 to 0.95, and the standard deviation decreased from 0.066 to 0.047, for example, in the low dose case (using VMIs v.s. bin-wise images from dual-energy bins; 3D UNet). The proposed method using VMIs improves training stability for multi-organ segmentation in PCCT, particularly when the number of energy bins issmall.

Using 2D U-Net convolutional neural networks for automatic acetabular and proximal femur segmentation of hip MRI images and morphological quantification: a preliminary study in DDH

BackgroundDevelopmental dysplasia of the hip (DDH) is a common pediatric orthopedic condition characterized by varying degrees of acetabular dysplasia and hip dislocation. Current 2D imaging methods often fail to provide sufficient anatomical detail for effective treatment planning, leading to higher rates of misdiagnosis and missed diagnoses. MRI, with its advantages of being radiation-free, multi-planar, and containing more anatomical information, can provide the crucial morphological and volumetric data needed to evaluate DDH. However, manual techniques for measuring parameters like the center–edge angle (CEA) and acetabular index (AI) are time-consuming. Automating these processes is essential for accurate clinical assessments and personalized treatment strategies.MethodsThis study employed a U-Net-based CNN model to automate the segmentation of hip MRI images in children. The segmentation process was validated using a leave-one-out method during training. Subsequently, the segmented hip joint images were utilized in clinical settings to perform automated measurements of key angles: AI, femoral neck angle (FNA), and CEA. This automated approach aimed to replace manual measurements and provide an objective reference for clinical assessments.ResultsThe U-Net-based network demonstrates high effectiveness in hip segmentation compared to manual radiologist segmentations. In test data, it achieves average DSC values of 0.9109 (acetabulum) and 0.9244 (proximal femur), with a 91.76% segmentation success rate. The average ASD values are 0.3160 mm (acetabulum) and 0.6395 mm (proximal femur) in test data, with Ground Truth (GT) edge points and predicted segmentation maps having a mean distance of less than 1 mm. Using automated segmentation models for clinical hip angle measurements (CEA, AI, FNA) shows no statistical difference compared to manual measurements (p > 0.05).ConclusionUtilizing U-Net-based image segmentation and automated measurement of morphological parameters significantly enhances the accuracy and efficiency of DDH assessment. These methods improve precision in automatic measurements and provide an objective basis for clinical diagnosis and treatment of DDH.

Reinforcement learning-based anatomical maps for pancreas subregion and duct segmentation.

The pancreas is a complex abdominal organ with many anatomical variations, and therefore automated pancreas segmentation from medical images is a challengingapplication. In this paper, we present a framework for segmenting individual pancreatic subregions and the pancreatic duct from three-dimensional (3D) computed tomography (CT) images. A multiagent reinforcement learning (RL) network was used to detect landmarks of the head, neck, body, and tail of the pancreas, and landmarks along the pancreatic duct in a selected target CT image. Using the landmark detection results, an atlas of pancreases was nonrigidly registered to the target image, resulting in anatomical probability maps for the pancreatic subregions and duct. The probability maps were augmented with multilabel 3D U-Net architectures to obtain the final segmentationresults. To evaluate the performance of our proposed framework, we computed the Dice similarity coefficient (DSC) between the predicted and ground truth manual segmentations on a database of 82 CT images with manually segmented pancreatic subregions and 37 CT images with manually segmented pancreatic ducts. For the four pancreatic subregions, the mean DSC improved from 0.38, 0.44, and 0.39 with standard 3D U-Net, Attention U-Net, and shifted windowing (Swin) U-Net architectures, to 0.51, 0.47, and 0.49, respectively, when utilizing the proposed RL-based framework. For the pancreatic duct, the RL-based framework achieved a mean DSC of 0.70, significantly outperforming the standard approaches and existing methods on different datasets. The resulting accuracy of the proposed RL-based segmentation framework demonstrates an improvement against segmentation with standard U-Netarchitectures.

Automated deep learning segmentation of cardiac inflammatory FDG PET

BackgroundFluorodeoxyglucose positron emission tomography (FDG PET) with suppression of myocardial glucose utilization plays a pivotal role in diagnosing cardiac sarcoidosis. Reorientation of images to match perfusion datasets and myocardial segmentation enables consistent image scaling and quantification. However, such manual tasks are cumbersome. We developed a 3D U-Net deep-learning (DL) algorithm for automated myocardial segmentation in cardiac sarcoidosis FDG PET. MethodsThe DL model was trained on FDG PET scans from 316 patients with left ventricular contours derived from paired perfusion datasets. Qualitative analysis of clinical readability was performed to compare DL segmentation with the current automated method on a 50-patient test subset. Additionally, left ventricle displacement and angulation, as well as SUVmax sampling were compared with inter-user reproducibility results. A hybrid workflow was also investigated to accelerate study processing time. ResultsDL segmentation enhanced readability scores in over 90% of cases compared with the standard segmentation currently used in the software. DL segmentation performed similar to a trained technologist, surpassing standard segmentation for left ventricle displacement and angulation, as well as correlation of SUVmax. Using the DL segmentation as initial placement for manual segmentation significantly decreased the processing time. ConclusionA novel DL-based automated segmentation tool markedly improves processing of cardiac sarcoidosis FDG PET. This tool yields optimized splash display of sarcoidosis FDG PET datasets with no user input and offers significant processing time improvement for manual segmentation of such datasets.

Knowledge-based planning, multicriteria optimization, and plan scorecards: A winning combination

Background and purposeThe ESTRO 2023 Physics Workshop hosted the Fully-Automated Radiotherapy Treatment Planning (Auto-RTP) Challenge, where participants were provided with CT images from 16 prostate cancer patients (6 prostate only, 6 prostate + nodes, and 4 prostate bed + nodes) across 3 challenge phases with the goal of automatically generating treatment plans with minimal user intervention. Here, we present our team’s winning approach developed to swiftly adapt to both different contouring guidelines and treatment prescriptions than those used in our clinic. Materials and methodsOur planning pipeline comprises two main components: 1) auto-contouring and 2) auto-planning engines, both internally developed and activated via DICOM operations. The auto-contouring engine employs 3D U-Net models trained on a dataset of 600 prostate cancer patients for normal tissues, 253 cases for pelvic lymph node, and 32 cases for prostate bed. The auto-planning engine, utilizing the Eclipse Scripting Application Programming Interface, automates target volume definition, field geometry, planning parameters, optimization, and dose calculation. RapidPlan models, combined with multicriteria optimization and scorecards defined on challenge scoring criteria, were employed to ensure plans met challenge objectives. We report leaderboard scores (0–100, where 100 is a perfect score) which combine organ-at-risk and target dose-metrics on the provided cases. ResultsOur team secured 1st place across all three challenge phases, achieving leaderboard scores of 79.9, 77.3, and 78.5 outperforming 2nd place scores by margins of 6.4, 0.4, and 2.9 points for each phase, respectively. Highest plan scores were for prostate only cases, with an average score exceeding 90. Upon challenge completion, a “Plan Only” phase was opened where organizers provided contours for planning. Our current score of 90.0 places us at the top of the “Plan Only” leaderboard. ConclusionsOur automated pipeline demonstrates adaptability to diverse guidelines, indicating progress towards fully automated radiotherapy planning. Future studies are needed to assess the clinical acceptability and integration of automatically generated plans.

Deep learning disconnectomes to accelerate and improve long-term predictions for post-stroke symptoms.

This study investigates the efficacy of deep-learning models in expediting the generation of disconnectomes for individualized prediction of neuropsychological outcomes one year after stroke. Utilising a 3D U-Net network, we trained a model on a dataset of N = 1333 synthetic lesions and corresponding disconnectomes, subsequently applying it to N = 1333 real stroke lesions. The model-generated disconnection patterns were then projected into a two-dimensional 'morphospace' via uniform manifold approximation and projection for dimension reduction dimensionality reduction. We correlated the positioning within this morphospace with one-year neuropsychological scores across 86 metrics in a novel cohort of 119 stroke patients, employing multiple regression models and validating the findings in an out-of-sample group of 20 patients. Our results demonstrate that the 3D U-Net model captures the critical information of conventional disconnectomes with a notable increase in efficiency, generating deep-disconnectomes 720 times faster than current state-of-the-art software. The predictive accuracy of neuropsychological outcomes by deep-disconnectomes averaged 85.2% (R 2 = 0.208), which significantly surpassed the conventional disconnectome approach (P = 0.009). These findings mark a substantial advancement in the production of disconnectome maps via deep learning, suggesting that this method could greatly enhance the prognostic assessment and clinical management of stroke survivors by incorporating disconnection patterns as a predictive tool.

Image quality improvement in single plane-wave imaging using deep learning

In ultrasound image diagnosis, single plane-wave imaging (SPWI), which can acquire ultrasound images at more than 1000 fps, has been used to observe detailed tissue and evaluate blood flow. SPWI achieves high temporal resolution by sacrificing the spatial resolution and contrast of ultrasound images. To improve spatial resolution and contrast in SPWI, coherent plane-wave compounding (CPWC) is used to obtain high-quality ultrasound images, i.e., compound images, by coherent addition of radio frequency (RF) signals acquired by transmitting plane waves in different directions. Although CPWC produces high-quality ultrasound images, their temporal resolution is lower than that of SPWI. To address this problem, some methods have been proposed to reconstruct a ultrasound image comparable to a compound image from RF signals obtained by transmitting a small number of plane waves in different directions. These methods do not fully consider the properties of RF signals, resulting in lower image quality compared to a compound image. In this paper, we propose methods to reconstruct high-quality ultrasound images in SPWI by considering the characteristics of RF signal of a single plane wave to obtain ultrasound images with image quality comparable to CPWC. The proposed methods employ encoder–decoder models of 1D U-Net, 2D U-Net, and their combination to generate the high-quality ultrasound images by minimizing the loss that considers the point spread effect of plane waves and frequency spectrum of RF signals in training. We also create a public large-scale SPWI/CPWC dataset for developing and evaluating deep-learning methods. Through a set of experiments using the public dataset and our dataset, we demonstrate that the proposed methods can reconstruct higher-quality ultrasound images from RF signals in SPWI than conventional method.

Label refinement network from synthetic error augmentation for medical image segmentation

Deep convolutional neural networks for image segmentation do not learn the label structure explicitly and may produce segmentations with an incorrect structure, e.g., with disconnected cylindrical structures in the segmentation of tree-like structures such as airways or blood vessels. In this paper, we propose a novel label refinement method to correct such errors from an initial segmentation, implicitly incorporating information about label structure. This method features two novel parts: (1) a model that generates synthetic structural errors, and (2) a label appearance simulation network that produces segmentations with synthetic errors that are similar in appearance to the real initial segmentations. Using these segmentations with synthetic errors and the original images, the label refinement network is trained to correct errors and improve the initial segmentations. The proposed method is validated on two segmentation tasks: airway segmentation from chest computed tomography (CT) scans and brain vessel segmentation from 3D CT angiography (CTA) images of the brain. In both applications, our method significantly outperformed a standard 3D U-Net, four previous label refinement methods, and a U-Net trained with a loss tailored for tubular structures. Improvements are even larger when additional unlabeled data is used for model training. In an ablation study, we demonstrate the value of the different components of the proposed method.

Robust and smooth Couinaud segmentation via anatomical structure-guided point-voxel network

Precise Couinaud segmentation from preoperative liver computed tomography (CT) is crucial for surgical planning and lesion examination. However, this task is challenging as it is defined based on vessel structures, and there is no intensity contrast between adjacent Couinaud segments in CT images. To solve this challenge, we design a multi-scale point-voxel fusion framework, which can more effectively model the spatial relationship of points and the semantic information of the image, producing robust and smooth Couinaud segmentations. Specifically, we first segment the liver and vessels from the CT image and generate 3D liver point clouds and voxel grids embedded with the vessel structure. Then, our method with two input-specific branches extracts complementary feature representations from points and voxels, respectively. The local attention module adaptively fuses features from the two branches at different scales to balance the contribution of different branches in learning more discriminative features. Furthermore, we propose a novel distance loss at the feature level to make the features in the segment more compact, thereby improving the certainty of segmentation between segments. Our experimental results on three public liver datasets demonstrate that our proposed method outperforms several state-of-the-art methods by large margins. Specifically, in out-of-distribution (OOD) testing of LiTS dataset, our method exceeded the voxel-based 3D UNet by approximately 20% in Dice score, and outperformed the point-based PointNet2Plus by approximately 8% in Dice score. Our code and manual annotations of the public datasets presented in this paper are available online: https://github.com/xukun-zhang/Couinaud-Segmentation.

Enhancing brain image quality with 3D U-net for stripe removal in light sheet fluorescence microscopy

Light Sheet Fluorescence Microscopy (LSFM) is increasingly popular in neuroimaging for its ability to capture high-resolution 3D neural data. However, the presence of stripe noise significantly degrades image quality, particularly in complex 3D stripes with varying widths and brightness, posing challenges in neuroscience research. Existing stripe removal algorithms excel in suppressing noise and preserving details in 2D images with simple stripes but struggle with the complexity of 3D stripes. To address this, we propose a novel 3D U-net model for Stripe Removal in Light sheet fluorescence microscopy (USRL). This approach directly learns and removes stripes in 3D space across different scales, employing a dual-resolution strategy to effectively handle stripes of varying complexities. Additionally, we integrate a nonlinear mapping technique to normalize high dynamic range and unevenly distributed data before applying the stripe removal algorithm. We validate our method on diverse datasets, demonstrating substantial improvements in peak signal-to-noise ratio (PSNR) compared to existing algorithms. Moreover, our algorithm exhibits robust performance when applied to real LSFM data. Through extensive validation experiments, both on test sets and real-world data, our approach outperforms traditional methods, affirming its effectiveness in enhancing image quality. Furthermore, the adaptability of our algorithm extends beyond LSFM applications to encompass other imaging modalities. This versatility underscores its potential to enhance image usability across various research disciplines.

Cross-site Validation of AI Segmentation and Harmonization in Breast MRI.

This work aims to perform a cross-site validation of automated segmentation for breast cancers in MRI and to compare the performance to radiologists. A three-dimensional (3D) U-Net was trained to segment cancers in dynamic contrast-enhanced axial MRIs using a large dataset from Site 1 (n = 15,266; 449 malignant and 14,817 benign). Performance was validated on site-specific test data from this and two additional sites, and common publicly available testing data. Four radiologists from each of the three clinical sites provided two-dimensional (2D) segmentations as ground truth. Segmentation performance did not differ between the network and radiologists on the test data from Sites 1 and 2 or the common public data (median Dice score Site 1, network 0.86 vs. radiologist 0.85, n = 114; Site 2, 0.91 vs. 0.91, n = 50; common: 0.93 vs. 0.90). For Site 3, an affine input layer was fine-tuned using segmentation labels, resulting in comparable performance between the network and radiologist (0.88 vs. 0.89, n = 42). Radiologist performance differed on the common test data, and the network numerically outperformed 11 of the 12 radiologists (median Dice: 0.85-0.94, n = 20). In conclusion, a deep network with a novel supervised harmonization technique matches radiologists' performance in MRI tumor segmentation across clinical sites. We make code and weights publicly available to promote reproducible AI in radiology.

Large-language-model empowered 3D dose prediction for intensity-modulated radiotherapy.

Treatment planning is currently a patient specific, time-consuming, and resource demanding task in radiotherapy. Dose-volume histogram (DVH) prediction plays a critical role in automating this process. The geometric relationship between DVHs in radiotherapy plans and organs-at-risk (OAR) and planning target volume (PTV) has been well established. This study explores the potential of deep learning models for predicting DVHs using images and subsequent human intervention facilitated by a large-language model (LLM) to enhance the planning quality. We propose a pipeline to convert unstructured images to a structured graph consisting of image-patch nodes and dose nodes. A novel Dose Graph Neural Network (DoseGNN) model is developed for predicting DVHs from the structured graph. The proposed DoseGNN is enhanced with the LLM to encode massive knowledge from prescriptions and interactive instructions from clinicians. In this study, we introduced an online human-AI collaboration (OHAC) system as a practical implementation of the concept proposed for the automation of intensity-modulated radiotherapy (IMRT) planning. The proposed DoseGNN model was compared to widely employed DL models used in radiotherapy, including Swin Transformer, 3D U-Net CNN, and vanilla MLP. For PTV, DoseGNN achieved the mean absolute error (MAE) of , , , and between true plans and predicted plans that were 64%, 53%, 64%, 61% of the best baseline model. For the worst case among OARs (left lung, right lung, chest wall, heart, spinal cord), DoseGNN achieved the mean absolute error of , , that were 85%, 91%, 80% of the best baseline model. Moreover, the LLM-empowered DoseGNN model facilitates seamless adjustment to treatment plans through interaction with clinicians using natural language. We developed DoseGNN, a novel deep learning model for predicting delivered radiation doses from medical images, enhanced by LLM to allow adjustment through seamless interaction with clinicians. The preliminary results confirm DoseGNN's superior accuracy in DVH prediction relative to typical DL methods, highlighting its potential to facilitate an online clinician-AI collaboration system for streamlined treatment planning automation.

Deep learning–based statistical robustness evaluation of intensity-modulated proton therapy for head and neck cancer

A dense, dilated 3D U-net was trained to predict the 5th and 95th percentile distributions of planning dose plan using the nominal dose and planning CT images. The data set comprised proton therapy plans for 582 H&N cancer patients. Ground truth percentile values were estimated for each patient through 600 dose recalculations, representing randomly sampled uncertainty scenarios. The comprehensive comparisons of different models were conducted for H&N cancer patients, considering those with and without a beam mask (BM) and diverse beam configurations, including varying beam angles, couch angles, and beam numbers. The performance of our model trained based on a mixture of patients with H&N and prostate cancer was also assessed in contrast with models trained based on data specific for patients with cancer at either site. The DL-based model's predictions of percentile dose distributions exhibited excellent agreement with the ground truth dose distributions. The average gamma index with 2 mm/2%, consistently exceeded 97% for both 5th and 95th percentile dose volumes. Mean dose-volume histogram error analysis revealed that predictions from the combined training set yielded mean errors and standard deviations that were generally similar to those in the specific patient training data sets. Our proposed DL-based method for evaluation of the robustness of proton therapy plans provides precise, rapid predictions of percentile dose for a given confidence level regardless of the beam arrangement and cancer site. This versatility positions our model as a valuable tool for evaluating the robustness of proton therapy across various cancer sites.

Intelligent diagnostic method for developmental hip dislocation

BackgroundDevelopmental dislocation of the hip joint (DDH) is a condition that severely threatens children’s healthy growth. Without timely and correct treatment, it will lead to osteoarthritis and hip dysfunction in the evolution of children.ObjectiveIt is essential to develop an intelligent model for diagnosing hip dislocation and performing accurate quantitative analysis.MethodsIn this paper, 46 cases of computed tomography (CT) images were retrospectively collected, including 19 cases of hip dislocation and 27 cases of healthy people. The experiment first uses ITK-SNAP to sketch the ilium and femoral head in the original image. Then, it uses 3D U-Net to send the label of the background, ilium, and femoral head into three channels, respectively, to realize the three-dimensional segmentation of the ilium and femoral head. Next, the extraction of the surface of the acetabulum and femoral head is performed. Subsequently, the erroneous points are eliminated, and the spherical surfaces of the acetabulum and femoral head are fitted using the least squares method. Ultimately, the spherical center distance is calculated quantitatively to predict whether the hip joint is dislocated.ResultsUnder the independent test set, the segmentation average dice coefficients of the ilium and femoral head are 89% and 93%, respectively. The spherical center distance between the acetabulum and femoral head is calculated quantitatively. If the value exceeds 10 mm, it is considered a hip dislocation. Compared with the doctor’s diagnosis, the accuracy result is 94.4%.ConclusionThis paper successfully implements a precise and automated intelligent diagnostic system for the identification of hip dislocation. Commencing with the development of a 3D segmentation algorithm for the ilium and femoral head, we further introduce a novel method that computes the spherical distance for the prediction of hip dislocation. This approach provides robust quantitative analysis, thereby facilitating more informed clinical decision-making.

Exploring the impact of network depth on 3D U-Net-based dose prediction for cervical cancer radiotherapy.

The 3D U-Net deep neural network structure is widely employed for dose prediction in radiotherapy. However, the attention to the network depth and its impact on the accuracy and robustness of dose prediction remains inadequate. 92 cervical cancer patients who underwent Volumetric Modulated Arc Therapy (VMAT) are geometrically augmented to investigate the effects of network depth on dose prediction by training and testing three different 3D U-Net structures with depths of 3, 4, and 5. For planning target volume (PTV), the differences between predicted and true values of D98, D99, and Homogeneity were statistically 1.00 ± 0.23, 0.32 ± 0.72, and -0.02 ± 0.02 for the model with a depth of 5, respectively. Compared to the other two models, these parameters were also better. For most of the organs at risk, the mean and maximum differences between the predicted values and the true values for the model with a depth of 5 were better than for the other two models. The results reveal that the network model with a depth of 5 exhibits superior performance, albeit at the expense of the longest training time and maximum computational memory in the three models. A small server with two NVIDIA GeForce RTX 3090 GPUs with 24 G of memory was employed for this training. For the 3D U-Net model with a depth of more than 5 cannot be supported due to insufficient training memory, the 3D U-Net neural network with a depth of 5 is the commonly used and optimal choice for small servers.

An hetero-modal deep learning framework for medical image synthesis applied to contrast and non-contrast MRI

Some pathologies such as cancer and dementia require multiple imaging modalities to fully diagnose and assess the extent of the disease. Magnetic resonance imaging offers this kind of polyvalence, but examinations take time and can require contrast agent injection. The flexible synthesis of these imaging sequences based on the available ones for a given patient could help reduce scan times or circumvent the need for contrast agent injection. In this work, we propose a deep learning architecture that can perform the synthesis of all missing imaging sequences from any subset of available images. The network is trained adversarially, with the generator consisting of parallel 3D U-Net encoders and decoders that optimally combines their multi-resolution representations with a fusion operation learned by an attention network trained conjointly with the generator network. We compare our synthesis performance with 3D networks using other types of fusion and a comparable number of trainable parameters, such as the mean/variance fusion. In all synthesis scenarios except one, the synthesis performance of the network using attention-guided fusion was better than the other fusion schemes. We also inspect the encoded representations and the attention network outputs to gain insights into the synthesis process, and uncover desirable behaviors such as prioritization of specific modalities, flexible construction of the representation when important modalities are missing, and modalities being selected in regions where they carry sequence-specific information. This work suggests that a better construction of the latent representation space in hetero-modal networks can be achieved by using an attention network.

Using 3D U-Net for Brain Tumour Segmentation from Magnetic Resonance Images

Brain tumours within the skull can lead to serious health issues. The rapid and accurate detection and segmentation of tumour regions allow patients to receive appropriate treatment at an early stage, increasing their chances of recovery and survival. Various medical imaging methods, such as Magnetic Resonance Imaging (MRI), Positron and digital pathology, Emission Tomography (PET), Computed Tomography (CT) are used for the detection of brain tumours. Nowadays, with advancing technology and hardware, concepts like artificial intelligence and deep learning (DL) are becoming increasingly popular. Many artificial intelligence methods are also being utilized in studies on brain tumour segmentation. This paper proposes a 3D U-Net DL model for brain tumour segmentation. The training and testing processes are carried out on the Brain Tumour Segmentation (BraTS) 2020 dataset, which is widely used in the literature. As a result, an Intersection over Union (IoU) score of 0.81, a dice score of 0.87 and a pixel accuracy of 0.99 are achieved. The proposed model has the potential to assist experts in diagnosing the disease and developing appropriate treatment plans, thanks to its ability to segment brain tumours quickly and with high accuracy.

Evaluation of an automated laminar cartilage T2 relaxation time analysis method in an early osteoarthritis model.

A fully automated laminar cartilage composition (MRI-based T2) analysis method was technically and clinically validated by comparing radiographically normal knees with (CL-JSN) and without contra-lateral joint space narrowing or other signs of radiographic osteoarthritis (OA, CL-noROA). 2D U-Nets were trained from manually segmented femorotibial cartilages (n = 72) from all 7 echoes (AllE), or from the 1st echo only (1stE) of multi-echo-spin-echo (MESE) MRIs acquired by the Osteoarthritis Initiative (OAI). Because of its greater accuracy, only the AllE U-Net was then applied to knees from the OAI healthy reference cohort (n = 10), CL-JSN (n = 39), and (1:1) matched CL-noROA knees (n = 39) that all had manual expert segmentation, and to 982 non-matched CL-noROA knees without expert segmentation. The agreement (Dice similarity coefficient) between automated vs. manual expert cartilage segmentation was between 0.82 ± 0.05/0.79 ± 0.06 (AllE/1stE) and 0.88 ± 0.03/0.88 ± 0.03 (AllE/1stE) across femorotibial cartilage plates. The deviation between automated vs. manually derived laminar T2 reached up to - 2.2 ± 2.6ms/ + 4.1 ± 10.2ms (AllE/1stE). The AllE U-Net showed a similar sensitivity to cross-sectional laminar T2 differences between CL-JSN and CL-noROA knees in the matched (Cohen's D ≤ 0.54) and the non-matched (D ≤ 0.54) comparison as the matched manual analyses (D ≤ 0.48). Longitudinally, the AllE U-Net also showed a similar sensitivity to CL-JSN vs. CS-noROA differences in the matched (D ≤ 0.51) and the non-matched (D ≤ 0.43) comparison as matched manual analyses (D ≤ 0.41). The fully automated T2 analysis showed a high agreement, acceptable accuracy, and similar sensitivity to cross-sectional and longitudinal laminar T2 differences in an early OA model, compared with manual expert analysis. Clinicaltrials.gov identification: NCT00080171.

A 3D U-NET BASED ON EARLY FUSION MODEL: IMPROVEMENT, COMPARATIVE ANALYSIS WITH STATE-OF-THE-ART MODELS AND FINE-TUNING

Multi-organ segmentation is the process of identifying and separating multiple organs in medical images. This segmentation allows for the detection of structural abnormalities by examining the morphological structure of organs. Carrying out the process quickly and precisely has become an important issue in today's conditions. In recent years, researchers have used various technologies for the automatic segmentation of multiple organs. In this study, improvements were made to increase the multi-organ segmentation performance of the 3D U-Net based fusion model combining HSV and grayscale color spaces and compared with state-of-the-art models. Training and testing were performed on the MICCAI 2015 dataset published at Vanderbilt University, which contains 3D abdominal CT images in NIfTI format. The model's performance was evaluated using the Dice similarity coefficient. In the tests, the liver organ showed the highest Dice score. Considering the average Dice score of all organs, and comparing it with other models, it has been observed that the fusion approach model yields promising results.